
On June 1, 2026, a hVM precompile call in a force-included transaction triggered a runtime error in the Hemi mainnet sequencer during block production, causing the sequencer to crash and halt the creation of blocks on Hemi mainnet.
Between 14:48 UTC and 16:50 UTC, Hemi Mainnet was unable to create new blocks or process transactions. This impacted anyone attempting to use the Hemi blockchain.
We’re very sorry for this incident. In this post, we explain what failed, why it occurred, and what we’re doing to make sure this doesn’t happen again.
Background
The Hemi blockchain is based on Optimism, and contains a Bitcoin node embedded in the EVM which smart contracts can query via new precompiles introduced as part of Hemi’s “hVM” system.
These precompiles provide smart contracts with direct access to indexed Bitcoin state including consensus and transaction details, address/script balances, and output spend status.
Being based on Optimism, Hemi also inherits Optimism’s Ethereum-based anti-censorship mechanism allowing the force-inclusion of L2 transactions by sending the intended transaction to the Portal Proxy contract on Ethereum. This force-inclusion process bypasses the normal RPC transaction submission and P2P transaction propagation paths.
What happened
At 14:46:47 UTC on June 1, 2026, a transaction on Ethereum with hash 0xa5ff29629f9e4ba7bfbe4d58897c6eea2da4a4b5038487192413747363f09fc3 triggered the creation of a force-inclusion transaction on Hemi Mainnet, which called a hVM precompile call with malformed input that caused an runtime error panic, halting block production and chain progression.
This transaction called the hVM btcTxConfirmations precompile (address 0x43) with a 4-byte argument a82f7c66.
This precompile is meant to be called with a 32-byte TxID as the argument. The precompile execution implementation performs input validation and checks the input length, but was missing a crucial return/error path. As a result, execution fell through where the malformed input is parsed, resulting in the following runtime error crashing the Sequencer at 14:48 UTC:
This missing error return in the btcTxConfirmations precompile was a known bug found by internal review and independently reported through our bug bounty program, and was scheduled to be fixed in an upcoming network upgrade, as changing precompile behavior requires a coordinated deployment with all node operators to prevent chain state divergences.
The bug was incorrectly deemed a low-severity vulnerability on initial review because a regular L2 transaction that performed this malformed precompile call would only result in a runtime error in the goroutine handling the transaction on RPC submission or P2P broadcast, which would simply prevent the transaction from being accepted into the mempool and sequenced into a block.
However, the bug could also be triggered by an L1->L2 force-inclusion transaction which bypasses RPC submission and P2P transaction propagation, causing the runtime error to be triggered during the Sequencer’s block creation, resulting in the Sequencer entering a crash loop and being unable to progress the chain.
The force-inclusion protocol rule prevented us from implementing a simple fix in the sequencer, as all other Hemi nodes on the network would reject blocks produced by the Sequencer which did not include the force-included transaction within the permitted maximum inclusion time drift. This meant fixing the issue required a hard-fork and coordinating updating all external Hemi nodes with a patched version where the force-included invalid call to the btcTxConfirmations precompile would be handled gracefully.
After deploying the patched op-geth release (commit 05d4d8f), the Sequencer successfully processed the force-included transaction in block 4538216 and all updated nodes on the network accepted the block and continued synchronizing the chain properly.
Timeline
Time | Description |
|---|---|
2026-06-01 14:47 UTC | Transaction |
2026-06-01 14:48 UTC | Hemi sequencer starts creating block 4538216 with the force-inclusion transaction from Ethereum L1, and crashes with a runtime error, halting the chain at block 4538215. |
2026-06-01 14:51 UTC | DevOps and Protocol Engineering teams are alerted to the last Hemi mainnet block being created more than 3 minutes ago. |
2026-06-01 14:57 UTC | DevOps and Protocol Engineering teams begin investigating the cause of the issue. |
2026-06-01 15:12 UTC | Root cause is determined to be a missing |
2026-06-01 15:36 UTC | Protocol Engineering team starts investigating whether a sequencer-only fix would be sufficient or if other nodes would continue to crash. |
2026-06-01 15:40 UTC | Crash is determined to have been triggered by the creation of a force-inclusion transaction on Ethereum L1, which executed the |
2026-06-01 15:44 UTC | Decision is made to hard-fork and force an upgrade of all Hemi nodes to properly patch the bug. |
2026-06-01 15:59 UTC | Patch is implemented in op-geth, and a new version of the |
2026-06-01 16:15 UTC | Patched version of op-geth is deployed to a limited set of nodes for testing. |
2026-06-01 16:48 UTC | Patch is confirmed to be working as expected and is rolled out to remaining op-geth nodes operated by Hemi. |
2026-06-01 16:50 UTC | Sequencer creates block 4538216 with force-included problematic transaction. |
2026-06-01 16:58 UTC | Coordination with partners to update nodes to the patched version begins. |
2026-06-01 17:12 UTC | hemilabs/hemi-node is updated with the patched version of |
2026-06-01 22:42 UTC | All Hemi Mainnet P2P nodes are updated with the patched version of |
Remediation and follow-up steps
An updated release of op-geth with the appropriate fix and additional hVM precompile hardening has been released (hemilabs/op-geth:05d4d8f), and the hemilabs/hemi-node repository has been updated to use the new build.
The fix for properly handling input sanitization for the btcTxConfirmations precompile was as follows:
If you operate Mainnet and/or Testnet Hemi nodes, please see the upgrade instructions here in order to keep your node synchronized with the network. Users of the network do not need to take any action.
This outage has highlighted the need for more robust layered defenses in critical paths in transaction processing / block building, a need for more rigorous internal review of the full impact surface of known/reported bugs, and further expansion of our automated testing procedures.
To prevent a recurrence of similar issues in the future, we are:
Adding a panic-recovery boundary to all hVM precompile call execution such that an unexpected fault in a precompile (either in the precompile processing code in the execution client, or the underlying Bitcoin indexer) will always be gracefully handled regardless of the underlying cause
Reviewing the rest of our execution and consensus client codebases for general robustness improvements and defense-in-depth layering beyond the hVM precompile surface
Expanding our automated testing, including robust test cases for hVM precompile calls including L1 force-inclusion paths, and adding our in-development hVM-aware fuzzing system to our automated tests
Reviewing our internal bug review procedures and allocating more team resources to our review process to ensure any bugs that are found internally or reported have a full impact investigation performed
We understand and apologize for the disruption this incident caused. Our team has learned a lot from this outage, and are actively making these improvements to ensure improved stability moving forward and to prevent similar future issues.