Cronos Mainnet node stopped getting new blocks after 12089664

[Aracki]

Version: v1.0.11-hardfork

One of two Cronos Mainnet nodes stopped getting new blocks after 12089664. It got:

6:17AM ERR prevote step: ProposalBlock is invalid err="wrong Block.Header.AppHash. Expected 8DF599A79B6799E160AEB9B26438A0B70428A660EFF77D12C4CE731C7A8B142E, got C7AD6CE2DFF0EC2D0558E4CDE1EBF329446A6FFB3C77EB9B90A4409CA7F34272" height=12089664 module=consensus round=0 server=node

6:17AM ERR CONSENSUS FAILURE!!! err="precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.AppHash. Expected 8DF599A79B6799E160AEB9B26438A0B70428A660EFF77D12C4CE731C7A8B142E, got C7AD6CE2DFF0EC2D0558E4CDE1EBF329446A6FFB3C77EB9B90A4409CA7F34272" module=consensus server=node stack="goroutine 140271667 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint/consensus/state.go:732 +0x4c\npanic({0x2d1d8a0, 0xc06802d060})\n\truntime/panic.go:884 +0x212\ngithub.com/tendermint/tendermint/consensus.(*State).enterPrecommit(0xc001d62000, 0xb87940, 0x0)\n\tgithub.com/tendermint/tendermint/consensus/state.go:1414 +0x1739\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc001d62000, 0xc05ccec1e0, {0xc0320e7f50, 0x28})\n\tgithub.com/tendermint/tendermint/consensus/state.go:2137 +0x13d8\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc001d62000, 0xc05ccec1e0, {0xc0320e7f50?, 0xc0235fdd00?})\n\tgithub.com/tendermint/tendermint/consensus/state.go:1963 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc001d62000, {{0x4347a60?, 0xc009288fc8?}, {0xc0320e7f50?, 0x0?}})\n\tgithub.com/tendermint/tendermint/consensus/state.go:861 +0x170\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc001d62000, 0x0)\n\tgithub.com/tendermint/tendermint/consensus/state.go:768 +0x3f9\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint/consensus/state.go:379 +0x12d\n"

[yihuang]

Can't see what happened, can you try cronosd rollback then resync with latest v1.0.x binary

[Aracki]

Already tried. Rollback is running for more than a day with no output at all.