Validator node getting bad blocks

[phahulin]

• Parity Ethereum version: 2.3.2
• Operating system: Linux
• Installation: binary
• Fully synchronized: yes
• Network: POA Core
• Restarted: yes

One of our POA Core validators is constantly getting bad blocks and falls out of consensus with the rest of the network.

He tried to resync his node from scratch, clear the list of known peers (by removing nodes.json file) and limit the number of peers (by setting reserved_only = true config option), but some time after the sync successfully completes he still gets bad blocks.

It might be related that we have an outdated community node (Bootnode Frankfurt) still running, but it really shouldn't be because validator shouldn't accept bad blocks.

@kPatch could you please try this one more time:

1. attach the logs containing the error message from last resync and +- 20 lines around it to get the context
2. double-check that you have correct chain spec file
   https://raw.githubusercontent.com/poanetwork/poa-chain-spec/core/spec.json
3. download the latest list of bootnodes
   https://raw.githubusercontent.com/poanetwork/poa-chain-spec/core/bootnodes.txt
   and remove the first one from this list
4. remove nodes.json file (/home/validator/parity_data/chains/Core/network/nodes.json)
5. drop the database ./parity --config node.toml db kill

[kPatch]

1. Double checked chain spec.json file. It's the same
2. Downloaded the latest list of bootnodes.txt, and removed first one from the list
3. Removed nodes.json
4. Dropped the database and restarted

Bad block issue still persists
Next step is to launch new node instance and delete previous one.

core-validator-irvin-20190509-1201et-parity-log-tail.txt

core-validator-irvin-20190509-1201et-parity-log-bad-block.txt