Enforce --force-sealing option for AuRa POSDAO

[afck]

I suspect that POSDAO requires --force-sealing to work, and it might otherwise stall if there are no pending transactions. We should test that, and if it's true, we should enforce that option (error if it isn't set).
In the longer term, we can make the miner smarter, and ask the engine whether it should create a block anyway.

[varasev]

@afck Yes, you were right: force_sealing is required for POSDAO.

[afck]

Is this still needed? My reading of the miner code on master is that even without force_sealing it will create and seal a block after max_resealing_period.

Configuring no max_resealing_period and not setting force_sealing basically amounts to saying: I want the network to stop during periods with no transactions. (Independent of the engine.) Which is a valid configuration.

[varasev]

Yes, it is needed. Validators on POA networks don't use max_resealing_period.

[afck]

But the POA validators can use force_resealing… is there a reason to hard-code (on OpenEthereum master, I mean) that you're not allowed to disable force_resealing with the POSDAO-related features? Other users might want to disable it and still use those features in their network.

[varasev]

Let's clarify why we need force_sealing = true in POSDAO case: without this parameter set to true the validators may miss their turn when calling Random contract functions or calling emitInitiateChange, right?

If so, I think we need to force it to true when the randomnessContractAddress or posdaoTransition is activated through spec.json.

In the first comment you proposed to test that: I tested that on August 6 and the POSDAO network really stalls without force_sealing = true and pending transactions.

[afck]

may miss their turn

I actually think they won't:

• In requires_reseal, it looks like we'll proceed anyway, if the engine is Ready, even if force_sealing is false.
• In seal_and_import_block_internally, we already have a block. And if it was our turn to make a randomness call, that block won't be empty, so we'll proceed even without force_sealing.

So I believe everything will work just fine without force_sealing, as long as the randomness phases are based on block height (and not wall-clock time).

Except, of course, that the network stalls if there are no transactions. Which is exactly what you want if you configure no force_sealing and no max_resealing_period!

So what we should probably test is: Does it work (even if it's slower) if only one of the validators has pending transactions?

On the other hand I'm wondering how this is supposed to work with Aura (or Clique) without POSDAO:
If half the validators don't have transactions, they'll never create a block. So even if the other half keeps creating blocks, they will never be finalized??

[varasev]

Good questions. Needs to be retested 👍

[varasev]

I made a few tests with posdao-test-setup regarding force_sealing option:

When I set force_sealing = false and leave reseal_on_txs = "none", the network doesn't start (because it doesn't accept local transactions).

For force_sealing = false and reseal_on_txs = "all", the blocks are produced until block #21 then the network stalls because the tests stop making new transactions. I was waiting for a few minutes but there was no movement (reseal_max_period = 10000 didn't help for some reason).

Then I sent a transaction through MetaMask to trigger the sealing and that worked fine. Then I repeated the same a few times. Sometimes there were two or three blocks produced after the transaction, so it triggered more than one block (I guess this is an answer to the question If half the validators don't have transactions, they'll never create a block. So even if the other half keeps creating blocks, they will never be finalized??).

Now, in general, I agree that force_sealing shouldn't be forced to true, but in our POSDAO network we will set that to true anyway to have 5-seconds blocks (and thus to have 1-week staking epoch) and to avoid possible bugs (see the next comment).

[varasev]

Also, for force_sealing = false I discovered that sometimes the block is produced by the same validator as the previous block (see blocks 29 and 30 in the example below):

Block 27
  Gas used:  21000 [1 txs]
  Gas limit: 10000000
  Validator: 0x522df396Ae70a058BD69778408630FdB023389B2

stakingEpoch 0
  startBlock 0
  applyBlock 1
  endBlock   76

collectionRound 0
  REVEALS PHASE

sentReveal:
  0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78
  0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441
  0x522df396Ae70a058BD69778408630FdB023389B2

SUCCESS transactions:
  0x0b2F5E2f3cbd864eAA2c642e3769c1582361CAF6 => 0x1645AfbDC5D2560D1d3601a779D66F874f374978
    gas used: 21000/21000, gas price: 5000000000, nonce: 3


=======================================================


Block 28
  Gas used:  21000 [1 txs]
  Gas limit: 10000000
  Validator: 0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441

stakingEpoch 0
  startBlock 0
  applyBlock 1
  endBlock   76

collectionRound 0
  REVEALS PHASE

sentReveal:
  0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78
  0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441
  0x522df396Ae70a058BD69778408630FdB023389B2

SUCCESS transactions:
  0x0b2F5E2f3cbd864eAA2c642e3769c1582361CAF6 => 0x1645AfbDC5D2560D1d3601a779D66F874f374978
    gas used: 21000/21000, gas price: 5000000000, nonce: 4


=======================================================


Block 29
  Gas used:  21000 [1 txs]
  Gas limit: 10000000
  Validator: 0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78

stakingEpoch 0
  startBlock 0
  applyBlock 1
  endBlock   76

collectionRound 0
  REVEALS PHASE

sentReveal:
  0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78
  0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441
  0x522df396Ae70a058BD69778408630FdB023389B2

SUCCESS transactions:
  0x0b2F5E2f3cbd864eAA2c642e3769c1582361CAF6 => 0x1645AfbDC5D2560D1d3601a779D66F874f374978
    gas used: 21000/21000, gas price: 5000000000, nonce: 5


=======================================================


Block 30
  Gas used:  21000 [1 txs]
  Gas limit: 10000000
  Validator: 0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78

stakingEpoch 0
  startBlock 0
  applyBlock 1
  endBlock   76

collectionRound 0
  REVEALS PHASE

sentReveal:
  0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78
  0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441
  0x522df396Ae70a058BD69778408630FdB023389B2

SUCCESS transactions:
  0x0b2F5E2f3cbd864eAA2c642e3769c1582361CAF6 => 0x1645AfbDC5D2560D1d3601a779D66F874f374978
    gas used: 21000/21000, gas price: 5000000000, nonce: 6


=======================================================


Block 31
  Gas used:  0 [0 txs]
  Gas limit: 10000000
  Validator: 0x522df396Ae70a058BD69778408630FdB023389B2

stakingEpoch 0
  startBlock 0
  applyBlock 1
  endBlock   76

collectionRound 0
  REVEALS PHASE

sentReveal:
  0xbbcaA8d48289bB1ffcf9808D9AA4b1d215054C78
  0x75df42383afE6Bf5194AA8fA0e9b3D5F9e869441
  0x522df396Ae70a058BD69778408630FdB023389B2

So, it seems force_sealing = false doesn't guarantee the order of block authors (maybe this is a bug). And there are a lot of Reported benign validator misbehaviour messages:

Node 1:

2020-02-20 17:32:10  Transaction mined (hash 0x17f60f983769859bfafeec1144fac541c10522ed529cb683fe8e0616c83b0486)
2020-02-20 17:32:10  Imported #21 0xba77…eae7 (2 txs, 0.40 Mgas, 5 ms, 0.88 KiB)
2020-02-20 17:32:15  Imported #22 0x7998…3103 (1 txs, 0.06 Mgas, 0 ms, 0.68 KiB)
2020-02-20 17:32:27     6/25 peers     74 KiB chain   96 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    8 req/s,  138 µs
2020-02-20 17:32:57     6/25 peers     74 KiB chain   96 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    2 req/s,  114 µs
2020-02-20 17:33:27     6/25 peers     74 KiB chain   96 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    2 req/s,  100 µs
2020-02-20 17:33:57     6/25 peers     74 KiB chain   96 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    2 req/s,  116 µs
2020-02-20 17:34:18  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:34:18  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:34:18  Imported #23 0xc934…cdef (0 txs, 0.00 Mgas, 0 ms, 0.55 KiB)
2020-02-20 17:34:18  Transaction canceled (hash 0x4c9305f7e195bb6ec37729815dbe6561344674353cb365663b0b0f3f4e767c16)
2020-02-20 17:34:22  Imported #24 0xd851…ab44 (1 txs, 0.06 Mgas, 3 ms, 0.68 KiB)
2020-02-20 17:34:26  Imported #25 0x0edc…4622 (1 txs, 0.02 Mgas, 2 ms, 0.66 KiB)
2020-02-20 17:34:27     6/25 peers     80 KiB chain   99 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    6 req/s,  133 µs
2020-02-20 17:34:57     6/25 peers     80 KiB chain   99 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    2 req/s,  114 µs
2020-02-20 17:35:27     6/25 peers     80 KiB chain   99 KiB db  0 bytes queue    6 KiB sync  RPC:  1 conn,    2 req/s,  113 µs
2020-02-20 17:35:57     6/25 peers     80 KiB chain   99 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,  100 µs
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:35:58  Validator 0x75df…9441 could not be reported Transaction import error: Transaction error (Already imported)
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:35:58  Transaction mined (hash 0xc55fc1c795869e8967b78f41d73a5727fbce3b6e2e3dbb3d8975641809ced997)
2020-02-20 17:35:58  Imported #26 0xfa44…6614 (2 txs, 0.04 Mgas, 25 ms, 0.77 KiB)
2020-02-20 17:35:58  Validator 0x522d…89b2 could not be reported Transaction import error: Transaction error (Already imported)
2020-02-20 17:35:58  Reorg to #26 0x9b38…42bd (0xfa44…6614 #25 0x0edc…4622 )
2020-02-20 17:36:06  Transaction canceled (hash 0x19c794bed2cd9d200011ebf5969d1dd0654b4c588ba30f6d18ed98281b08dfd9)
2020-02-20 17:36:27     6/25 peers     82 KiB chain   95 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,  158 µs
2020-02-20 17:36:57     6/25 peers     82 KiB chain   95 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,  147 µs
2020-02-20 17:37:05  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:37:05  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:37:05  Imported #27 0x581d…bc5c (1 txs, 0.02 Mgas, 26 ms, 0.66 KiB) + another 1 block(s) containing 2 tx(s)
2020-02-20 17:37:18  Transaction canceled (hash 0x7b946bfc3da16f90083a568867d78902192b6ebb0f5fb2f7cab868c3dc8265b6)
2020-02-20 17:37:27     6/25 peers     87 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   74 µs
2020-02-20 17:37:34  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:37:34  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:37:34  Imported #28 0x0a57…c703 (1 txs, 0.02 Mgas, 22 ms, 0.66 KiB)
2020-02-20 17:37:42  Transaction canceled (hash 0xcfb5aa2d93fbfea288fbe5f2d91ebdb140730f5a35d7ef05c96fc2acebc54325)
2020-02-20 17:37:57     6/25 peers     89 KiB chain   99 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   80 µs
2020-02-20 17:38:27     6/25 peers     89 KiB chain   99 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   73 µs
2020-02-20 17:38:43  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:38:43  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:38:43  Imported #29 0xa890…76c4 (1 txs, 0.02 Mgas, 1 ms, 0.66 KiB)
2020-02-20 17:38:43  Transaction canceled (hash 0x3bcf1b823204dbf51804e292b71e1b1d854a0cac709fa6a67e327341f7b25758)
2020-02-20 17:39:02     6/25 peers     89 KiB chain   96 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,  117 µs
2020-02-20 17:39:06  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:39:06  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:39:06  Imported #30 0xa3d7…618d (1 txs, 0.02 Mgas, 0 ms, 0.65 KiB)
2020-02-20 17:39:06  Transaction canceled (hash 0x2805c2bf82f5d6263b0822e543db9937a292e442329cd73d1dc4b32bf0540053)
2020-02-20 17:39:14  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:39:14  Imported #31 0x6ba8…2ecc (0 txs, 0.00 Mgas, 9 ms, 0.55 KiB)
2020-02-20 17:39:18  Transaction canceled (hash 0x8683122cbcb09e80ea453ec527d0184fe7045de223eff42ade15c544431be134)
2020-02-20 17:39:32     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,   80 µs
2020-02-20 17:40:02     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,   89 µs
2020-02-20 17:40:32     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,  113 µs
2020-02-20 17:41:02     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   83 µs
2020-02-20 17:41:32     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   80 µs
2020-02-20 17:42:02     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,  103 µs
2020-02-20 17:42:32     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,   82 µs
2020-02-20 17:43:02     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    3 req/s,   82 µs
2020-02-20 17:43:32     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   90 µs
2020-02-20 17:44:02     6/25 peers     90 KiB chain   98 KiB db  0 bytes queue    7 KiB sync  RPC:  1 conn,    2 req/s,   93 µs

Node 2:

2020-02-20 17:32:10  Imported #21 0xba77…eae7 (2 txs, 0.40 Mgas, 4 ms, 0.88 KiB)
2020-02-20 17:32:15  Imported #22 0x7998…3103 (1 txs, 0.06 Mgas, 2 ms, 0.68 KiB)
2020-02-20 17:32:28     6/25 peers     35 KiB chain  396 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   81 µs
2020-02-20 17:32:58     6/25 peers     35 KiB chain  396 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   81 µs
2020-02-20 17:33:28     6/25 peers     35 KiB chain  396 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   81 µs
2020-02-20 17:33:58     6/25 peers     35 KiB chain  396 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    2 req/s,   24 µs
2020-02-20 17:34:18  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:34:18  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:34:18  Imported #23 0xc934…cdef (0 txs, 0.00 Mgas, 25 ms, 0.55 KiB)
2020-02-20 17:34:22  Transaction canceled (hash 0xe36a255373b568395aeec8f684b555578614228f2d2a0b80af22080e6d158b0e)
2020-02-20 17:34:22  Imported #24 0xd851…ab44 (1 txs, 0.06 Mgas, 0 ms, 0.68 KiB)
2020-02-20 17:34:26  Transaction mined (hash 0xfe04ccb1435cf44c641a4a1b7009e5c2bfc6fe4ad6b8152bdff5ef488ba49139)
2020-02-20 17:34:26  Imported #25 0x0edc…4622 (1 txs, 0.02 Mgas, 1 ms, 0.66 KiB)
2020-02-20 17:34:28     6/25 peers     41 KiB chain  401 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    2 req/s,   34 µs
2020-02-20 17:34:58     6/25 peers     41 KiB chain  401 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   34 µs
2020-02-20 17:35:28     6/25 peers     41 KiB chain  401 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    1 req/s,   25 µs
2020-02-20 17:35:58  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:35:58  Transaction mined (hash 0x5175ecbc3228c8926769b6ba25b18a41da0a112c7c07562e506a46fd8b217e65)
2020-02-20 17:35:58  Imported #26 0xfa44…6614 (2 txs, 0.04 Mgas, 1 ms, 0.77 KiB)
2020-02-20 17:35:58  Transaction canceled (hash 0x5c5babd12e4587928910b432743a2d18f4920ddf63415eacec79e45f7124f622)
2020-02-20 17:35:58  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:35:58  Reorg to #26 0x9b38…42bd (0xfa44…6614 #25 0x0edc…4622 )
2020-02-20 17:35:58  Transaction canceled (hash 0x5c5babd12e4587928910b432743a2d18f4920ddf63415eacec79e45f7124f622)
2020-02-20 17:35:58     6/25 peers     44 KiB chain  404 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    5 req/s,   34 µs
2020-02-20 17:36:28     6/25 peers     44 KiB chain  404 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    3 req/s,   34 µs
2020-02-20 17:36:58     6/25 peers     44 KiB chain  404 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   25 µs
2020-02-20 17:37:05  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:37:05  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:37:05  Transaction mined (hash 0xafa73f7c5122543bc4d6e437ee8c1722bad2ad475c4d55a2b6fdb305745d0374)
2020-02-20 17:37:05  Imported #27 0x581d…bc5c (1 txs, 0.02 Mgas, 26 ms, 0.66 KiB) + another 1 block(s) containing 2 tx(s)
2020-02-20 17:37:10  Transaction canceled (hash 0x6e8091342902810bdde90a33541868cf397df82c27103428fb87c878a88858eb)
2020-02-20 17:37:28     6/25 peers     49 KiB chain  469 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   24 µs
2020-02-20 17:37:34  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:37:34  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:37:34  Transaction mined (hash 0xf7cf411b8a008f117702a436a075835acb7458026c6770f81d69320916b33064)
2020-02-20 17:37:34  Imported #28 0x0a57…c703 (1 txs, 0.02 Mgas, 0 ms, 0.66 KiB)
2020-02-20 17:37:35  Transaction canceled (hash 0x2aa7f201cef80bc11bea37b0f97f08dd70bed887607d19c385692b3b42d2bd3a)
2020-02-20 17:37:58     6/25 peers     51 KiB chain  469 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,   34 µs
2020-02-20 17:38:28     6/25 peers     51 KiB chain  469 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    1 req/s,   34 µs
2020-02-20 17:38:43  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:38:43  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:38:43  Transaction mined (hash 0xe9198e8f42dff3d3b3471749bf8b4ce4b078c824ad5deaade54183e21e2ad76a)
2020-02-20 17:38:43  Imported #29 0xa890…76c4 (1 txs, 0.02 Mgas, 25 ms, 0.66 KiB)
2020-02-20 17:38:46  Transaction canceled (hash 0x07791bf1752452eb97e65a06c5e6912d16c5ca9039f6d9f634b9c1674f1f1045)
2020-02-20 17:38:58     6/25 peers     52 KiB chain  473 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   20 µs
2020-02-20 17:39:06  Reported benign validator misbehaviour 0x522d…89b2
2020-02-20 17:39:06  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:39:06  Transaction mined (hash 0x6abe7abc00db6f958148a5b6487cb07ca8acbbefc0007bd012ddaa3f7740a76c)
2020-02-20 17:39:06  Imported #30 0xa3d7…618d (1 txs, 0.02 Mgas, 27 ms, 0.65 KiB)
2020-02-20 17:39:10  Transaction canceled (hash 0xd02803ee7f4e3dc842a080af50c144f75d48f724e6872bd3ac7e2856fb87b48f)
2020-02-20 17:39:14  Imported #31 0x6ba8…2ecc (0 txs, 0.00 Mgas, 1 ms, 0.55 KiB)
2020-02-20 17:39:28     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    1 req/s,   31 µs
2020-02-20 17:39:58     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    5 req/s,   31 µs
2020-02-20 17:40:28     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:40:58     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:41:28     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:41:58     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:42:28     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:42:59     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:43:29     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:43:59     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:44:29     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs
2020-02-20 17:44:59     6/25 peers     52 KiB chain  475 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,   31 µs

Node 3:

2020-02-20 17:32:10  Imported #21 0xba77…eae7 (2 txs, 0.40 Mgas, 1 ms, 0.88 KiB)
2020-02-20 17:32:15  Imported #22 0x7998…3103 (1 txs, 0.06 Mgas, 2 ms, 0.68 KiB)
2020-02-20 17:32:29     6/25 peers     35 KiB chain  400 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:32:59     6/25 peers     35 KiB chain  400 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:33:29     6/25 peers     35 KiB chain  400 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:33:59     6/25 peers     35 KiB chain  400 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:34:18  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:34:18  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:34:18  Imported #23 0xc934…cdef (0 txs, 0.00 Mgas, 24 ms, 0.55 KiB)
2020-02-20 17:34:22  Imported #24 0xd851…ab44 (1 txs, 0.06 Mgas, 2 ms, 0.68 KiB)
2020-02-20 17:34:26  Transaction canceled (hash 0xf877366bd517bacc4d06c10558fb1958a7c0681f5a899cac27c5ec2e26523506)
2020-02-20 17:34:26  Imported #25 0x0edc…4622 (1 txs, 0.02 Mgas, 0 ms, 0.66 KiB)
2020-02-20 17:34:29     6/25 peers     40 KiB chain  403 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:34:59     6/25 peers     40 KiB chain  403 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:35:29     6/25 peers     40 KiB chain  403 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:35:58  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:35:58  Imported #26 0xfa44…6614 (2 txs, 0.04 Mgas, 25 ms, 0.77 KiB)
2020-02-20 17:35:58  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:35:58  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:35:58  Reorg to #26 0x9b38…42bd (0xfa44…6614 #25 0x0edc…4622 )
2020-02-20 17:35:59     6/25 peers     42 KiB chain  408 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:36:02  Transaction canceled (hash 0xaf26045dd5d0842c8898b1a6eb75bb4505f6baf66bd3c6705282ab4b0ab7f32b)
2020-02-20 17:36:29     6/25 peers     42 KiB chain  408 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:36:59     6/25 peers     42 KiB chain  408 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:37:05  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:37:05  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:37:05  Imported #27 0x581d…bc5c (1 txs, 0.02 Mgas, 0 ms, 0.66 KiB) + another 1 block(s) containing 2 tx(s)
2020-02-20 17:37:05  Transaction canceled (hash 0x760082fe6afafc8d141ef5d5c3a26a3e23f835377cbc868aeb207a871faba775)
2020-02-20 17:37:29     6/25 peers     46 KiB chain  470 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:37:34  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:37:34  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:37:34  Imported #28 0x0a57…c703 (1 txs, 0.02 Mgas, 23 ms, 0.66 KiB)
2020-02-20 17:37:38  Transaction canceled (hash 0x37302850de27cb208d1d893671ff684b3532ee4df289e325280b185699aa0117)
2020-02-20 17:37:59     6/25 peers     48 KiB chain  476 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:38:30     6/25 peers     48 KiB chain  476 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:38:43  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:38:43  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:38:43  Imported #29 0xa890…76c4 (1 txs, 0.02 Mgas, 25 ms, 0.66 KiB)
2020-02-20 17:38:50  Transaction canceled (hash 0x76562ffaf1fb1ff051ad5fbb0bb7dfb160152f01742dd8401f28d22d65768416)
2020-02-20 17:39:00     6/25 peers     48 KiB chain  478 KiB db  0 bytes queue    7 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:39:06  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:39:06  Reported benign validator misbehaviour 0xbbca…4c78
2020-02-20 17:39:06  Imported #30 0xa3d7…618d (1 txs, 0.02 Mgas, 26 ms, 0.65 KiB)
2020-02-20 17:39:14  Transaction canceled (hash 0xbe9d723389b0622d9426001c72f304240fa4ee39c392477a961880f07c488907)
2020-02-20 17:39:14  Reported benign validator misbehaviour 0x75df…9441
2020-02-20 17:39:14  Imported #31 0x6ba8…2ecc (0 txs, 0.00 Mgas, 0 ms, 0.55 KiB)
2020-02-20 17:39:14  Transaction canceled (hash 0x3c7d292dad3e7f164fbc8776ce431eb1cb003956983a3099a40dee11aabd3477)
2020-02-20 17:39:30     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:40:00     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:40:35     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:41:05     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:41:35     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:42:05     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:42:40     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:43:10     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:43:40     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:44:15     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:44:45     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs
2020-02-20 17:45:15     6/25 peers     48 KiB chain  477 KiB db  0 bytes queue    6 KiB sync  RPC:  0 conn,    0 req/s,  106 µs

[afck]

Thank you for the detailed results!

It looks like disabling force sealing currently prevents Aura from producing both blocks and empty step messages if there are no pending transactions. That means the network will fail to finalize any blocks whenever ½ resp. ⅓ of the validators have no pending transactions for a while. However, Aura considers skipping your turn "benign misbehavior" and reports that to the validator set contract.

The empty steps transition seems to have the same intention as disabling force_sealing: To prevent empty blocks.

I wonder whether Aura should generally ignore the --force-sealing option then (i.e. consider it always enabled).
Alternatively, we could make it illegal to run Aura without --force-sealing, and panic with an appropriate message.

But the wiki currently says:

The consensus can be run with --force-sealing which ensures that blocks are produced

That sounds like the current behavior is indeed intended: You can run it without force sealing, but then blocks won't get finalized "in a timely fashion". Maybe the wiki should just make it more clear that a lack of pending transactions, even just in some validator nodes, can prevent finalization indefinitely, and that in such a setting the validator set contract must ignore "benign misbehavior". (Or a bit better: Aura could report skipped steps only if force sealing is enabled.)

Anyway, all of this is independent of POSDAO, and all of these modifications would be very nontrivial, because of how the code is organized. Maybe let's move this issue to OpenEthereum once the Parity code has migrated.

[varasev]

Maybe let's move this issue to OpenEthereum once the Parity code has migrated.

Agree 👌