COPP with ~350 rules take more than 10 min to install at iptables

[Ezrickd]

Sonic swith running 201911 build commit f6a8678
switch has 4 tables 2 for IPv4 and 2 for IPv6 for each there are two protocols SSH & SNMP.
when config reload and check iptables -L -v -n --line-number we see the list of rules go up and down, as if the table is flushed for every rule that is installed. I can see the same on syslog file that rules are insterted to iptales again and again....

it takes the device to install all rules more than 10 Min when at this time you can access the device though the rule applyed if Drop.
untill rule table is syncd with iptbels.

i have used an old caclmgrd config file and the issue is resolved, so ~350 rules are installed instanlly.
the file was taken from old post which is updated by now..

https://github.com/Azure/sonic-buildimage/blob/master/files/image_config/caclmgrd/caclmgrd

attached show from terminal display the Iptables number going up and down, also syslog file that dispaly the insertion of the rules.

attached rule & tables from config_db.json
attched old caclmgrd file that resolve the issue.

ACL_Rule & Table .txt
iptables_line number.txt
log_iptables .txt
old caclmgrd file taken from git .txt
caclmgrd_comes with onic hash f6a8678d.txt

[anshuv-mfst]

Issue in 2019 release, causing degradation in performance.

[jleveque]

@abdosi: It appears this regression occurred with the introduction of multi-ASIC support in caclmgrd. Can you please investigate?

[abdosi]

@jleveque It is not with multi-asic changes. Issue is coming with "caclmgrd_comes with onic hash f6a8678.txt" which does not have multi-asic changes.

It is working with "old caclmgrd file taken from git .txt" which is pretty old colde where we have different thread to do processing.

[jleveque]

Thanks for the clarification, @abdosi . I believe I misinterpreted the issue description. I'll continue to investigate. I have been considering bringing back the update thread for a while to speed up updates by preventing unnecessary updates. I think it is time.

[jleveque]

@Ezrickd: I see you provided the ACL tables and rules output from the CLI, but can you please also provide the ACL configuration being used (you can simply provide the entire config_db.json file if that is acceptable for you). This will help me reproduce the issue.

[Ezrickd]

Hi Joe. No problem will do that on Sunday 👍 Have a great weekend Ezrick

…

Sent from my iPhone Sorry for his typing errors (-; On 11 Sep 2020, at 0:27, Joe LeVeque <[email protected]> wrote:  @Ezrickd<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEzrickd&data=02%7C01%7Cezrick%40nvidia.com%7Ce31a5aebd5514953a28208d855d05c79%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637353700679131241&sdata=YkOM4mxdtfXdNwn1QuEjHD%2BY1oQKNsrWqm5yyi1RXB8%3D&reserved=0>: I see you provided the ACL tables and rules output from the CLI, but can you please also provide the ACL configuration being used (you can simply provide the entire config_db.json file if that is acceptable for you). This will help me reproduce the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fsonic-buildimage%2Fissues%2F5275%23issuecomment-690741470&data=02%7C01%7Cezrick%40nvidia.com%7Ce31a5aebd5514953a28208d855d05c79%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637353700679141236&sdata=LlzEnRlbUVyOFZ5s4nhb1j0CnV0XhGOu1zS6yR5Ap08%3D&reserved=0>, or unsubscribe<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMGJOK7BOWU2C5HPXRCAZMLSFFAETANCNFSM4QPSUFBQ&data=02%7C01%7Cezrick%40nvidia.com%7Ce31a5aebd5514953a28208d855d05c79%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637353700679151229&sdata=R%2BRJr1Won2zTt3oa9ejRNuH6vI4LX9zaxHKeP5dStbA%3D&reserved=0>.

[qiluo-msft]

@Ezrickd Is this issue a regression or a history one?
If regression, do you have a good version under comparision?

[abdosi]

@Ezrickd can you run your test again on latest 201911 and post the result.

cc @jleveque @liat-grozovik

[jleveque]

@Ezrickd: I will also be merging #5312 soon. I would also like you to take another measurement after those changes merge as well for comparison.