Issue #244 Initial draft of performance testing page including summarized case study of DSPT use of APDEX

[chrisclarknhsnet]

As per discussions with Andy

[andyblundell]

This is great - will re-review tomorrow, in the meantime have suggested one extra section. Is there any code from the case study that would be worth sharing? Maybe the Jenkins config snippet? Obviously your jmeter config is bespoke.

[stefaniuk]

Would it be ok to change the location of the new images and the naming convention for the files, please? E.g. practices/images/dspt-case-study-aggregated-apdex-scores.png

[stefaniuk]

This case study reads really well. Nice work!

[chrisclarknhsnet]

No I don’t think it is. I’ll amend the markup to keep it simple Regards Chris Chris Clark Technical Lead Product Development – Cyber Security ***@***.******@***.***> 07779 289399 From: andyblundell ***@***.***> Sent: 17 February 2022 17:39 To: NHSDigital/software-engineering-quality-framework ***@***.***> Cc: CLARK, Christopher (NHS DIGITAL) ***@***.***>; Author ***@***.***> Subject: Re: [NHSDigital/software-engineering-quality-framework] Issue #244 Initial draft of performance testing page including summarized case study of DSPT use of APDEX (PR #245) @andyblundell commented on this pull request.

________________________________ In practices/performancetesting.md<#245 (comment)>:

+

+ +As such APDEX can help us answer and take action (e.g. fail the pipeline) on such fundamental questions as: + +* Are our users happy? +* Have we made performance worse? +* Would our users become unhappy when there is an increased load? + +## A case study + +For the Data Security Protection Toolkit (DSPT) we decided to use APDEX so that, prior to a fortnightly release, we could answer the question: + +> Has this release made the performance of the system worse? + +### A case study (know your audience) +<div style="float:right"> I'm not sure this is rendering how it was supposed to...? — Reply to this email directly, view it on GitHub<#245 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMQ3RH3WTGK6HISKA4LYGQTU3UXCJANCNFSM5OSHN4KA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> ************************************************************************************** ****************************** This message may contain confidential information. If you are not the intended recipient please: i) inform the sender that you have received the message in error before deleting it; and ii) do not disclose, copy or distribute information in this e-mail or take any action in relation to its content (to do so is strictly prohibited and may be unlawful). Thank you for your co-operation. NHSmail is the secure email, collaboration and directory service available for all NHS staff in England. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch visit Joining NHSmail – NHSmail Support<https://support.nhs.net/article-categories/joining-nhsmail/>

[chrisclarknhsnet]

Yep no problem, agree with that just wasn’t sure where to place them and there was one image already at the root of practices…. So I’ll move that one as well. Regards Chris Chris Clark Technical Lead Product Development – Cyber Security ***@***.******@***.***> 07779 289399 From: Dan Stefaniuk ***@***.***> Sent: 17 February 2022 16:59 To: NHSDigital/software-engineering-quality-framework ***@***.***> Cc: CLARK, Christopher (NHS DIGITAL) ***@***.***>; Author ***@***.***> Subject: Re: [NHSDigital/software-engineering-quality-framework] Issue #244 Initial draft of performance testing page including summarized case study of DSPT use of APDEX (PR #245) @stefaniuk requested changes on this pull request. Would it be ok to change the location of the new images and the naming convention for the files, please? E.g. practices/images/dspt-case-study-aggregated-apdex-scores.png This reads really well. Nice work!

________________________________ In practices/performancetesting.md<#245 (comment)>:

@@ -0,0 +1,110 @@

+# Performance Testing Can this file be renamed to performance-testing.md, please?

________________________________ In practices/performancetesting.md<#245 (comment)>:

+<img src="./dsptcasestudy-scenarios.png" style=" alt="DSPT user scenarios" />

+</div> +<div style="padding-right:30px;padding-bottom:30px;"> +Previously we had defined a list of user scenarios for the typical actions undertaken on the system which we named after Mr Men characters. We also defined for every 100 users how many (i.e. what percentage) would be likely to be performing a given Mr Man scenario +</div> +We used these scenarios to define our thread groups within JMeter and decided we would run our performance tests for 250 users at a time which would constitute a heavy load for the system. + +### A case study (what does good look like?) + +We decided that we wanted to know if the performance for a particular scenario had degraded by more than 5% compared to previous average performance. If it did we wanted to fail the pipeline so we could investigate any new pieces of code further. + +### A case study (approach) + +Although you can apply APDEX figures to JMeter it only calculates them per endpoint whereas we wanted to aggregate our APDEX figures at the Mr Man scenario level. + +We therefore wrote a Python program which would take the raw JMeter results file (a sample of shown below) and using regular expressions to group results by thread names matching a Mr Man scenario, would calculate the aggregate APDEX score per scenario. Would you be happy to include your Python code and configuration that could be helpful to others as part of the case study? — Reply to this email directly, view it on GitHub<#245 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMQ3RH65G7RR6QIIT2UV6Q3U3USMTANCNFSM5OSHN4KA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> This message originated from outside of NHSmail. Please do not click links or open attachments unless you recognise the sender and know the content is safe. ************************************************************************************** ****************************** This message may contain confidential information. If you are not the intended recipient please: i) inform the sender that you have received the message in error before deleting it; and ii) do not disclose, copy or distribute information in this e-mail or take any action in relation to its content (to do so is strictly prohibited and may be unlawful). Thank you for your co-operation. NHSmail is the secure email, collaboration and directory service available for all NHS staff in England. NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. For more information and to find out how you can switch visit Joining NHSmail – NHSmail Support<https://support.nhs.net/article-categories/joining-nhsmail/>

images have been moved