{admiral} 1.0.0

[manciniedoardo]

Hi @pharmaverse/admiral,

We've been talking about releasing {admiral} v1.0.0 for a while now, and myself and @bms63 feel like the next release following v0.12.0 (i.e. the December 2023 release) would be a good time to do it. Our reasoning is as follows:

• Largely, we are happy with the current contents of the package. Even if some functions here and there may overlap in places, that's alright. We feel like continually remoulding the package to completely remove these is not only not a good use of our time, but also comes at the detriment of our current users (who see breaking changes disrupting their code) and our future users, who now lose a possible tool to derive a variable/parameter. In general, we are happy about how the package is balancing between the two paradigms of "one function per variable/parameter" and "one function per whole ADaM"; we don't want to upset this equilibrium too much. With all of this in mind, after release 0.12.0 our idea is to draw a line under the function rework effort that has been going on, gathering any remaining small bits and pieces relating to standardising the arguments of our functions into a new "argument standardisation" effort but tabling any work relating to combining/drastically changing actual {admiral} functions.

• In connection to the above, some of the reservations around 1.0 were related to the associated notions of package stability that naturally come with this milestone. Having given it some thought, we came to the conclusion that while increased stability is something that we think is naturally expected of 1.0, perhaps a better term to use for 1.0 is package maturity. In essence, we feel {admiral} is ready to to be used by ADaM programmers, and as such we as a dev team should start pivoting from a mindset focused on new functionality to one centered around maintaining our package.

• This doesn't mean that we should never introduce breaking changes anymore - there will be times in the future where we wish to make changes to the {admiral}. Instead, our idea is to move to a twice-yearly release schedule (excluding any patches) so that any changes happen slower and thus users have more time to adapt. For instance, our deprecation cycle (warning/error/removed) would take 18 months instead of 9 months.

• Additionally, we believe releasing 1.0 will increase the credibility of our package for external companies/users that may not be involved with it yet. These groups may now see immediately from the version number that the package is indeed mature enough to satisfy their ADaM needs. I know that I myself am more likely to use a package that is labelled as version 1.x rather than 0.x.

• Lastly, a relatively minor thing is that releasing 1.0 at the end of the year enables us to start 2024 with a new, fresh mindset focusing on the points I mentioned above.

Having said all of the above, we want to know how you feel about 1.0. What are your thoughts, concerns, ideas around the subject? What do you think we should be doing while moving towards this milestone, and what do you think we should stop doing?

[zdz2101]

One of my mentor's once said: "Don't let perfect be the enemy of the good" and I think that is especially applicable here. The reality is our work will always be a moving train, and we'll always be able to find something to justify changes/modifications.

I agree we're certainly approaching a point where the package is usable and at the end of the day, I think we have to remind ourselves that we were never meant to be the end all one stop shop either:

With that said, I'm all for a Q4 1.0.0 release, the one thing I recommend afterwards is it'll become imperative we have a very robust team-user feedback mechanism:

On the user side

• Perhaps we can add 2 buttons:
  1. a slack button that reroutes to our slack instead of parsing through community tab
  2. a mail button that reroutes to some general email account that pings core developers

On the team side
Although we ideally want users individually coming to us with issues/providing feedback, I think it is just as important we engage with users to help trigger said feedback mechanism, ideas here include:

1. Pinning a slack post, that reposts bi-weekly or monthly gathering feedback using a google form/our issue template? Which may translate into work for us to do
2. More blog posts or linkedin posts to demo functions, we can even recycle our function examples
   2b) Maybe our hello admiral meeting turns into more of a user-feedback meeting
3. Perhaps some sort of weekly challenge, I imagine a much much smaller scale version of rfordatascience/tidytuesday

I felt like there was a sense of excitement from our hackathon earlier this year, wondering how we can tap into that

[bms63]

We can add the slack icon next to the github icon.

All for more blog posts!! We have a loose schedule but nothing is critical, but we can look into being more targeted in our content and releases

The weekly challenges/hackathons/presentations...my observations of this is that it is an incredible amount of work (having done and participated in these types of events). I'm weary of doing anything that requires a lot of continual effort on the part of the devs. Especially as we all reduce our time on this project.

[bundfussr]

We could also consider reviewing code of studies which used (or are using) admiral. I would not expect much feedback until there are some experienced users in the user community.

[manciniedoardo]

Great idea @bundfussr, especially at Roche we can start doing this soon to see any gaps in the package.

[bms63]

I'm eager to get to a 1.0 quickly and demonstrate a mature package to our community.

I think our reputations is that we are always breaking things, re-working things. Not always a bad thing as it demonstrates our desire to meet the needs of the community, CDISC and company standards. However, users at GSK have expressed frustration with how their code breaks in every release, which can be frustrating when using admiral for a study.

Once we reach 1.0 I want to really try and either not break things and if we do it takes quite a while for it to impact a user. Personally, I go back and forth on the release cycle. I'm learning towards a 12 month cycle instead of 6 months.

My biggest concern is our programming strategy and our manifesto. I think we should spend some time this cycle reviewing, simplifying and updating these documents.

[bundfussr]

I'm eager to get to a 1.0 quickly and demonstrate a mature package to our community.

admiral is well tested but so far not used in real studies. In my opinion we need feedback from real studies to "complete" admiral. And I would expect that the feedback results in changes to admiral. From what is stated here and on the admiral main page, I (as a user) would not spend time on feedback regarding improvements but just report bugs. I think we have to be very careful in how we are communicating "maturity". Otherwise we may prevent users from giving feedback and contributing.

I think our reputations is that we are always breaking things, re-working things. Not always a bad thing as it demonstrates our desire to meet the needs of the community, CDISC and company standards. However, users at GSK have expressed frustration with how their code breaks in every release, which can be frustrating when using admiral for a study.
They could just stay at the old admiral version. We could also prolong our deprecation periods, e.g., the first 6 months a note is issued, the next 6 months a warning, and then an error for the next six months.

Once we reach 1.0 I want to really try and either not break things and if we do it takes quite a while for it to impact a user. Personally, I go back and forth on the release cycle. I'm learning towards a 12 month cycle instead of 6 months.

I think regarding deprecation 12 or 6 months are OK. For new features it is quite long. Why should users contribute if they have to wait a year until the feature is available? Maybe we should decouple deprecation and features.

My biggest concern is our programming strategy and our manifesto. I think we should spend some time this cycle reviewing, simplifying and updating these documents.

I agree that programming strategy and manifesto need a revision but I would revise the functions and the reference first because they are more important for the users.

[bms63]

I think de-coupling the deprecation and features for releases is an interesting idea. For example, we release new features (1.1.0) and bug fixes (1.1.1) as fast as possible while any deprecation is done over the major releases (2.0). I'm already seeing a problem here with this, but just a thought...

As far as programming strategy - I feel like if we are going to overhaul anything, then we have to get the strategy right first before we do approach another overhaul.

[manciniedoardo]

I think 12monthly releases with nothing else around them might be too slow and give the idea that the package is not being properly maintained. I agree that we need a review of our programming strategy ahead of 1.0.

[rossfarrugia]

Fully agree with the sentiment of all the above - and from a selfish perspective 2024 is going to see a mass of admiral adoption at Roche, and frankly the constant changing our package extensions and admiralroche to keep in line with ever changing minor tweaks of admiral core functions has become really challenging to keep on top of. I love @zdz2101 point on "Don't let perfect be the enemy of the good" - I was thinking similar here, but I'd say "Don't let perfect be the enemy of the great" - as I see admiral already is great, so let's not risk losing new users and companies adopting whilst we strive for "even greater". We should ruthlessly prioritize the Q4 clean-up items and be willing to drop the nice-to-haves. Good team discussion!

[bundfussr]

I think if we can release a "stable" (or "mature") admiral version in Q4 2023 depends on what we mean by that term. I see three possible meanings:

• admiral is stable in the sense that there are no (major) bugs, i.e., all functions produce the expected results. Regarding this sort of stability we are pretty good and I would consider admiral stable in that sense.
• admiral is stable in the sense that it is a well-designed tool with a clear structure and documentation and a consistent behavior. We are moving into this direction but we have not reached this state.
  A user request a few weeks ago pointed this out again. The user wanted to derive a new parameter as sum. She used derive_summary_records() to derive AVAL and it produced the desired result. For the new records ADT should be the maximum date of all the contributing records. I recommended to use derive_param_exposure() because it is exactly doing this, it's derive_summary_records() plus the minimum and maximum date. Of course, she did not consider derive_param_exposure() because she was working on a tumor results dataset. So obviously the function is not named well. Unfortunately, she could not use the function because in derive_summary_records() a condition can be specified to select the input records but for derive_param_exposure() a single parameter code needs to be specified.So the usage of the functions is inconsistent.
  Reaching this state this year is definitely challenging but we can try it. I think we need to consider admiral as a whole. I have created a document for this: admiral Functions Overview
• admiral is stable in the sense that it supports efficient creation of ADaM dataset for most submissions in reality. I have no idea if we are at that state. Most likely we are not because as far as I know there is no submission which has used admiral. I would not trust a package which claims that.

What are you aiming for?

[manciniedoardo]

Thanks for putting together the thoughts and document, @bundfussr. I'm just back from business travel and so am only sharing my thoughts on both just now. Admittedly I also only glanced at the document you shared, and will review more carefully this week.

admiral definitions of maturity

• Totally agree with @rossfarrugia - I think as far as admiral "maturity" goes, definition 1 is a must have, and definition 2 is what we want to aim for long-term.
• I say "aim for definition 2" because the reality of ADaM is that there will always be a user that will point out gaps in our tool. We can of course recognise legitimate cases for improvements of our portfolio, but I fear that eternally trying to enhance what we provide will turn us into Sisyphus, eternally rolling a boulder up a hill, only for it to fall back down right when we think we are done.
• This is also why I really don't think we should even aim for definition 3 - ever.

Some comments on Stefan's document

• Naming conventions: I like the thought behind this, but more for the long-term than the short term. Right now, what we provide is fine, and is being utilised by users with some successes, and I don't believe we should again overhaul the names of lots of functions (thus breaking lots of code) in favour of a better naming convention. I agree with @bms63 that some users may even like to older naming convention more. I strongly believe we should bookmark something like this for an admiral 2.0 release, e.g. at the end of next year, so that users already on admiral 1.x could consciously decide to upversion or not. admiral is too established to do such a big overhaul right now.
  
• use_call: I also like the thought behind this, my major concern being how exactly to present this to the users. I would want to somehow label it as an "advanced" use of admiral, so that beginner users do not have to worry about this. Introducing another layer of abstraction like so will alienate our more beginner users, and also goes against the modularity and readable code manifestos under which we work. If we were to implement this, I would suggest to make an "Advanced admiral" vignette where we describe how far one could take admiral, with the proviso that you do not have to do this. In the same vein, I strongly believe we should not insert this sort of code in the existing templates for ADaM datasets.
• Historic derivations: agree we can prune over time, but this is a nice to have - it's ok for the package functionality to have some overlap and this is easier for the users to work with than constant deprecations.

[bundfussr]

Per the docx you shared, here are some of my thoughts:

1. The direction/where we can go is highlighted very well especially with table that follows the Naming Conventions header, this looks and feels like the bulk of the work we can do
   a. My concern is if we rehaul this many things before our major release, it might be a little too disruptive, but I'll let others chime in here as well.

I tried to identify as many things as possible which need to be updated. After the 1.0 release we can not do such a revision for a long time.

b. When to introduce tidyselect compatilbility seems to be a tricky topic here.
2. Sounds like the special purpose derivations are good as is so minimal work needs to be done there.

To be honest, I did not review them. I focused on the main functions because they have the most impact. If time allows, we should also review the special purpose derivations.

3. The historic derivations look like perfect candidates for lifecycle::superseded.
   a. We can push users in the direction we want to steer them to, but I don't see any reason we should remove these.

I like the idea of marking them as superseded.

4. I really like the idea of the call store, wondering if we really need to have it as a separate package though. It almost seems like a mini-version of use_ad_template()

I would implement the call store as a separate package because:

• The functionality is different from admiral. You can use admiral without using callstore. You could also use callstore without using admiral, e.g., for SDTMs or summaries.
• Adding the call store functionality to admiral would conflict with our goal of having a stable admiral version and longer release cycles.

[bundfussr]

I'd love to spend time revisiting our programming design and strategy and slowly bend admiral in that direction over the coming years.

My biggest fear is that we continually break and disrupt admiral and folks become less and less interested in the package.

This looks contradictory to me. If we "slowly bend admiral over the coming years", we continue to break and disrupt admiral in every minor release (and we do not get rid of all the inconsistencies).

My idea was to make the major breaks now such that we achieve a structure which requires no major breaks in the next years. Then in a few years we can do another revision for admiral 2.0.

[bundfussr]

• I strongly believe we should bookmark something like this for an admiral 2.0 release, e.g. at the end of next year, so that users already on admiral 1.x could consciously decide to upversion or not. admiral is too established to do such a big overhaul right now.

Do you want to do a major release every year? I would have expected admiral 2.0 in two or three years.

If we think that we should do these changes, why should we wait another year?
Users who like the old names can just use admiral 0.12.0.
Why should new users learn the old names if we think they are not good?
Why should we call admiral 1.0 "stable" or "mature" if we plan to break most of it in a year? I think this could upset our users.

[manciniedoardo]

Do you want to do a major release every year? I would have expected admiral 2.0 in two or three years.

It's just an idea/suggestion, no real planning behind it yet.

If we think that we should do these changes, why should we wait another year? Users who like the old names can just use admiral 0.12.0. Why should new users learn the old names if we think they are not good?

While I think that in a vacuum, the new names you suggested are more standardised, I do not agree that the old names are "not good". I think they are plenty good enough for a 1.0 release, and I wouldn't use lots of deprecation cycles right now to overhaul them again. We need to start thinking about winding down our breaking changes.

Why should we call admiral 1.0 "stable" or "mature" if we plan to break most of it in a year? I think this could upset our users.

A lot of it has to do with the optics around what a 1.0 release means (mature vs development version). Once we have released a 1.0 version, with the associated messaging that this can be used and relied upon, then this can function as our "base" version for some time. Users who adopt this version can be comforted in knowing that the version is mature enough that most things shouldn't break after small updates. Accordingly, in the future we can then release a 2.0 version where we again make some deep changes, but we will already have a 1.0 version that users can stick to if they are in the middle of a filing or too committed to it. It's just not the same if we have no "base" version to fall back on. Clear messaging around this should mitigate users being upset, but I expect some portion of users will be upset no matter what we do.

[bundfussr]

• use_call: I also like the thought behind this, my major concern being how exactly to present this to the users. I would want to somehow label it as an "advanced" use of admiral, so that beginner users do not have to worry about this.

I completely agree. I would implement it as a separate package. Then it is obvious that admiral can be used without use_call(). And it would avoid that admiral gets to big.

Introducing another layer of abstraction like so will alienate our more beginner users, and also goes against the modularity and readable code manifestos under which we work.

I do not think that the call store concept goes against "the modularity and readable code manifestos". On the contrary we would regain modularity and readability. When we started the ADaM scripts looked like a use_call() call. For example

adcm <- cm %>% 
  derive_var_astdtm_adcm() %>%
  derive_var_astdt() %>%
  derive_var_astdy() 

This did not work because it would have resulted in thousands of functions with a lot of duplication of code. It was also not readable because nobody can remember all these functions and what was actually done in a derivation was hidden in the function.

Therefore we created more general functions. This solved the duplication issue partly but it made the ADaM scripts very long. The scripts are a big chunk of code, sometimes more than thousand lines. This is hard to read and not modular. So at the moment we are not really modular.

The call store concept solves these issues.

• Derivation can be easily added or removed by adding their ids.
• Each derivation is maintained separately, i.e., it is modular.
• The programmer and reviewer just need to know the few admiral functions.
• The complete information of what is done is available via the generated script.

It is another layer but the concept is quite simple. So I would expect that it does not require much time to learn it. In return creating and maintaining the ADaM scripts should be much more efficient.

If we were to implement this, I would suggest to make an "Advanced admiral" vignette where we describe how far one could take admiral, with the proviso that you do not have to do this. In the same vein, I strongly believe we should not insert this sort of code in the existing templates for ADaM datasets.

I agree, we should not change the existing templates. In admiralroche I would generate the templates by use_call() and provide the script with the use_call() calls in addition. Then the users can decide which way they want to go (and we have less work maintaining the templates). I am not sure if we should do this in admiral core as well. I would consider the admiral core script more as examples than as code which is really used in a study.

[manciniedoardo]

Summary of discussion

General

• We'll move forward with 1.0, releasing this December.
• We accept that the package can never be perfect, but we won't let that stop us from releasing as good a package as possible. Equally, we won't endlessly strive for perfection.
• To that end, we will continue function rework efforts for another month, but focusing on alignment of arguments/documentation rather than creations of new functions or deprecations. Focus should be on cleanup rather than nice to haves. At the start of November, function rework efforts will cease as we try and wrap up any loose ends ahead of release (e.g. improve our programming strategy vignette).
• Focus for 1.0 will be on releasing a "mature" package that users can adopt at the start of a study and use all the way across the study's lifespan, with most of their ADaM needs fulfilled.
• Even in the 1.0 release, we will try not to break any functionality, but just enhance.
• Generally we are happy with the current mix of variable-specific functions and general functions. We don't want to upset this balance too much either way.
• In this release, and especially in the ones after, we will phase out deprecations in favour of superseding functionality in order to minimise impact to users.

Communication around 1.0

• It will be key to communicate that although we are releasing 1.0 this does not mean that {admiral} development stops. If we do not do this, users may be disincentivised to provide their feedback.
• We will all need to make some noise about this release on our various channels as it's a big deal!

Looking past 1.0:

• It will be key to actively and proactively collect user feedback once more users start to onboard onto 1.0 - I will actively do this on the Roche side as we will have mass adoption in 2024. For those at other companies, it would be nice for you to try and to the same.
• Future release cycle still TBC.
• I will make an "admiral long-term improvements" discussion on our github to collect any ideas that devs or users may have about this topic. We can collect such thoughts here so as not to distract focus from the 1.0 release.

I don't know about you all but I'm super excited for the months ahead!