Enable storing the callback outputs in the persistence storage

[petar-qb]

Fixes #2678

This PR:

• Solves Component properties set through dynamic callbacks cannot be persisted #2678
• Solves the question from the Dash Plotly forum.

TLDR:
This PR adds a callback's property enable_persistence: bool (by default False). If it's True, callback output value will be written within the browser's persistence storage for its dcc components that has persistence=True set.

---

For the following code (also posted in the Dash Plotly forum question that's linked above):

from dash import Dash, dcc, html, Input, Output

app = Dash(__name__)

app.layout = html.Div([
    html.Button("Select All", id="select_all_button"),
    dcc.Checklist(
        id='checkbox',
        options=[1, 2, 3],
        value=[],
        persistence=True,
        persistence_type='session'
    ),
])


@app.callback(
    Output('checkbox', 'value'),
    Input('select_all_button', 'n_clicks'),
    prevent_initial_call=True,
    # Uncomment the following configuration:
    # enable_persistence=True
)
def select_all_options(n_clicks):
    return [1, 2, 3]


if __name__ == '__main__':
    app.run_server(debug=True)

There are screen recordings of how it works from the plotly/dash::dev branch vs the petar-qb/dash::feature/callback_enable_persistence branch:

How it works from the plotly/dash::dev (it works exactly the same from the feature branch too if the callback enable_persistence is not set or is set to False):

Screen.Recording.2025-01-30.at.11.07.24.mov

How it works from the petar-qb/dash::feature/callback_enable_persistence when enable_persistence=True is added :

Screen.Recording.2025-01-30.at.11.09.25.mov

---

There's a sketch that represents the current behaviour:

There's a sketch that represents the new behaviour:

---

Contributor Checklist

• I have broken down my PR scope into the following TODO tasks

  • Add enable_persistence into the callback and clientside_callback signature (by default it's False).
  • Call recordUiEdit even on callback response.
  • Within executedCallback:
    • Do not prune persistence if enable_persistence=True,
    • Do not apply persistence if enable_persistence=True,
    • Call updateProps that will call recordUiEdit only if enable_persistence=True
  • Enable that the recordUiEdit can recursively record edits for children. This enables that setting the persistence from the callback output works even if nested object is returned.

• I have run the tests locally and they passed. (refer to testing section in contributing)

• I have added tests, or extended existing tests, to cover any new features or bugs fixed in this PR

optionals

• I have added entry in the CHANGELOG.md
• If this PR needs a follow-up in dash docs, community thread, I have mentioned the relevant URLS as follows
  • this GitHub #PR number updates the dash docs
  • here is the show and tell thread in Plotly Dash community

[petar-qb]

Every time I make changes to the dash-rendered code locally, I have to run renderer build local before testing.
Is there a simpler or more efficient way to test the changes from the dash example app that uses the new dash-rendered local version?

[petar-qb]

Is "enable_persistence" the best name for this? Would something like "set_persistence" be a better fit? What do you think?

[petar-qb]

I see this more as a bug fix rather than a new feature. If you agree, we should then default the enable_persistence to True and explicitly set it to False for the callbacks like:

dash.py -> LN: 2237

            @self.callback(
                Output(_ID_CONTENT, "children"),
                Output(_ID_STORE, "data"),
                inputs=inputs,
                prevent_initial_call=True,
                enable_persistence=False
            )
            def update(pathname_, search_, **states):

What are your thoughts?

[ndrezn]

I would regard this as a bug fix. That being said I don't see why it needs to be enabled/disabled as a flag; is there an argument against having this be the default behaviour?

[petar-qb]

Hey @ndrezn 👋

The current (plotly:dev) behaviour of persistence varies depending on whether the returned server value is a nested object or a single property.

Current Code:

• If a callback returns a nested object (e.g., Output("div_id", "children")), persistence applies only if the returned server value matches the persisted originalVal. Otherwise, the returned value is set on the UI component, and persistence is pruned. This is how an internal _dash_pages callback works too. 👍
• If a callback directly returns a persisted property (e.g., Output("dropdown_id", "value")), the UI component is updated, and persistence is always (unquestionably) pruned which leads to unexpected behaviour. 👎

New Code:

• By default (enable_persistence=False), behaviour remains the same as the current code. 👍
• If enable_persistence=True:
  • The returned server value is always set on the UI component. 👍
  • Persistence storage is updated accordingly, ensuring consistency. 👍

This means there’s no change for existing users unless they explicitly opt in by setting enable_persistence=True.

Why keep the flag?

If we remove enable_persistence and default to treating all user's callbacks as enable_persistence=True:

• The buggy case (where a single property return forces persistence pruning) would be fixed, which is the expected behaviour. 👍
• However, there’s a potential breaking change for users who intentionally return a persisted property inside a nested object (e.g., Output("div_id", "children") containing e.g. a dropdown. Currently, this applies persistence to the UI component, but with the new approach, the persistence storage would be overwritten instead.❓

Thus, keeping enable_persistence allows users to opt into the fix while avoiding unintended disruptions. However, if the last use-case (marked with "❓") is not a breaking change (unintended disruption), and if you think that there's no users who rely on the fact that returning nested objects from the custom callback can actually apply the persistence (if the returned value of the nested object matches the originalVal) then we can completely remove enable_persistence flag and declare this PR as a bugfix.

Would love to hear your thoughts!

P.S. I'll update the sketch from the PR description with the things described in this comment.

Edit: Sketches from the PR description are updated.

[antonymilne]

Awesome PR @petar-qb! I know how much work has gone into getting to the bottom of this over the last few months!

Just for context @ndrezn: we had a phone call with @T4rk1n and @gvwilson last year to discuss this. It was actually @gvwilson's idea to release this with a flag, even if it's regarded as a bug fix, to ensure that any apps that rely on the current behaviour could revert to it. He said that you guys would be able to test the changes against a suite of representative Dash apps that would tell us whether it's likely to break anyone's app or not.

Personally I agree with you and would just consider this a bug fix and not put a flag in at all for it. The cases where someone was actually relying on the old behaviour seem so edge to me that I can't imagine anyway is deliberately using that behaviour.

Edit: this is not quite correct - we might need the flag after all. See next comment...

[antonymilne]

I just had a good chat with @petar-qb who explained things some more (every time I discuss this I get confused, there's so many different cases to consider, sometimes a face to face chat is just much easier 😅 And Petar has really spent A LOT of time figuring all of them out). What I said above about not exposing the flag is not correct due to the special status of the inbuilt Dash pages update callback that runs when you change page.

Basically, AIUI, it doesn't make any sense for that Dash pages callback to have enable_persistence=True, only for it to have enable_persistence=False. For all user-defined callbacks it makes sense to have enable_persistence=True (unless for some reason someone is relying on the current behaviour, but that feels unlikely to me, as above).

So to ensure that the Dash pages update callback and user-defined ones work correctly there's basically two options:

• a new flag enable_persistence that is True by default but set to False for the Dash pages callback. This would also allow user-defined callbacks that for some reason rely on the current behaviour to maintain it
• some other bit of logic in Dash that discriminates between the Dash pages update callback and user-defined ones so that the Dash pages update one behave as if enable_persistence=False and all others behave as if it's True

Did I get that right @petar-qb?

[gvwilson]

@T4rk1n please have a look at this one and let us know if it can go into Dash 3.0 without disrupting things or whether we should target 3.1. thanks (and thanks to @antonymilne and @petar-qb for their hard work) - @gvwilson

[petar-qb]

Did I get that right @petar-qb?

You're absolutely right! Both of your callback configuration API suggestions work for me, and I'm curious to hear what the Dash team thinks.

P.S. I've updated the sketches of the current and new flowchart to better illustrate how this behaves.

[gvwilson]

@T4rk1n please have another look at this and give @petar-qb and @antonymilne some feedback when you can.

[T4rk1n]

I think it would be better without a flag, I always thought it was weird that persistence didn't apply for the callback return value. I seem to remember @alexcjohnson saying it was done this way for a reason but I can't remember why?

I would treat this as bugfix, get in dev for the final 2.18.3 (if any) and 3.1 release.

[T4rk1n]

This fix only for < 3.0

[alexcjohnson]

My original argument was:

if a value is set via callback, then the thing to be persisted should be the user input that led to that value, and that callback-set value will flow from the persisted user input.

But @antonymilne shows some good use cases where this doesn't work, and I'm not finding cases where allowing these to persist would cause problems, so I've come around to agreeing it should just always behave this way, no flag needed.

[antonymilne]

Thanks very much for weighing in @alexcjohnson! Now we are all agreed that we can change this behaviour everywhere and don't need a flag for users to enable/disable it.

However, this does still leave a question of how to handle the Dash pages update callback, which I believe needs to work differently from the new behaviour. As above:

Basically, AIUI, it doesn't make any sense for that Dash pages callback to have enable_persistence=True, only for it to have enable_persistence=False. For all user-defined callbacks it makes sense to have enable_persistence=True (unless for some reason someone is relying on the current behaviour, but that feels unlikely to me, as above).

So to ensure that the Dash pages update callback and user-defined ones work correctly there's basically two options:

• a new flag enable_persistence that is True by default but set to False for the Dash pages callback. This would also allow user-defined callbacks that for some reason rely on the current behaviour to maintain it
• some other bit of logic in Dash that discriminates between the Dash pages update callback and user-defined ones so that the Dash pages update one behave as if enable_persistence=False and all others behave as if it's True

So if we don't expose a flag to users then we should go for the second solution here and put in a manual exception for the Dash pages callback. What would be the best way to do this?

[T4rk1n]

So if we don't expose a flag to users then we should go for the second solution here and put in a manual exception for the Dash pages callback. What would be the best way to do this?

You mean it should treat the page callback as the layout and don't apply persistence for that one?

The only thing that comes to mind is a heavy refactor of the page callback logic to be inside the layout route instead of the callback, but that would be way much change.

[petar-qb]

Hey team @T4rk1n @gvwilson @ndrezn @antonymilne 👋

I just pushed the 1a3173e that implements the bugfix solution without the enable_persistence=True flag.

TLDR:

• Dash internal _pages_content callback works the same as before. So, it applies persited value from the storage to the UI component (if the new server value matches the "originalVal" from the persistence storage. Otherwise, persisted value is pruned) - Again, it behaves the same as before. ✅
• NEW: Now every user's callback overwrites (no matter whether it "updates" or "recreates" the object) the value from the persistence storage. So, the value returned from a callback always takes the precedence over the value from the persistence storage. ✅

N.B. Distinguishment between the internal _pages_content and a custom user callback is handled within the executedCallback.ts with if (id === '_pages_content') {...}.

[T4rk1n]

Shouldn't this be the opposite id !== '_pages_content' for it to record apply the persistence?
In any case the comments here are now confusing, might be worth rewriting them with the actual logic of what it does here and how it get there.

[petar-qb]

Hey team 👋 @T4rk1n @gvwilson @ndrezn @antonymilne here's an update for you.

The current situation with this PR is that it's implemented like a bugfix (so without the enable_persistence flag).

There are 13 CI failing tests, and all of them fail because they contain a callback that recreates the object. Here's the part of the #3144 (comment) where I described that the "break" might happen:

Why keep the flag?

If we remove enable_persistence and default to treating all user's callbacks as enable_persistence=True:

• The buggy case (where a single property return forces persistence pruning) would be fixed, which is the expected behaviour. 👍
• However, there’s a potential breaking change for users who intentionally return a persisted property inside a nested object (e.g., Output("div_id", "children") containing e.g. a dropdown. Currently, this applies persistence to the UI component, but with the new approach, the persistence storage would be overwritten instead.❓

Thus, keeping enable_persistence allows users to opt into the fix while avoiding unintended disruptions. However, if the last use-case (marked with "❓") is not a breaking change (unintended disruption), and if you think that there's no users who rely on the fact that returning nested objects from the custom callback can actually apply the persistence (if the returned value of the nested object matches the originalVal) then we can completely remove enable_persistence flag and declare this PR as a bugfix.

---

I created another PR #3173 that solves the same issue but with an additional optional callback flag called enable_persistence. That PR behaves more like a feature, and not the bugfix. So, for that PR all tests pass and nothing is broken. (I created that PR just so we can do the quick comparison and once we decide about the approach I will push the latest code here in this PR)

---

To continue with the contribution I would need the input from you.

Do you think this should be implemented as a feature (with the optional callback argument enable_persistence) or as a bugfix (without the flag)?

Here's TLDR section about these two approaches:

1. Feature solution with the enable_persistence flag - This approach gives some flexibility to the user to decide wether the callback that recreates the object:
   1.1. gives the precedence to the value from the persistence storage or
   1.2. gives the precedence to the value from the server.
2. Bugfix solution without the flag - This approach changes (breaks) the behaviour of the callback that recreates the object. Previously (on the plotly:dev), callbacks that recreate the object always give a precedence to the persisted value, and with this change the value that's returned from the server will always take the precedence. (This is not the case for the _dash_pages internal callback which always (if possible) applies the persistence from the storage to the UI components)

P.S. Both approaches solve the bug when a callback directly "updates" the component property. The only debatable case is what happens and what to do with callbacks that "recreate" the output.

[ndrezn]

@petar-qb could you give an example of a practical type of app that wouldn't be possible to build any longer if this was implemented without a flag?

I'm always hesitant to add & maintain more API unless we know that there's a specific set of use cases. It'll be hard for most developers to grok what the functional difference is whether the flag is enabled or not and it likely would remain mostly untouched. Knowing a couple example situations where you might want to disable this behaviour would be useful to decide here.

[petar-qb]

Hey @ndrezn 👋

@petar-qb could you give an example of a practical type of app that wouldn't be possible to build any longer if this was implemented without a flag?

TLDR:
Without the flag solution means that all callbacks (except the internal _dash_pages one) behave like the enable_persistence=True is set in the solution with the flag. It means that the value that's returned from the user's callback always takes the precedence over the persisted value. However, we sometimes want that the persisted value take the precedence and here's the practical reason.

Implementation of the custom server-side tabs functionality. So, if I update checklist.value in Tab 1, then switch to Tab 2, and then return to Tab 1, I expect to see my last selection retained.

---

Here's the more detailed break-through.

There are 4 different solutions/configurations:

1. The current plotly:dev
2. Solution with the flag and enable_persistence=False in the callback
3. Solution with the flag and enable_persistence=True in the callback
4. Solution without the flag

All these four solutions share the two same principles:

1. If a user changes the component’s value manually and persistence=True, the new value is saved in the storage. ✅
2. On page refresh, Dash's _dash_pages callback tries to restore the persisted value. If returned server value doesn’t match the originalVal, persistence is cleared. ✅

There are three more ambiguous cases which results differ between these four aforementioned solutions/configurations:

Legend:
🟡 - Persisted value takes the precedence.
❌ - Returned server value takes the precedence. But the persistence storage is pruned.
✅ - Returned server value takes the precedence. But this value is written in the persistence storage.

# No	Solutions/configurations	Callback directly "updates" the component	Callback "recreates" the component with the ORIGINAL value	Callback "recreates" the component with the NEW value
# 1	plotly:dev	❌	🟡	❌
# 2	With flag enable_persistence=False	❌	🟡	❌
# 3	With flag enable_persistence=True	✅	✅	✅
# 4	Without the flag	✅	✅	✅

Now let's deep dive into the table.

1. The solution with the flag is represented in two rows # 2 and # 3.
2. # 1 is the same as # 2
3. # 3 is the same as # 4

So, everyone easily agrees that the Callback directly "updates" the component is a bug and that it should be fixed.
❌ -> ✅

For # 1 and # 2, the callback that "recreates" the component behaves exactly the same the internal _dash_pages callback. It applies the persistence from the persistence storage to the UI component, if the returned server value matches the originalVal 🟡. Otherwise, server value takes the precedence but the persistence is pruned. ❌
Dash users have to have the same ability in future for the reason that's described in the TLDR section at the beginning of this comment.

The only question is: Should Dash users have the ability to explicitly configure custom callback that "recreates" a component to ensure that the server-returned value always takes precedence over the persistence storage and saves this result in the persistence storage? So, should 🟡,❌ -> ✅ be enabled on demand?

The theoretical reasons behind this could be:
If callbacks that directly "update" a component's property take precedence and overwrite the persistence storage, why shouldn't users have the same capability for callbacks that "recreate" the component?

Here's the practical example too:
Let's say I have the same app (with two tabs) I described at the beginning of this comment. But, there's also the "Fetch the latest" button that fetches the latest data from the external source (which takes some time) and gives the latest results back. So, on demand (on "Fetch the latest" button click), the entire tab content is recreated, the newest server returned values are displayed and also persisted in the storage.

Why it matters that the newest values are persisted in the storage?? Because if I switch to another tab and then get back, I really want to see the latest changes without a need to run the long "Fetch the latest" job.

So, the only unclear case is when a callback "recreates" a component. Should users be able to explicitly set the returned server component to take precedence and overwrite the persistence storage, just like when a callback "updates" the component’s value?

Thanks for your patience and sorry for that long comment..
If you want we can jump on a call and continue this "discussion" online.