Two different current draw modes on ESP32-S2

[ArnieO]

I am by this reaching out to persons with ESP32 competence in our community to discuss a phenomena that we are struggling to understand. Maybe one of you have seen this before, or heard rumors that could help figuring out what is going on here?

Note:
This discussion is maybe a bit premature, as we have not yet gotten around to investigating it with firmware v2.2.22 or newer. But since we indicated the issue yesterday in #627, it could anyway be the right time.

One very interesting functionality that was introduced in v2.2.22 was the ability to disable "802.11b legacy rates" (it is by default enabled). By disabling it, my Pow-K has now not yet done a "power-down reboot" since I upgraded 3 days ago. Voltage logging of the input voltage to the board while running v2.2.21 showed significant voltage drops lasting for several seconds 2-3 times per day. Until v2.2.22 my hypothesis was that the power supply in the Kamstrup struggled (maybe linked to power being used to transmit information to the grid company). But the change of behavior after disabling 802.11b legacy rates kind of killed that hypothesis.

This is an example (probably one of the "worst", i.e. longest lasting) voltage drop.
The voltage that is logged here is the input voltage to the board, which is nominally 4,15V. This voltage is regulated down on the board to 3,3V, using a low-dropout linear voltage regulator (LDO). Logging interval i 1 second, corresponding to each red dot.

To ensure we avoid brownout, this board has a voltage supervisor that resets the board if the operating voltage drops below 3.08V. The LDO has a dropout voltage specified to 130 mV at 100 mA (and we are operating it at lower current, so in our application the dropout voltage is lower; maybe around 100 mV. This means that if the input voltage drops below around 3.2V, the board reboots.

I have not gotten around to repeating this logging after switching to v2.2.22 (disabling 802.11b legacy rate), as I want to see if the reboots are really gone before I pull the board out of the meter.

Now, this interesting change serves as background for the issue I wanted to discuss:

Two distinctly different current draw patterns on ESP32-S2 boards when running firmware v2.2.21

Measurements are done using a Nordic Power Profiler Kit II.
Current measured while powering boards from USB (5V), before the LDO. But since this is a linear regulator, and there is no consumer on 5V, the current draw we measure on 5V is virtually identical to the 3,3V current draw.

This is what we see. The plot starts right after I pressed the reset button, then there are four more reset button presses done with around 20 second interval.

This is typically what we see:

In "Mode 1" we measure an average consumption of around 45 mA after the device has stabilized:

In "Mode 2" we measure an average consumption of around 80 mA after the device has stabilized:

As you can see, the current draw pattern is distinctly different.
In Mode 2, it looks like the ESP does not fall to "auto modem sleep" current consumption (which it should).

I would say the "toggle" pattern is almost 100% repeatable: Each time I reboot it using the Reset button (which actuates the EN pin of the ESP module), it toggles to the other mode. I have seen cases where it stays in the same mode - but that is rare.

We do not know if this applies to other ESP32 variants. I have tried some random older code versions - and see the same thing. We have tried (based on rumors) rebuilding the code for 240 MHz - and see the same thing.

I will report back here when I have done the same test with firmware v2.2.22 (802.11b legacy rate disabled).

In the mean time:
Any ideas, anyone?

[dbeinder]

Your circuit is like this?
Kamstrup 4.15V out - LDO 3.3V - Supercap - ESP
My hunch is that the Kamstrup completely shuts the output off due to overcurrent, but you see a drop to ~3V at the input because the LDO starts conducting backwards. The TPS736 may have 'reverse current blockage' but even a few µA could be enough to hold up the voltage at the input, making it look like the Kamstrup is putting out something instead of just voltage leaking back across the LDO.
Is the Kamstrups overcurrent protection slow enough that you don't need a current limiter when charging the cap?

In any case, the 2nd current trace looks like the Wifi receiver is almost continuously active. Modem sleep means the CPU is always running (~20mA), just the Wifi receiver sleeps and wakes up periodically to receive beacons.
If you deactivate power saving, RX should be always on (~70mA) and the trace should look like your second one on every boot? Or are the spikes down to 20mA gone in this case?
Also, if the ESP fails to connect to an AP, it will keep scanning and the receiver will never shut off. Same for AP mode.
Maybe also try to see if you can sniff WiFi traffic directly and check if there is a difference between the two current draw modes.

[ArnieO]

Thank you very much for your thoughts - it always helps to get other views during such brainstorming! 👍

Your circuit is like this?
Kamstrup 4.15V out - LDO 3.3V - Supercap - ESP

Yes, that's it.

My hunch is that the Kamstrup completely shuts the output off due to overcurrent, but you see a drop to ~3V at the input because the LDO starts conducting backwards. The TPS736 may have 'reverse current blockage' but even a few µA could be enough to hold up the voltage at the input, making it look like the Kamstrup is putting out something instead of just voltage leaking back across the LDO.

Very good point. I know for a fact that the LDO we use leaks backwards in such a situation.

New information though: With firmware v2.2.22 and 802.11b legacy rate disabled, my board no longer reboots.
This is quite puzzling, as the only way I can see that the board is the cause for the observed voltage drops is a strong increase in current consumption, to a point where the Kamstrup power source can no longer cope.

However, my understanding is that if the Kamstrup "trips" it will be down for some minutes (has happened to me during testing...). As far as I know there is no "minor" trip level where it stops the supply just very briefly. It might be the case, I just do not know.

I could potentially install a shottky diode on the 4.15V input to prevent backwards current leak - and do a new logging to investigate further.

Is the Kamstrups overcurrent protection slow enough that you don't need a current limiter when charging the cap?

Yes, apparently. We have tried adding some current limiting, but we see that we do not have much leeway here. The board consumes significantly more than the above plots while OTA firmware updating is ongoing, actually slightly above the 75 mA limit for the Kamstrup power. So even adding a small current limitation jeopardizes OTA firmware upgrade.

In any case, the 2nd current trace looks like the Wifi receiver is almost continuously active. Modem sleep means the CPU is always running (~20mA), just the Wifi receiver sleeps and wakes up periodically to receive beacons.
If you deactivate power saving, RX should be always on (~70mA) and the trace should look like your second one on every boot? Or are the spikes down to 20mA gone in this case?

Good proposition! I don't have time for that right now but will run such a test (deactivated power saving) this evening.

Also, if the ESP fails to connect to an AP, it will keep scanning and the receiver will never shut off. Same for AP mode.
Maybe also try to see if you can sniff WiFi traffic directly and check if there is a difference between the two current draw mode

As the pattern is very systematic "every second reset", I would be surprised if this had something to do with other traffic in the network.

Here is a plot showing five consecutive restarts, illustrating it (total time span is 1 minute):

[dbeinder]

Yes, apparently. We have tried adding some current limiting, but we see that we do not have much leeway here. The board consumes significantly more than the above plots while OTA firmware updating is ongoing, actually slightly above the 75 mA limit for the Kamstrup power. So even adding a small current limitation jeopardizes OTA firmware upgrade.

The high consumption during update comes from the data transfer, the flashing step is actually relatively low power. I have run into problems when WiFi is really bad <-85dBm myself. But I am able to update on a limited energy budget by slowing the upload down using the browsers Dev Console (set to simulate a 2G/3G connection). It should also be possible to do the same thing on the ESP by tweaking the TCP stack parameters but I haven't had time to look into it yet.

As the pattern is very systematic "every second reset", I would be surprised if this had something to do with other traffic in the network.

For me this does not happen, every reboot looks the same: (My own HW with ESP32-C3 at 5V input, firmware 2.2.22, current limit bypassed)

Does the pattern also hold if you completely take power from the ESP, or just when you trigger reset?
I suspect something with your AP, maybe try a mobile hotspot for a quick comparison?

[ArnieO]

I am able to update on a limited energy budget by slowing the upload down using the browsers Dev Console (set to simulate a 2G/3G connection). It should also be possible to do the same thing on the ESP by tweaking the TCP stack parameters but I haven't had time to look into it yet.

Interesting. What do you think @gskjold - is this a potential way of both making the download/upgrade process more robust, and reduce the consumption?

Does the pattern also hold if you completely take power from the ESP, or just when you trigger reset?
I suspect something with your AP, maybe try a mobile hotspot for a quick comparison?

Good questions! Will definitively test what you propose.
For some reason, @gskjold has not been able to reproduce these "dual current draw modes" in his setup.
So my Wifi AP might be guilty in some way (a TP-Link Deco mesh system).

PS
I intended to do some testing yesterday evening, but friends passed by and dropped off of a bucket of live crabs and a lobster - so I had to take care of that in stead. 😅 (Yes - it was delicious!)
So I expect to run tests this evening.

[gskjold]

Interesting indeed, I will have a look at possibility to slow down download for bus-powered and test if possible

[ArnieO]

If you find a way, I don't think you need to restrict to bus-powered devices.
Pow-K+ is above the 75 mA "spec" while downloading, and I do not think it is critically important for anyone to minimize download time. It is more annoying if the install fails and must be repeated (we've had several feedbacks on that).

[ArnieO]

No way to get testing done tonight - too much urgent work in my other business areas. --> postponed to weekend.

[ArnieO]

More testing, proposed or inspired by comments from @dbeinder
Tests done on Pow-K+, v1.5.1 running firmware v2.2.21. Not same physical board as previous tests.

TEST 1: What happens when Power saving is turned OFF?
The drops to around 20 mA disappear, no change if I reset or power cycle:

TEST 2: Does the same happen if device is powered off/on (not just using RESET button)?
Power saving set to Minimum (default).
Answer seems to be Yes.
However, it now seems like there are more than three current draw modes (not only two).
What I get most frequently now are these, toggling mode at each power cycling:

1. Around 50 mA
2. Around 35 mA

These are from the same measurement plot, I just use the marker on two different regions:

Zoom-in on pattern 1:

Zoom-in on pattern 2:

TEST 3: Set Power saving to Maximum
I still get two distinctly different patterns, but at lower levels compared to Power saving Minimum.
And still, the mode seems to toggle at each reset.

TEST 4: Could this be "something with the AP"? Set up a mobile hotspot as AP - and see what happens.
Huh... this gives a very different pattern. I can find the IP on my mobile, and can access GUI on the mobile phone.

Power saving Minimum:

Power saving Maximum (I see no difference):

Conclusion (?)
The testing seems to support the hypothesis "something with your AP".
I have no clue what that "something" could be - or if it could be remedied somehow by any settings in the code.

My Wifi system is TP-link Deco M9 Plus: a mesh system with 3 nodes. The node closest to my workbench is not the main node (i.e. not the node connected directly to internet). This node is less than 2 meters from the amsreader device, which gives very strong signal, around -43 dBm.

Comments and opinions are welcome!