Auto-checkout: Large orders take longer to complete checkout - selected slot may be lost.

[genericjim]

The script is working well and picks the slot. It presses continue on the payment methods page. However it throws and exception on the next page which is the final confirmation place order page.

Running Windows + Chrome 81

`INFO:main:Attempting to select slot and checkout
INFO:main:Selecting slot: Tomorrow::8:00 AM - 10:00 AM
INFO:nav:Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
INFO:nav:Navigating <Waypoint ('xpath', "//*[contains(@Class, 'ufss-overview-continue-button')]") -> 'gp/buy/payselect/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/payselect/handlers/display.html'
INFO:nav:Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
WARNING:nav:Current URL 'https://www.amazon.com/gp/buy/payselect/handlers/display.html' does not match target
Handling possible redirect (timeout in 10s)
Traceback (most recent call last):
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 96, in navigate
WebDriverWait(driver, timeout).until(
File "C:\Users\Hello\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run.py", line 170, in
main_loop(driver, args)
File "run.py", line 144, in main_loop
site_config.Routes.CHECKOUT.navigate(driver)
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 100, in navigate
raise e
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 78, in navigate
self.navigate_waypoint(driver, waypoint, timeout)
File "C:\Users\Hello\Desktop\whole-foods-deliverance\nav.py", line 59, in navigate_waypoint
raise NavigationException(
nav.NavigationException: Message: Navigation to 'gp/buy/spc/handlers/display.html' failed
`

[mark-thompson]

Interesting - looks like it timed out waiting for gp/buy/handlers/display.html to load.
It waits for 10 seconds if it thinks you're being redirected, but the log seems to suggest you didn't leave the pay select page before the timeout.
Though you must have eventually, if you saw the place order button upon your return.
A simple fix here may be to increase the redirect timeout.

Would you mind running with the --debug flag next time? This will dump the page source if it encounters an error, so you can see if there's something unique about your pay select page.

[genericjim]

Mark,
Thank you for replying. It could be the timeout, time loading between clicks is so slow with the volume of users by the time you click place order most slots are gone. I will update the timeout and add the debug flag.

[mark-thompson]

I'll be pushing some changes shortly and will increase the timeout

[mark-thompson]

@b3ll0v1c Looks like you're both having the same issue here; I haven't been able to reproduce it. Try the new version of the code with the longer timeout, and run with the --debug flag if you don't mind so you can examine the source.
It may be that their servers are under a lot of load today, or there's something specific about how you have your payment preferences set up that cause this to time out.
Let me know how it goes!

[b3nw]

Same here, running with --debug now.

INFO:__main__:Found 1 slots
INFO:notify:Alerting user with message: 'Delivery slots found'
'afplay' is not recognized as an internal or external command,
operable program or batch file.
ERROR:notify:send_sms() requires a config file at 'conf.toml' with key 'twilio'
ERROR:notify:send_telegram() requires a config file at 'conf.toml' with key 'telegram'
INFO:__main__:Attempting to select slot and checkout
INFO:__main__:Selecting slot: Today::By 6:00 PM
INFO:nav:Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
INFO:nav:Navigating <Waypoint ('xpath', "//*[contains(@class, 'ufss-overview-continue-button')]") -> 'gp/buy/payselect/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/payselect/handlers/display.html'
INFO:nav:Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/spc/handlers/display.html'
INFO:nav:Navigating <Waypoint ('xpath', "//input[contains(@class, 'place-your-order-button')]") -> 'gp/buy/thankyou/handlers/display.html'>
Traceback (most recent call last):
  File ".\run.py", line 149, in <module>
    site_config.Routes.CHECKOUT.navigate(driver)
  File "C:\Users\b3nw\Documents\whole-foods\whole-foods-deliverance\nav.py", line 64, in navigate
    self.navigate_waypoint(driver, waypoint, timeout)
  File "C:\Users\b3nw\Documents\whole-foods\whole-foods-deliverance\nav.py", line 53, in navigate_waypoint
    raise NavigationException(
nav.NavigationException: Message: Navigation to 'gp/buy/thankyou/handlers/display.html' failed

[mark-thompson]

@b3nw Try pulling the most recent version with the longer timeout

[genericjim]

@mark-thompson
I had increased the timeout manually as you suggested and had a successful purchase a few minutes ago, that might be it. Thank you

[mark-thompson]

@genericjim Excellent!

This could also be an issue of order size? There's probably some kind of stock validation done per cart item upon purchase. I've generally been testing on small orders, so this may be why I'm not running into any problems.

[genericjim]

@mark-thompson
That makes a lot of sense, my order was large 50+ items.

[ghost]

@b3ll0v1c Looks like you're both having the same issue here; I haven't been able to reproduce it. Try the new version of the code with the longer timeout, and run with the --debug flag if you don't mind so you can examine the source.
It may be that their servers are under a lot of load today, or there's something specific about how you have your payment preferences set up that cause this to time out.
Let me know how it goes!

@mark-thompson Testing the latest pull. Interesting theory about the number of items in the order. It could be actually the case as I have a huge (>80 items) order! Let's see how it goes and I'll report back.

[mark-thompson]

@b3ll0v1c that's an absolute unit of an order. you may want to increase the timeout further, though I suspect the slot may be gone by the time it validates all of your items.

[ghost]

@mark-thompson This time it did not throw an exception but it failed with:

INFO:main:Found 1 slots:
Tomorrow::11:00 AM - 1:00 PM
INFO:notify:Alerting user with message: 'Delivery slots found'
ERROR:notify:send_sms() requires a config file at 'conf.toml' with key 'twilio'
ERROR:notify:send_telegram() requires a config file at 'conf.toml' with key 'telegram'
INFO:main:Attempting to select slot and checkout
INFO:main:Selecting slot: Tomorrow::11:00 AM - 1:00 PM
INFO:nav:Navigating <Route beginning at 'https://www.amazon.com/gp/buy/shipoptionselect/handlers/display.html' with 3 stops>
INFO:nav:Navigating <Waypoint ('xpath', "//*[contains(@Class, 'ufss-overview-continue-button')]") -> 'gp/buy/payselect/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/payselect/handlers/display.html'
INFO:nav:Navigating <Waypoint ('id', 'continue-top') -> 'gp/buy/spc/handlers/display.html'>
INFO:nav:Navigated to 'gp/buy/spc/handlers/display.html'
INFO:nav:Navigating <Waypoint ('xpath', "//input[contains(@Class, 'place-your-order-button')]") -> 'gp/buy/thankyou/handlers/display.html'>
WARNING:main:Checkout failed: Redirected to slot select

Trying now with a small order (25 items) with --debug and see what happens...

[btsands]

@b3ll0v1c I just got the exact same output for a slot at 7 PM today. I was watching in real time. It appeared to select the button to complete the order but the slot was taken before the transaction completed. Is this the script or is there really that much competition for open slots? This whole thing is starting to feel like it's pure luck the draw (I hope not though).

[ghost]

@mark-thompson @btsands Success! A 25 item order went through fast and smoothly. It probably takes too long for larger orders to be validated/placed and the slots get lost. I'll try again to validate that it was not pure luck. However, I have not been successful for days with placing a large order, so this strategy seems to have more logic behind than just sheer luck. One other thing: I lower the sleep time (jitter) to 10, as I think that waiting for 20 +/- 20% seconds (if remember correct how the sleep time is calculated) is too long IMHO.

[btsands]

@b3ll0v1c Good for you. I'll try the same. I currently have 49 items in cart. I'd rather have 25 than zero...lol

[mark-thompson]

@b3ll0v1c @btsands In your earlier attempts where you were redirected to slot select page after the unsuccessful checkout attempt, did it continue to wait for slots as intended?

It's sounding like smaller orders are the way to go for high (relatively speaking) demand areas.

As for the refresh delay, feel free to edit as you see fit, at your own peril. I chose a value high enough to avoid being obnoxious (and the potential account flags that could be associated with such behavior).
That said, 10 seconds is probably totally reasonable.

[ghost]

@mark-thompson Yes on your first question. I was redirected to the slot select page. I agree about not being obnoxious, however I have seen other software using value of 6 and even 4 second, so I do agree that 10 is reasonable given the circumstances.

Also, I just run into a use case where the code was not able to handle when some items becomes unavailable. I was able to capture the debug log if you are interested. Pls LMK.

[mark-thompson]

@b3ll0v1c That would be great, yes. Could you just send the source_dump_15xxxxxxx.html file that was produced on that page? There may be personal information (address, etc.) on the page, so censor as you see fit.

[btsands]

@b3ll0v1c @mark-thompson For the first time my order went through! I ended up with 29 items, and I did change the jitter value to 10. Mark thanks so, so much for sharing this.

[ghost]

@mark-thompson I didn't see any source_dump_*.html file in the directory where I started the script. Does the dump file get deleted when run.py restarts? I captured the log from the standard output. Attached. Thanks again!

source_dump_202004131036.txt

[mark-thompson]

@b3ll0v1c Ah, looks like you must have manually clicked continue before it timed out? Thus avoiding the error. When it encounters a Selenium error, it saves the source HTML to the current directory