How to keep Plombery GUI updating while long process run

[rafacla]

Guys, first of all, thanks for the great script, it was just what I was looking for my Scrapy UI.
I have a silly doubt, but, how can I keep Plombery UI working while a long process run.

I'm using the example using subprocess.run but it locks the entire UI during the many seconds (some time hours) my Scrapy script runs, is there any example on how to do this properly?

Thanks,
Rafacla

[lucafaggianelli]

Hi Rafacla, good to hear that :)

I'm not very familiar with Scrapy, do you have some code snippet from your app that you can share so I can take a look?

In the meanwhile I'll try to make a Scrapy + Plombery demo!

[lucafaggianelli]

So I was trying to run Scrapy from Plombery and indeed I had your same issue. Normally plombery webpage and server don't freeze during a pipeline run, in the end plombery is made for that: running long tasks.

In this case there is a compatibility problem between Twisted (the async framework used by Scrapy) and python's builtin asyncio that is used by Plombery. Apparently (thanks StackOverflow :) there is a way to let the 2 work together but I can't make it fully working.
I'll keep looking into it.

[rafacla]

Hi Luca! Sorry for taking so long to come back here, and you already investigated the issue without my help, I'm sorry and thanks!

With the hints you gave I'll try to look for it as well, meanwhile I'm just using with the freezes, as this is small web scrap project, there's little to worry about it.

The code I'm using right now is in my repo:
https://github.com/rafacla/financas-brasil-scraper/blob/main/finance-plombery.py