Initial implementation

[Gallaecio]

Implementation differences compared to Scrapy

While the code is heavily based on Scrapy’s, and intended to replace it in the future, it has some notable implementation differences worth pointing out for discussion:

• The API is a single form2request function that returns a Request dataclass.
• The default method, if a form does not indicate one or indicates a non-standard one, is GET, in line with the standard and browser behavior. In Scrapy, it is POST if non-empty user data is specified.
• The dialog method raises NotImplementedError.
• There is no utility for getting a FormElement through parameters (formname, formnid, formnumber, formxpath, formcss). I thought the API would be cleaner if instead we expect a FormElement as input. To compensate, I documented how to get a FormElement in all the different ways Scrapy supports using XPath. Parsel support has also been added, and the docs cover how to use Scrapy.
• clickdata and dont_click are merged into a single click keyword-only parameter. Similar to form, click accepts an HtmlElement instead of parameters (clickdata) that resolve to one. It also supports None (default, same as before), False (equivalent to dont_click=True) and True (like None, but raises a ValueError if there is no button).
• Clicked buttons can now override action and method with their corresponding, form-prefixed attributes. I found about the existence of those attributes while reviewing the implementation, and found it trivial to add support.
• Removed some Scrapy code (_value, _select_value) that does not seem necessary anymore with the minimum supported version of lxml.
• When enctype (or formenctype in a submit button) is multipart/form-data and method is POST, NotImplementedError is raised.
• Users can override enctype.
• Added support for text/plain enctype, since it was trivial, although it should rarely be used in the wild.

Let’s discuss these choices, specially naming and the not implementation of form selection and click selection parameters.

To do

• Base implementation.
• Docs.
• Remove the AI mention from the description.
• Support formaction.
• Full test coverage.
• Cover all implementation differences when compared to Scrapy and the reasoning behind.
• Allow overriding enctype to handle scenarios where a different value is set via JavaScript.

[kmike]

TIL

[kmike]

Love the docs and the explanations, great work! It's something which is currently missing in Scrapy's FormRequest docs.

[kmike]

Looks great @Gallaecio!