Load entities from local file

[sdransfeld]

Hi,

Some services have huge metadata. For example Dynamics 365 has over 30MB. It would be nice to be able to load the metadata from a local file.

I have tried "reflect_output_package", but it gives me an error:
File "C:\Users\104615sedr\devel\extern\python-odata\odata\service.py", line 156, in init
self._write_reflected_types(metadata_url=self.metadata_url, package=reflect_output_package)
File "C:\Users\104615sedr\devel\extern\python-odata\odata\service.py", line 166, in _write_reflected_types
outputter.write_reflected_types()
File "C:\Users\104615sedr\devel\extern\python-odata\odata\reflector.py", line 120, in write_reflected_types
template.render_context(context)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\template.py", line 455, in render_context
runtime.render_context(self, self.callable, context, *args, **kwargs)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 916, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 943, in exec_template
callable(context, *args, **kwargs)
File "main_mako", line 41, in render_body
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 793, in include_file
callable(ctx, **kwargs)
File "entity_mako", line 28, in render_body
File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\enum.py", line 782, in getattr
raise AttributeError(name)
AttributeError: odata_collection

[eblis]

Can you show me the code how you used reflect_output_packages ?

[eblis]

I am loading an 15mb persisted file without issues.

[eblis]

Here's an example how you can use it for NorthWind service:

import requests
from datetime import datetime

# you can only import this on the second run, the first run will create the package
# import generated.northwind
from odata import ODataService
session = requests.Session()

url = 'http://services.odata.org/V4/Northwind/Northwind.svc/'
service = ODataService(
    url="http://services.odata.org/V4/Northwind/Northwind.svc/",
    session=session,
    base=generated.northwind.ReflectionBase,
    reflect_entities=True,
    reflect_output_package="generated.northwind")
OrderDetails = generated.northwind.Order_Details

[sdransfeld]

def main():
    import requests
    from datetime import datetime

    # you can only import this on the second run, the first run will create the package
    # import generated
    from odata import ODataService
    session = requests.Session()

    url = 'https://mg-prod.operations.dynamics.com/data/'
    service = ODataService(
        url=url,
        session=session,
        # base=generated.dynamics.ReflectionBase,
        reflect_entities=True,
        reflect_output_package="generated.dynamics",
        auth=get_auth())
    # MGRProjects = generated.dynamics.MGRProjects

Traceback (most recent call last):
  File "C:\Users\104615sedr\devel\py-erp\reflect.py", line 41, in <module>
    main()
  File "C:\Users\104615sedr\devel\py-erp\reflect.py", line 30, in main
    service = ODataService(
              ^^^^^^^^^^^^^
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\service.py", line 156, in __init__
    self._write_reflected_types(metadata_url=self.metadata_url, package=reflect_output_package)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\service.py", line 166, in _write_reflected_types
    outputter.write_reflected_types()
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\odata\reflector.py", line 120, in write_reflected_types
    template.render_context(context)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\template.py", line 455, in render_context
    runtime._render_context(self, self.callable_, context, *args, **kwargs)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 916, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 943, in _exec_template
    callable_(context, *args, **kwargs)
  File "main_mako", line 41, in render_body
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\site-packages\mako\runtime.py", line 793, in _include_file
    callable_(ctx, **kwargs)
  File "entity_mako", line 28, in render_body
  File "C:\Users\104615sedr\devel\services\build\python-3.11.9.amd64\Lib\enum.py", line 782, in __getattr__
    raise AttributeError(name)
AttributeError: __odata_collection__

[sdransfeld]

python-odata=0.5.4
Mako=1.3.3
python=3.11.9

[eblis]

Could you share the metadata response or is that private ?
Looks like you have enums in your metadata, there's another pull request open for that, i'm currently looking into it. Would help if I had an example metadata file with enums.

[sdransfeld]

The data is mostly standard MS Dynamics data (https://www.microsoft.com/en-us/dynamics-365)
metadata.zip

[eblis]

Can you try with python-odata version 0.6.0 ? I added support for enums. I didn't test with your metadata yet, I tested with the ODATA example service, TripPin.

[jonnerd154]

I was having the same problem...freshly minted v0.6.0 fixed it for me. 🎉 I was chasing this problem all night - very happy to see this discussion (and resolution!) this morning.

THANK YOU! This is a great library - much appreciated.

(I'm working with enums galore in SAP Business One's Service Layer)

[jonnerd154]

I've reworked my includes/etc to use the reflected-to-file classes (thank you for adding enum support @eblis!). Now I'm having a different issue and not sure how best to solve it - or maybe I'm using the library incorrectly.

Using the file-reflected Classes, the entity classes don't have the __odata_service__ attribute, so Query can't retrieve the odata service base URL to form it's complete request url.

File "****************\.venv\Lib\site-packages\odata\entity.py", line 101, in __odata_url__     
    return urljoin(cls.__odata_service__.url, cls.__odata_collection__)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'url'

I patched it changing Service to pass the odata service's base url to the reflector, and down to main.mako to add an anonymous object to the ReflectionBase class called __odata_service__ with a url attribute:

class ReflectionBase(EntityBase):
    __odata_service__ = type('',(object,),{"url": "${service_url}"})()
    pass

...This solves the problem, but feels a little hacky to have the URL static in the generated class file. Thoughts?

[eblis]

Make sure you pass base from the reflected package, check the example above with NorthWind.

I made some changes related to this so it will automatically try to import and use that package for you, it's coming in 0.6.1, but only in the next days.

[jonnerd154]

Make sure you pass base from the reflected package, check the example above with NorthWind.

That did it, thank you! Looking forward to your updates.

Also, I have a couple of changes that allow compatibility with services that expose entity attributes that are read-only. For example, SAP B1 Service Layer will return an error if Orders.RelatedTyp is transmitted as null when trying to insert a new record. The library initializes all the instance variables to be None, which causes issues, so those attributes need to be un-set before the request is sent. I need to do a little more testing, but I'll open a PR later this week.

Thanks again.

[eblis]

Can you give me the URL for SAP service, is it public?
If not, could you share the metadata file so I can have a look at those properties as well? Are they marked in any special way?

[jonnerd154]

Happy to share the metadata file. The server/service is private, but here's nothing proprietary in the metadata file. I'll email it to you.

Edit: the properties that are causing issues are contextually read-only: sometimes it's OK to write to them, but not always, and never during an insert.

I also just opened a PR #5 with two workarounds that enable my application - you clearly know more about the odata spec, all advice/edits welcome.