Rust Extensions issue with setup.py

[AndrewJSchoen]

Sorry in advance if this belongs in another repo.

I have been looking for ways to get some rust-based code to play nicely with a bunch of python-based ROS 2 (Foxy) code. Given that the last I checked, official ROS2 bindings were not quite working for rust, I thought I might be able to wrap up the rust code (which is really standalone) I had into a single module, and then create bindings for it with PyO3. That way, after building, I could simply import the rust library within python and use ROS 2 at that point.

I created a super-simple test package.
I have gotten it to work when cd-ing into the package and running python3 setup.py bdist_wheel, and it does indeed generate the required .so files, which work correctly from within python.

However, when I execute colcon build --symlink-install --packages-select rust_test, it fails with the following error:

Starting >>> rust_test
--- stderr: rust_test                   
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/colcon_ros/task/ament_python/build.py", line 51, in build
    setup_py_data = get_setup_data(self.context.pkg, env)
  File "/usr/local/lib/python3.9/site-packages/colcon_core/task/python/__init__.py", line 20, in get_setup_data
    return dict(pkg.metadata[key](env))
  File "/usr/local/lib/python3.9/site-packages/colcon_ros/package_identification/ros.py", line 129, in getter
    return get_setup_information(
  File "/usr/local/lib/python3.9/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 241, in get_setup_information
    _setup_information_cache[hashable_env] = _get_setup_information(
  File "/usr/local/lib/python3.9/site-packages/colcon_python_setup_py/package_identification/python_setup_py.py", line 286, in _get_setup_information
    return ast.literal_eval(output)
  File "/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    {'package_data': {}, 'dist_files': [], 'src_root': None, 'dependency_links': [], 'setup_requires': [], 'convert_2to3_doctests': [], 'eager_resources': None, 'entry_points': {'console_scripts': []}, 'exclude_package_data': None, 'extras_require': {}, 'include_package_data': True, 'install_requires': ['setuptools', 'wheel', 'setuptools_rust'], 'namespace_packages': None, 'packages': ['rust_test'], 'python_requires': None, 'test_loader': None, 'test_runner': None, 'test_suite': None, 'tests_require': ['pytest'], 'use_2to3': None, 'use_2to3_exclude_fixers': None, 'use_2to3_fixers': None, 'zip_safe': False, 'rust_extensions': [<setuptools_rust.extension.RustExtension object at 0x104bb6e20>], 'cffi_modules': None, 'verbose': 1, 'dry_run': 0, 'help': 0, 'command_packages': None, 'script_name': 'setup.py', 'script_args': ['--dry-run'], 'command_options': {'install': {'prefix': ('/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg', '/usr/local')}, 'build_ext': {'include_dirs': ('/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg', '/usr/local/include:/usr/local/opt/[email protected]/include:/usr/local/opt/sqlite/include'), 'library_dirs': ('/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/distutils.cfg', '/usr/local/lib:/usr/local/opt/[email protected]/lib:/usr/local/opt/sqlite/lib')}}, 'package_dir': None, 'py_modules': None, 'libraries': None, 'headers': None, 'ext_package': None, 'include_dirs': None, 'extra_path': None, 'scripts': None, 'data_files': [('share/ament_index/resource_index/packages', ['resource/rust_test']), ('share/rust_test', ['package.xml'])], 'password': '', 'command_obj': {}, 'have_run': {}, 'want_user_cfg': True, 'metadata': {'name': 'rust_test', 'version': '0.1.0', 'author': None, 'author_email': None, 'maintainer': 'schoen', 'maintainer_email': '[email protected]', 'url': None, 'license': 'TODO: License declaration', 'description': 'TODO: Package description', 'long_description': None, 'keywords': None, 'platforms': None, 'classifiers': None, 'download_url': None, 'provides': None, 'requires': None, 'obsoletes': None, 'long_description_content_type': None, 'project_urls': {}}}
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          ^
SyntaxError: invalid syntax
---
Failed   <<< rust_test [1.97s, exited with code 1]

Summary: 0 packages finished [3.04s]
  1 package failed: rust_test
  1 package had stderr output: rust_test
invalid syntax (<unknown>, line 1)

From what I see, without knowing too much about colcon itself, is that the package data object seems to have something unexpected as a result of the additional logic, and it isn't being handled once it is received. Alternatively, I have sometimes noticed similar errors when setuptools extensions create stdout, but I am not sure that this is the same thing. In any case, does this seem like something that can be handled, or is it outside of scope for this repo/project?

Thanks!

[Update] I found that the setuptools_rust package allows a quiet flag, so I tested this in case the stdout was causing an issue. However, this seemed to make no difference and the error was the same.

[dirk-thomas]

From what I see, without knowing too much about colcon itself, is that the package data object seems to have something unexpected as a result of the additional logic, and it isn't being handled once it is received.

Correct, the rust_extensions contain a complex object of type setuptools_rust.extension.RustExtension which the current code isn't able to serialize / deserialize. The underlying problem is the same as in #33 and therefore I will close this ticket as a duplicate.

Eventually on of the pending PRs (#34 or #35) will remove this limitation in the future.