"protoc-gen-python_betterproto: program not found or is not executable"

[rebello95]

Hello! I've been using python-betterproto for quite some time, but after setting up the installation on a new machine (M1 MacBook Pro), I am getting the following errors when using the compiler. It seems to be exclusive to this plugin, as I've been able to successfully invoke protoc with other output plugins. My protoc installation was done with Homebrew.

michael@Michaels-MacBook-Pro path % python3 -m pip install "betterproto[compiler]"                                            
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: betterproto[compiler] in /Users/michael/Library/Python/3.8/lib/python/site-packages (1.2.5)
Requirement already satisfied: stringcase in /Users/michael/Library/Python/3.8/lib/python/site-packages (from betterproto[compiler]) (1.2.0)
Requirement already satisfied: grpclib in /Users/michael/Library/Python/3.8/lib/python/site-packages (from betterproto[compiler]) (0.3.2)
Requirement already satisfied: jinja2 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from betterproto[compiler]) (2.11.2)
Requirement already satisfied: black in /Users/michael/Library/Python/3.8/lib/python/site-packages (from betterproto[compiler]) (22.3.0)
Requirement already satisfied: protobuf in /Users/michael/Library/Python/3.8/lib/python/site-packages (from betterproto[compiler]) (3.12.2)
Requirement already satisfied: pathspec>=0.9.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (0.9.0)
Requirement already satisfied: platformdirs>=2 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (2.5.1)
Requirement already satisfied: mypy-extensions>=0.4.3 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (0.4.3)
Requirement already satisfied: typing-extensions>=3.10.0.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (4.1.1)
Requirement already satisfied: click>=8.0.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (8.1.2)
Requirement already satisfied: tomli>=1.1.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from black->betterproto[compiler]) (2.0.1)
Requirement already satisfied: multidict in /Users/michael/Library/Python/3.8/lib/python/site-packages (from grpclib->betterproto[compiler]) (4.7.6)
Requirement already satisfied: h2 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from grpclib->betterproto[compiler]) (3.2.0)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from jinja2->betterproto[compiler]) (1.1.1)
Requirement already satisfied: setuptools in /Users/michael/Library/Python/3.8/lib/python/site-packages (from protobuf->betterproto[compiler]) (62.0.0)
Requirement already satisfied: six>=1.9 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from protobuf->betterproto[compiler]) (1.15.0)
Requirement already satisfied: hpack<4,>=3.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from h2->grpclib->betterproto[compiler]) (3.0.0)
Requirement already satisfied: hyperframe<6,>=5.2.0 in /Users/michael/Library/Python/3.8/lib/python/site-packages (from h2->grpclib->betterproto[compiler]) (5.2.0)
michael@Michaels-MacBook-Pro path % protoc --python_betterproto_out="./out" -I "./protos" $(find protos -name "*.proto")
protoc-gen-python_betterproto: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.

Could there be something misconfigured with my installation? Any tips would be greatly appreciated.

[Gobot1234]

Please can you try installing with the --pre flag in pip?

[rebello95]

@Gobot1234 thanks for the prompt reply!

Force reinstalling with python3 -m pip install --pre --force-reinstall "betterproto[compiler]" helped. It looks like it installed to $HOME/Library/Python/3.8/bin which was not in my PATH. Adding export PATH="$HOME/Library/Python/3.8/bin:$PATH" to my shell profile after reinstalling the pre-release version resolved the issue with finding the binary.

However, running the generator now fails with the error --python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1. and no additional logs. I also tried a simpler version of my original command with a single proto file, but got the same error. Any suggestions?

[Gobot1234]

Can you run pip show?

[rebello95]

Yep:

michael@Michaels-MacBook-Pro Development % pip3 show "betterproto[compiler]"
WARNING: Package(s) not found: betterproto[compiler]
michael@Michaels-MacBook-Pro Development % pip3 show "betterproto"          
Name: betterproto
Version: 2.0.0b4
Summary: A better Protobuf / gRPC generator & library
Home-page: https://github.com/danielgtaylor/python-betterproto
Author: Daniel G. Taylor
Author-email: [email protected]
License: MIT
Location: /Users/michael/Library/Python/3.8/lib/python/site-packages
Requires: grpclib, python-dateutil
Required-by: 
michael@Michaels-MacBook-Pro Development %

[Gobot1234]

Oh sorry my bad I meant pip freeze

[rebello95]

michael@Michaels-MacBook-Pro Development % pip3 freeze
appdirs==1.4.4
attrs==20.1.0
betterproto==2.0.0b4
black==22.3.0
cachetools==4.1.1
certifi==2020.6.20
cffi==1.14.5
chardet==3.0.4
click==8.1.2
decorator==4.4.2
google-crc32c==1.1.2
googleapis-common-protos==1.56.0
grpcio==1.44.0
grpcio-tools==1.44.0
grpclib==0.4.3rc2
h2==4.1.0
hpack==4.0.0
httplib2==0.18.1
hyperframe==6.0.1
idna==2.10
imageio==2.14.1
imageio-ffmpeg==0.4.5
install==1.3.5
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==2.1.1
moviepy==1.0.3
msgpack==1.0.0
multidict==6.0.2
mypy==0.812
mypy-extensions==0.4.3
numpy==1.21.5
packaging==20.9
pathspec==0.9.0
pep517==0.12.0
pg8000==1.16.5
Pillow==9.0.0
pip-tools==6.6.0
platformdirs==2.5.1
proglog==0.1.9
protobuf==3.12.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pygeohash==1.2.0
pyparsing==2.4.7
python-dateutil==2.8.2
pytz==2020.1
regex==2020.7.14
requests==2.24.0
rsa==4.6
scramp==1.2.0
six==1.16.0
SQLAlchemy==1.3.18
stringcase==1.2.0
toml==0.10.1
tomli==2.0.1
tqdm==4.62.3
typed-ast==1.4.1
typing_extensions==4.1.1
uritemplate==3.0.1
urllib3==1.25.10
Werkzeug==1.0.1

[Gobot1234]

Can you try with grpc's compiler python -m grpc_tools.protoc --python_betterproto_out="./out" -I "./protos" $(find protos -name "*.proto"). Not really sure whats up if it's not that unless pip3 isn't the same thing as python3 -m pip.

[rebello95]

Just tried with both python and python3 but with the same result:

michael@Michaels-MacBook-Pro path % python -m grpc_tools.protoc --python_betterproto_out="./out" -I "./protos" $(find protos -name "*.proto")
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.
michael@Michaels-MacBook-Pro path % python -m grpc_tools.protoc --python_betterproto_out="./out" -I "./protos" $(find protos -name "*.proto")
michael@Michaels-MacBook-Pro path % python3 -m grpc_tools.protoc --python_betterproto_out="./out" -I "./protos" $(find protos -name "*.proto")
--python_betterproto_out: protoc-gen-python_betterproto: Plugin failed with status code 1.

Regarding pip3 versus python3 -m, I got the same result when running the latter:

michael@Michaels-MacBook-Pro path % python3 -m pip show "betterproto"          
Name: betterproto
Version: 2.0.0b4
Summary: A better Protobuf / gRPC generator & library
Home-page: https://github.com/danielgtaylor/python-betterproto
Author: Daniel G. Taylor
Author-email: [email protected]
License: MIT
Location: /Users/michael/Library/Python/3.8/lib/python/site-packages
Requires: grpclib, python-dateutil
Required-by: 

Is there any way to get additional logs out of the protoc invocation that would be helpful to debug?

[Gobot1234]

I'm sorry I don't know then. This is one of the main reasons I've been working on #165 (which completely removes protoc's CLI from the compilation process), unfortunately the accompanying module which that PR needs doesn't compile on anything other than macOS for reasons I haven't been able to figure out.

[rebello95]

Got it. Could it be that this is because I'm using a machine with an M1 chip? I'm curious to know if you've seen the plugin work properly on that hardware

[Gobot1234]

If that's what you think it could be you could try compiling the protobuf to a standard google.protobuf package. betterproto's plugin doesn't interact with any c-api so if it isn't compiling due to protoc it's likely the compiler itself is the issue.

[rebello95]

Thanks a bunch for all the help with this @Gobot1234. It looks like this was an issue with gRPC on M1 machines. I ended up getting this working again thanks to this tip: grpc/grpc#25082 (comment)

For future reference, here's what I did:

sudo xcode-select -s /Library/Developer/CommandLineTools
brew install pyenv
CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.10.3
pyenv global 3.10.3
pip3 install -v https://files.pythonhosted.org/packages/65/75/8b706e1170e2c7b6242b1675259e47986bb4fc490f29387989a965972e6e/grpcio-1.44.0.tar.gz