Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storm 1.0.3 #90

Open
dmitry-saritasa opened this issue Apr 14, 2017 · 17 comments
Open

Storm 1.0.3 #90

dmitry-saritasa opened this issue Apr 14, 2017 · 17 comments

Comments

@dmitry-saritasa
Copy link

hi Barry,

Petrel complains that Apache storm version is not equal to 1.0.2 when I submit into storm cluster v 1.0.3 (due to supplied Jar file?)

../../jvmpetrel/target/storm-petrel-*-SNAPSHOT.jar

Where can I get jar file for 1.0.3?

Thanks,
Dmitry
@barrywhart
Copy link
Contributor

Are you using the 1.0.1.0.3 version from here? That one should work, unless I built it incorrectly.

If necessary, you can build from source by installing your desired Storm version, then running python setup.py bdist_egg from the Petrel source directory.

Let me know if the pypi version is wrong, I can build and upload a corrected one. The build process is still slightly manual, unfortunately, for example, the jvmpetrel/pom.xml file has to be updated for each new Storm version. There may be a better way to do it; I am not a Maven expert.

@barrywhart
Copy link
Contributor

Instructions on building from source: https://github.com/AirSage/Petrel#installing-petrel-from-source

@dmitry-saritasa
Copy link
Author

https://pypi.python.org/simple/petrel/

petrel-0.9.3.0.3-py2.7.egg
petrel-0.9.3.0.1-py2.7.egg
petrel-0.9.0.0.1-py2.7.egg
petrel-1.0.1.0.3-py2.7.egg
petrel-0.10.0.0.3-py2.7.egg
petrel-0.9.3.0.2-py2.7.egg
petrel-1.0.1.0.3-py2-none-any.whl
petrel-1.0.2.0.3-py2.7.egg
petrel-1.0.2.0.3-py3.5.egg
petrel-0.8.2.0.1-py2.7.egg
petrel-0.9.4.0.3-py2.7.egg

Only petrel-1.0.2.0.3-py3.5.egg is available for python 3.5

@dmitry-saritasa
Copy link
Author

Just in case the following is the message I get:

(storm) > $ petrel submit  --destjar ../app.v0.1.3.jar --config topology.yaml app-persistence-v013
[Errno 2] No such file or directory: '/home/devops/.pyenv/versions/3.5.2/envs/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/generated/storm-petrel-1.0.3-SNAPSHOT-jar-with-dependencies.jar'
Traceback (most recent call last):
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/cmdline.py", line 126, in main
    func(**args.__dict__)
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/cmdline.py", line 54, in submit
    logdir=logdir)
  File "/home/devops/.pyenv/versions/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/package.py", line 100, in build_jar
    shutil.copy(source_jar_path, dest_jar_path)
  File "/home/devops/.pyenv/versions/3.5.2/lib/python3.5/shutil.py", line 235, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/devops/.pyenv/versions/3.5.2/lib/python3.5/shutil.py", line 114, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/devops/.pyenv/versions/3.5.2/envs/storm/lib/python3.5/site-packages/petrel-1.0.2.0.3-py3.5.egg/petrel/generated/storm-petrel-1.0.3-SNAPSHOT-jar-with-dependencies.jar'

@barrywhart
Copy link
Contributor

I made some mistakes uploading. Apparently even though the Python code is compatible with both 2.7 and 3.5, I need to explicitly upload a wheel file for Python 3.5. I'll try and do this in the next day or two.

@barrywhart
Copy link
Contributor

I uploaded a new build, petrel-1.0.3.0.3-py3-none-any.whl. Uploading the corresponding Python 2.7 wheel next. Have not yet done a test download/install.

@barrywhart
Copy link
Contributor

Download from pypi in Python 3.5 looks good:

(Petrel-test-3.5.3) barry@Dell660s:~/dev/Petrel$ pip install petrel==1.0.3.0.3
Collecting petrel==1.0.3.0.3
  Downloading petrel-1.0.3.0.3-py3-none-any.whl (313kB)
    100% |████████████████████████████████| 317kB 745kB/s 
Requirement already satisfied: six==1.10.0 in /home/barry/.pyenv/versions/3.5.3/envs/Petrel-test-3.5.3/lib/python3.5/site-packages (from petrel==1.0.3.0.3)
Collecting PyYAML==3.10 (from petrel==1.0.3.0.3)
  Using cached PyYAML-3.10.tar.gz
Collecting thrift==0.9.3 (from petrel==1.0.3.0.3)
  Using cached thrift-0.9.3.tar.gz
Building wheels for collected packages: PyYAML, thrift
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /home/barry/.cache/pip/wheels/cc/2a/d6/5a7108e2281e4c783740d79c40eac3ebc2d4157b1c7e4f17ef
  Running setup.py bdist_wheel for thrift ... done
  Stored in directory: /home/barry/.cache/pip/wheels/e5/20/32/cbe6d90e33b19825ea6d251ff0b1c778df8941750f5d5c3d3f
Successfully built PyYAML thrift
Installing collected packages: PyYAML, thrift, petrel
Successfully installed PyYAML-3.10 petrel-1.0.3.0.3 thrift-0.9.3

@barrywhart
Copy link
Contributor

Getting an error when the wordcount topology starts up:

5131 [SLOT_1024] ERROR o.a.s.d.worker - Error on initialization of server mk-worker
java.io.FileNotFoundException: File '/tmp/6a9ce996-17a2-4065-adae-f82db42d6aee/supervisor/stormdist/test%20topology-1-1492557826/stormconf.ser' does not exist

@barrywhart
Copy link
Contributor

The problem occurs with both Python 2.7 and 3.5 on Storm 1.0.3. I do not see the same issue with Storm 1.0.1. I will try Storm 1.0.2.

@barrywhart
Copy link
Contributor

I uploaded 2.7 and 3.5 wheel files to pypi and verified that Storm 1.0.2 works correctly with Python 3.5.3. The error that is occurring in 1.0.3 sounds like a Storm issue that other people have reported at various times. (Try doing a web search for "stormconf.ser does not exist".)

It seems likely to be a Storm bug or a problem with my environment. Can you try Storm 1.0.3 with Petrel 1.0.3.0.2 and see if it works for you?

@dmitry-saritasa
Copy link
Author

dmitry-saritasa commented Apr 19, 2017 via email

@dmitry-saritasa
Copy link
Author

dmitry-saritasa commented Apr 26, 2017 •
edited
Loading

this is the error I got once topology is submitted

Already using interpreter /usr/local/bin/python3.5                                                                                                                                                                                        [0/0]
Using base prefix '/usr/local'
New python executable in /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv/bin/python3.5
Also creating executable in /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv/bin/python
Installing setuptools, pip, wheel...done.
Installing thrift==0.9.3
The directory '/home/storm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/storm/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting thrift==0.9.3
  Downloading thrift-0.9.3.tar.gz
Installing collected packages: thrift
  Running setup.py install for thrift: started
    Running setup.py install for thrift: finished with status 'done'
Successfully installed thrift-0.9.3
Installing PyYAML==3.10
The directory '/home/storm/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/storm/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting PyYAML==3.10
  Downloading PyYAML-3.10.tar.gz (241kB)
Installing collected packages: PyYAML
  Running setup.py install for PyYAML: started
    Running setup.py install for PyYAML: finished with status 'done'
Successfully installed PyYAML-3.10
Installing petrel==1.0.3.0.3
Searching for petrel==1.0.3.0.3
Reading https://pypi.python.org/simple/petrel/
No local packages or working download links found for petrel==1.0.3.0.3
error: Could not find suitable distribution for Requirement.parse('petrel==1.0.3.0.3')

And then I'm getting tons of *.log files representing bolts and spouts tasks around 300 bytes long with similar content

[devops@storm-wk1-prod]~/storm/supervisor/stormdist/trivver-persistence-v019-debug-20-1493249743/resources% cat petrel23982_tbolts.log
Beginning task setup
/usr/bin/flock
Using existing venv: /tmp/petrel-proj-persistence-v019-debug-20-1493249743/venv
Task setup took 1 seconds
Launching: python -m petrel.run tbolts /var/log/storm/supervisor/stormdist/proj-persistence-v019-debug-20-1493249743/resources/petrel23982_tbolts.log

setup.sh

cat setup.sh
pip install -r requirements/production.txt
export PYTHONPATH=`pwd`

requirements/production.txt:

# Apache Storm Py lib
# you will need to install this lib manually
# so it's not called inside storm's supervisor
# petrel==1.0.3.0.3

# Important
# pip install --upgrade setuptools
# pip install --upgrade pip
# or you get nasty error
# see this thread
# https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2990
setuptools==34.3.2
thrift==0.9.3

# Database abstraction layer
psycopg2==2.7.1

# Timezones and datetime utils
pytz==2016.10

# redis SDK
redis==2.10.5

# nice string conversion procedures
inflection==0.3.1

# opbeat
opbeat==3.5.2

# boltons = super cool batteries included
# https://boltons.readthedocs.io/en/latest/
boltons==17.1.0

# msgpack for faster messaging to avoid JSONs
# http://msgpack.org/
msgpack-python==0.4.8
u-msgpack-python==2.3.0

# csv for fake data
tablib==0.11.4

# fake data generator
faker==0.7.10

# google cloud SDK (for BigTable)
# https://github.com/GoogleCloudPlatform/google-cloud-python
google-cloud==0.23.0

# signal processing
blinker==1.4

# cached property on classes
cached-property==1.3.0

# click for CLI
# http://click.pocoo.org/5/
click==6.7

# slacker
# https://github.com/os/slacker
# slack channel notifications
slacker==0.9.42

# kafka client
kafka-python==1.3.3

# this is needed for petrel
pyyaml==3.10

switched back petrel-1.0.2.0.3 installed with easy_install and now topology works successfully, snapshot of my virtualenv log after install
https://gist.github.com/dmitry-saritasa/381c66cb5738f77c09149d238720dee1

@barrywhart
Copy link
Contributor

I found an issue which I think explains this error:

Installing petrel==1.0.3.0.3
Searching for petrel==1.0.3.0.3
Reading https://pypi.python.org/simple/petrel/
No local packages or working download links found for petrel==1.0.3.0.3
error: Could not find suitable distribution for Requirement.parse('petrel==1.0.3.0.3')

Petrel was still trying to use easy_install to install itself from PyPI. I think now that it's being distributed as .whl files, easy_install won't work -- pip must be used instead.

I made that change and pushed a new version to PyPI: 1.0.3.0.3.post1. Because of the stormconf.ser issue I mentioned before, I can't actually test this new release, but I did verify it installs successfully with pip:

(Petrel-test-3.5.3) barry@Dell660s:~/dev/Petrel/samples/wordcount$ pip install petrel==1.0.3.0.3.post1
Collecting petrel==1.0.3.0.3.post1
  Downloading petrel-1.0.3.0.3.post1-py3-none-any.whl (556kB)
    100% |████████████████████████████████| 563kB 703kB/s 
Collecting thrift==0.9.3 (from petrel==1.0.3.0.3.post1)
Collecting PyYAML==3.10 (from petrel==1.0.3.0.3.post1)
Requirement already satisfied: six==1.10.0 in /home/barry/.pyenv/versions/3.5.3/envs/Petrel-test-3.5.3/lib/python3.5/site-packages (from petrel==1.0.3.0.3.post1)
Installing collected packages: thrift, PyYAML, petrel
Successfully installed PyYAML-3.10 petrel-1.0.3.0.3.post1 thrift-0.9.3

I have created a PR so you can see the changes related to adding 1.0.3 support. Please take a look; in case of issues, this may help you figure out any remaining problems more quickly.

#93

@LW-YUNKAI
Copy link

hi Barry, thank you for your work!
I got the same issue when I submit the topology, I wondered if I should start up petrel directly or start up zookeeper and storm first, and I want to run it on a cluster, should I change storm.yaml in storm?

6109 [SLOT_1027] ERROR o.a.s.d.worker - Error on initialization of server mk-worker
java.io.FileNotFoundException: File '/tmp/f9ad8a9c-52ad-49b2-9c18-2302fd5b6449/supervisor/stormdist/test%20topology-1-1508916726/stormconf.ser' does not exist

@barrywhart
Copy link
Contributor

@LW-YUNKAI: I am no longer an AirSage employee, and AirSage asked me to turn over control of the AirSage GitHub organization to them. As a result, I am no longer supporting Petrel. I regret I cannot help you.

@dmitry-saritasa
Copy link
Author

dmitry-saritasa commented Oct 25, 2017 via email

@LW-YUNKAI
Copy link

@barrywhart It is a pity! I hope that STORM could have better support for python in the future! Thank you anyway!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@barrywhart @dmitry-saritasa @LW-YUNKAI