$PWD/miniconda/bin also needs to be in PATH

[johannct]

else the correct python framework is not selected against the local one.

[johannct]

miniconda/bin/conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build fails when trying to build the first package lsst-doxygen. The log points to failing to run
/bin/bash -x -e /home/cohen/lsst/dm/conda-lsst/recipes/lsst-doxygen/build.sh
which indeed returns :
++ git rev-parse --show-toplevel

• export SRC_DIR=/home/cohen/lsst/dm/conda-lsst
• SRC_DIR=/home/cohen/lsst/dm/conda-lsst
• cd /home/cohen/lsst/dm/conda-lsst
• [[ linux-gnu == darwin* ]]
  +++ which c++
  ++ dirname /usr/bin/c++
• [[ /usr/bin -ef /bin ]]
• export LIBRARY_PATH=:/lib
• LIBRARY_PATH=:/lib
• [[ -f SConstruct ]]
  ++ basename 'ups/*.table' .table
• PRODUCT='*'
• EUPS_VERSION=1.8.5-1-g323f995
• PREFIX='/opt/lsst/*'
• source eups-setups.sh
  /home/cohen/lsst/dm/conda-lsst/recipes/lsst-doxygen/build.sh: line 71: eups-setups.sh: No such file or directory

[mjuric]

Hi @johannct,
That's unexpected, as eups-setups.sh should automatically be installed in the _build environment, picked up as a dependency (you shouldn't have to have it installed in your root environment).

Could you send me the full log from the doxygen failure, and also the output of the conda lsst make-recipes .... --build command?

[johannct]

I tried to relaunch it, but now I have yet another error :
cohen@port-jct:/lsst/dm/conda-lsst$ echo $PATH
/home/cohen/lsst/dm/conda-lsst/miniconda/bin:/home/cohen/lsst/dm/conda-lsst/bin:/home/cohen/sources/astro/Healpix_3.30/src/cxx//bin:/home/cohen/sources/astro/Healpix_3.30/bin:/home/cohen/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/anaconda1anaconda2anaconda3/opt/eups/bin
cohen@port-jct:/lsst/dm/conda-lsst$ conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build
updating built package cache [from file:///home/cohen/lsst/dm/conda-lsst/miniconda/conda-bld/linux-64] done.
updating built package cache [from http://eupsforge.net/conda/dev/linux-64] ..+.+.+++++.++++.++.+.+++++++.+.+++++++++++++.++++..+++.+++.++++++++++++++++.++.++.+.++++++.++.+++.++++++.++++.++++.+++.+++++++..+.+++++++++++++++++.+++.+++++.+.++++++.+.++.++..++++++..+.+..+++.++Traceback (most recent call last):
File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 147, in
db.reindex(config.channels)
File "/home/cohen/lsst/dm/conda-lsst/conda_lsst/recipe_db.py", line 186, in reindex
self.reindex_channel(channel)
File "/home/cohen/lsst/dm/conda-lsst/conda_lsst/recipe_db.py", line 263, in reindex_channel
all = tf.getnames()
File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 1844, in getnames
return [tarinfo.name for tarinfo in self.getmembers()]
File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 1836, in getmembers
self._load() # all members, we first have to
File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 2416, in _load
tarinfo = self.next()
File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 2347, in next
self.fileobj.seek(self.offset - 1)
EOFError: compressed file ended before the logical end-of-stream was detected

Surprisingly it tries to recache everything here, so maybe something changed on the server? Anyway I will try a complete reinstall (with the ./local easy_install.pth kludge and adding miniconda/bin to PATH) and will report back

[johannct]

Ok here is the full screen output of my clean install attempt :
cohen@port-jct:$ cd lsst/dm/
cohen@port-jct:/lsst/dm$ git clone [email protected]:mjuric/conda-lsst.git
Cloning into 'conda-lsst'...
remote: Counting objects: 499, done.
remote: Total 499 (delta 0), reused 0 (delta 0), pack-reused 498
Receiving objects: 100% (499/499), 158.89 KiB | 0 bytes/s, done.
Resolving deltas: 100% (245/245), done.
Checking connectivity... done.
cohen@port-jct:/lsst/dm$ cd conda-lsst/
cohen@port-jct:/lsst/dm/conda-lsst$ export PATH=$PWD/bin:^C
cohen@port-jct:/lsst/dm/conda-lsst$ ls
bin conda_lsst etc README.md samples scripts
cohen@port-jct:/lsst/dm/conda-lsst$ bin/bootstrap.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 24.2M 100 24.2M 0 0 906k 0 0:00:27 0:00:27 --:--:-- 1037k
PREFIX=/home/cohen/lsst/dm/conda-lsst/miniconda
installing: _cache-0.0-py27_x0 ...
installing: python-2.7.11-0 ...
installing: conda-env-2.4.5-py27_0 ...
installing: openssl-1.0.2d-0 ...
installing: pycosat-0.6.1-py27_0 ...
installing: pyyaml-3.11-py27_1 ...
installing: readline-6.2-2 ...
installing: requests-2.9.0-py27_0 ...
installing: sqlite-3.8.4.1-1 ...
installing: tk-8.5.18-0 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-0 ...
installing: conda-3.19.0-py27_0 ...
installing: pycrypto-2.6.1-py27_0 ...
installing: pip-7.1.2-py27_0 ...
installing: wheel-0.26.0-py27_1 ...
installing: setuptools-18.8.1-py27_0 ...
Python 2.7.11 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Fetching package metadata: ......
Solving package specifications: ........................
Package plan for installation in environment /home/cohen/lsst/dm/conda-lsst/miniconda:

The following packages will be downloaded:

package	build
openssl-1.0.2f	0 3.2 MB
patchelf-0.8	0 258 KB
sqlite-3.9.2	0 3.9 MB
markupsafe-0.23	py27_0 30 KB
requests-2.9.1	py27_0 605 KB
setuptools-19.6.2	py27_0 365 KB
sqlalchemy-1.0.11	py27_0 1.3 MB
conda-3.19.1	py27_0 177 KB
jinja2-2.8	py27_0 263 KB
pip-8.0.2	py27_0 1.5 MB
conda-build-1.19.0	py27_0 211 KB

------------------------------------------------------------
                                       Total:        11.7 MB

The following NEW packages will be INSTALLED:

conda-build: 1.19.0-py27_0
jinja2:      2.8-py27_0   
markupsafe:  0.23-py27_0  
patchelf:    0.8-0        
sqlalchemy:  1.0.11-py27_0

The following packages will be UPDATED:

conda:       3.19.0-py27_0 --> 3.19.1-py27_0
openssl:     1.0.2d-0      --> 1.0.2f-0     
pip:         7.1.2-py27_0  --> 8.0.2-py27_0 
requests:    2.9.0-py27_0  --> 2.9.1-py27_0 
setuptools:  18.8.1-py27_0 --> 19.6.2-py27_0
sqlite:      3.8.4.1-1     --> 3.9.2-0      

Fetching packages ...
openssl-1.0.2f 100% |################################| Time: 0:00:05 638.08 kB/s
patchelf-0.8-0 100% |################################| Time: 0:00:00 321.44 kB/s
sqlite-3.9.2-0 100% |################################| Time: 0:00:05 739.08 kB/s
markupsafe-0.2 100% |################################| Time: 0:00:00 172.33 kB/s
requests-2.9.1 100% |################################| Time: 0:00:01 504.23 kB/s
setuptools-19. 100% |################################| Time: 0:00:01 268.56 kB/s
sqlalchemy-1.0 100% |################################| Time: 0:00:02 455.35 kB/s
conda-3.19.1-p 100% |################################| Time: 0:00:00 210.42 kB/s
jinja2-2.8-py2 100% |################################| Time: 0:00:00 392.95 kB/s
pip-8.0.2-py27 100% |################################| Time: 0:00:03 505.93 kB/s
conda-build-1. 100% |################################| Time: 0:00:00 246.62 kB/s
Extracting packages ...
[ COMPLETE ]|###################################################| 100%
Unlinking packages ...
[ COMPLETE ]|###################################################| 100%
Linking packages ...
[ COMPLETE ]|###################################################| 100%
Collecting requests-file
Collecting six (from requests-file)
Using cached six-1.10.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests>=1.0.0 in ./miniconda/lib/python2.7/site-packages (from requests-file)
Installing collected packages: six, requests-file
Successfully installed requests-file-1.4 six-1.10.0
Miniconda has been installed in /home/cohen/lsst/dm/conda-lsst/miniconda. Add it to your path:

export PATH="/home/cohen/lsst/dm/conda-lsst/miniconda/bin:$PATH"

and continue.
cohen@port-jct:/lsst/dm/conda-lsst$ export PATH=/home/cohen/lsst/dm/conda-lsst/bin:/home/cohen/lsst/dm/conda-lsst/miniconda/bin:$PATH
cohen@port-jct:/lsst/dm/conda-lsst$ conda config --add channels http://eupsforge.net/conda/dev
Skipping channels: http://eupsforge.net/conda/dev, item already exists
cohen@port-jct:/lsst/dm/conda-lsst$ conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build
updating built package cache [from file:///home/cohen/lsst/dm/conda-lsst/miniconda/conda-bld/linux-64] not found. skipping.
updating built package cache [from http://eupsforge.net/conda/dev/linux-64] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ done.
https://raw.githubusercontent.com/lsst/versiondb/master/manifests/b1852.txt
generating recipes:
lsst-doxygen-1.8.5.1...
eups-1.5.9...
lsst-sconsutils-2015_11.0.2.post1...
lsst-product-configs-0.0.1...
lsst-eups-environment-2015_10.0.post11...
lsst-boost-1.59.5...
lsst-base-2015_10.0.2...
lsst-pex-exceptions-2015_10.0.1.post11...
lsst-utils-2015_10.0.3.post9...
lsst-daf-base-2015_10.0.3.post9...
lsst-mysqlclient-5.1.73.3...
lsst-pex-logging-2015_10.0.1.post11...
lsst-pex-policy-2015_10.0.1.post12...
lsst-python-d2to1-0.2.12...
lsst-stsci-distutils-0.3.7.1...
lsst-pyfits-3.2.4.2.post1...
lsst-daf-persistence-2015_10.0.6.post5...
lsst-pex-config-2015_10.0.1.post11...
lsst-eigen-3.2.5...
lsst-fftw-3.3.4.2...
lsst-ndarray-0.10.1.post55...
lsst-cfitsio-3360.4...
lsst-wcslib-4.14.2...
lsst-minuit2-5.28.00.2...
lsst-gsl-1.16.3...
lsst-afw-2015_10.0.14...
lsst-geom-0.10.0.post47...
lsst-skypix-0.10.0.post330...
lsst-daf-butlerutils-2015_10.0.4.post1...
lsst-obs-test-2015_10.0.3.post6...
lsst-pipe-base-2015_10.0.5...
lsst-coadd-utils-2015_10.0.4.post11...
lsst-meas-base-2015_10.0.4.post11...
lsst-meas-algorithms-2015_10.0.4.post3...
lsst-meas-deblender-2015_10.0.3.post22...
lsst-shapelet-2015_10.0.1.post21...
lsst-astrometry-net-0.50.2...
lsst-meas-astrom-2015_10.0.5...
lsst-astrometry-net-data-0.10.0.post56...
lsst-ip-isr-2015_10.0.5...
lsst-ip-diffim-2015_10.0.1.post31...
lsst-coadd-chisquared-2015_10.0.1.post27...
lsst-skymap-2015_10.0.4.post1...
lsst-pipe-tasks-2015_10.0.13...
lsst-meas-modelfit-2015_10.0.2.post12...
lsst-mysqlpython-1.2.3.1.post1...
lsst-obs-lsstsim-2015_10.0.4...
lsst-obs-sdss-2015_10.0.5...
lsst-xpa-2.1.15.3...
lsst-display-ds9-2015_10.0.post27...
lsst-psfex-2015_10.0.1.post8...
lsst-meas-extensions-psfex-2015_10.0.2.post22...
lsst-apps-0.11.0.post73...
lsst-apr-1.3.3.3...
lsst-apr-util-1.3.4.3...
lsst-log4cxx-0.10.0.6...
lsst-log-2015_10.0.2.post9...
lsst-sqlalchemy-1.0.8.2.post1...
lsst-db-2015_10.0.post15...
lsst-cat-2015_10.0.1.post16...
lsst-activemqcpp-3.9.0.2.post1...
lsst-ctrl-events-2015_10.0.2.post10...
lsst-ctrl-provenance-2015_10.0.1.post16...
lsst-ctrl-orca-2015_10.0.2.post17...
lsst-ctrl-execute-2015_10.0.3.post17...
lsst-ctrl-platform-gordon-0.11.0.post44...
lsst-ctrl-platform-lsst-0.11.0.post44...
lsst-mariadb-0.10.1.9...
lsst-scisql-0.3.4.1.post1...
lsst-datarel-2015_10.0.1.post41...
lsst-distrib-0.11.0.post86...
lsst-pyephem-3.7.5.1.1...
lsst-palpy-1.7.0...
lsst-freetds-0.91.112.2...
lsst-pymssql-2.1.1.post3...
setuptools-git-1.1...
lsst-pykg-config-1.2.0.post5...
lsst-healpy-1.8.1.post15...
lsst-sims-data-0.10.1...
lsst-sims-utils-2.1.2.1.post1...
lsst-sims-catalogs-generation-2.1.2.1.post1...
lsst-sims-sed-library-2014.10.06...
lsst-sims-dustmaps-0.10.1.3...
lsst-throughputs-master.20160107215629...
lsst-sims-photutils-2.1.2.2.post1...
lsst-sims-coordutils-2.1.2.post8...
lsst-sims-maf-2.1.3.post3...
lsst-tmv-0.72.3...
lsst-galsim-1.2.2.1.post18...
lsst-sims-catalogs-measures-2.1.2.post4...
lsst-sims-catutils-2.1.2.2.post2...
lsst-sims-galsiminterface-2.1.2.post12...
lsst-sncosmo-1.1.1.1...
astropy-helpers-1.1.1...
lsst-sims-master.20150928224422.post102...
done.
generating rebuild script:
already built: eups-1.5.9-7
will build: lsst-doxygen-1.8.5.1-323f995_3
already built: lsst-product-configs-0.0.1-0
will build: lsst-sconsutils-2015_11.0.2.post1-9960fc2_0
will build: lsst-eups-environment-2015_10.0.post11-0
will build: lsst-boost-1.59.5-5
will build: lsst-base-2015_10.0.2-89e5401_0
will build: lsst-pex-exceptions-2015_10.0.1.post11-f7aee88_0
will build: lsst-utils-2015_10.0.3.post9-530cd83_0
will build: lsst-daf-base-2015_10.0.3.post9-4d32b56_0
will build: lsst-mysqlclient-5.1.73.3-3
will build: lsst-pex-logging-2015_10.0.1.post11-f6c85b3_0
will build: lsst-pex-policy-2015_10.0.1.post12-d4eb80a_0
will build: lsst-python-d2to1-0.2.12-4
will build: lsst-stsci-distutils-0.3.7.1-b22a065_3
will build: lsst-pyfits-3.2.4.2.post1-3
will build: lsst-daf-persistence-2015_10.0.6.post5-5a5f333_0
will build: lsst-pex-config-2015_10.0.1.post11-c006da1_0
will build: lsst-eigen-3.2.5-5
will build: lsst-fftw-3.3.4.2-3
will build: lsst-ndarray-0.10.1.post55-0
will build: lsst-cfitsio-3360.4-4
will build: lsst-wcslib-4.14.2-0
will build: lsst-minuit2-5.28.00.2-3
will build: lsst-gsl-1.16.3-3
will build: lsst-afw-2015_10.0.14-7c5ed66_0
will build: lsst-geom-0.10.0.post47-0
will build: lsst-skypix-0.10.0.post330-0
will build: lsst-daf-butlerutils-2015_10.0.4.post1-ca2b07e_0
will build: lsst-obs-test-2015_10.0.3.post6-c4c7f1b_0
will build: lsst-pipe-base-2015_10.0.5-417e839_0
will build: lsst-coadd-utils-2015_10.0.4.post11-74d0a18_0
will build: lsst-meas-base-2015_10.0.4.post11-87d2c31_0
will build: lsst-meas-algorithms-2015_10.0.4.post3-270941b_0
will build: lsst-meas-deblender-2015_10.0.3.post22-a5a97e7_0
will build: lsst-shapelet-2015_10.0.1.post21-7c5d349_0
will build: lsst-astrometry-net-0.50.2-0
will build: lsst-meas-astrom-2015_10.0.5-7977c95_0
will build: lsst-astrometry-net-data-0.10.0.post56-0
will build: lsst-ip-isr-2015_10.0.5-1b37747_0
will build: lsst-ip-diffim-2015_10.0.1.post31-c9560c6_0
will build: lsst-coadd-chisquared-2015_10.0.1.post27-470bb2d_0
will build: lsst-skymap-2015_10.0.4.post1-65a3ccf_0
will build: lsst-pipe-tasks-2015_10.0.13-4f8857f_0
will build: lsst-meas-modelfit-2015_10.0.2.post12-4183c33_0
will build: lsst-mysqlpython-1.2.3.1.post1-3
will build: lsst-obs-lsstsim-2015_10.0.4-17d213c_0
will build: lsst-obs-sdss-2015_10.0.5-b0be3e2_0
will build: lsst-xpa-2.1.15.3-0
will build: lsst-display-ds9-2015_10.0.post27-0
will build: lsst-psfex-2015_10.0.1.post8-eb1eb67_0
will build: lsst-meas-extensions-psfex-2015_10.0.2.post22-ce306f0_0
will build: lsst-apps-0.11.0.post73-0
will build: lsst-apr-1.3.3.3-0
will build: lsst-apr-util-1.3.4.3-0
will build: lsst-log4cxx-0.10.0.6-0
will build: lsst-log-2015_10.0.2.post9-81df675_0
will build: lsst-sqlalchemy-1.0.8.2.post1-0
will build: lsst-db-2015_10.0.post15-0
will build: lsst-cat-2015_10.0.1.post16-a13b24a_0
will build: lsst-activemqcpp-3.9.0.2.post1-0
will build: lsst-ctrl-events-2015_10.0.2.post10-b6a79c0_0
will build: lsst-ctrl-provenance-2015_10.0.1.post16-ef242a4_0
will build: lsst-ctrl-orca-2015_10.0.2.post17-4f54dab_0
will build: lsst-ctrl-execute-2015_10.0.3.post17-4ff3670_0
will build: lsst-ctrl-platform-gordon-0.11.0.post44-0
will build: lsst-ctrl-platform-lsst-0.11.0.post44-0
will build: lsst-mariadb-0.10.1.9-0
will build: lsst-scisql-0.3.4.1.post1-0
will build: lsst-datarel-2015_10.0.1.post41-63a66a4_0
will build: lsst-distrib-0.11.0.post86-0
will build: lsst-pyephem-3.7.5.1.1-7
will build: lsst-palpy-1.7.0-7
will build: lsst-freetds-0.91.112.2-4
already built: setuptools-git-1.1-py27_0
will build: lsst-pymssql-2.1.1.post3-4
will build: lsst-pykg-config-1.2.0.post5-7
will build: lsst-healpy-1.8.1.post15-3
will build: lsst-sims-data-0.10.1-0
will build: lsst-sims-utils-2.1.2.1.post1-fd59bd9_0
will build: lsst-sims-catalogs-generation-2.1.2.1.post1-ced3926_0
will build: lsst-sims-sed-library-2014.10.06-0
will build: lsst-sims-dustmaps-0.10.1.3-ffe99e1_3
will build: lsst-throughputs-master.20160107215629-3df0d76b78_2
will build: lsst-sims-photutils-2.1.2.2.post1-ecf83be_0
will build: lsst-sims-coordutils-2.1.2.post8-0
will build: lsst-sims-maf-2.1.3.post3-0
will build: lsst-tmv-0.72.3-0
will build: lsst-galsim-1.2.2.1.post18-6f8a81a_0
will build: lsst-sims-catalogs-measures-2.1.2.post4-0
will build: lsst-sims-catutils-2.1.2.2.post2-9e89ac4_0
will build: lsst-sims-galsiminterface-2.1.2.post12-0
already built: astropy-helpers-1.1.1-0
will build: lsst-sncosmo-1.1.1.1-cea177f_0
will build: lsst-sims-master.20150928224422.post102-c2d9827e8b_0
done.
calling /home/cohen/lsst/dm/conda-lsst/recipes/rebuild.sh to build:
lsst-doxygen (ver 1.8.5.1-323f995_3) [from doxygen-1.8.5-1-g323f995] ... ERROR
Traceback (most recent call last):
File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 149, in
args.func(config, args)
File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 30, in main_make_recipes
subprocess.check_call('bash %s/rebuild.sh' % (config.output_dir), shell=True)
File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /home/cohen/lsst/dm/conda-lsst/recipes/rebuild.sh' returned non-zero exit status 255
cohen@port-jct:/lsst/dm/conda-lsst$

[johannct]

the log is pretty atrocious, but only the last line, already know, seems relevant....
_build.txt

[mjuric]

You're missing flex. From the log file:

  Checking for perl... using /usr/bin/perl
  Checking for flex... not found!

[johannct]

darn, thanks and sorry! I missed it in the log....

[johannct]

for the record, I was also missing bison.

[johannct]

I am kipping this issue open for the different troubles I have, as it might prove useful to others trying conda install on their laptop (esp. Ubuntu 14.04 as mine). So I am now failing at lsst-utils, where numpy seems to have a problem at compilation. The only place I could see a hint at a problem is in :

Checking who built the CC compiler...error: no result
CC is unknown version unknown
Checking for C++11 support
Checking whether the C++ compiler worksyes
C++11 supported with '-std=c++11'
Setting up environment to build package 'utils'.
Ignoring prefix /home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/var/opt/eups/Linux64/utils/2015_10.0-3-g530cd83+9 from EUPS_PATH
numpy was found but did not pass configuration checks.

I attach the log file again. Sorry for the trouble!
_build.txt

[johannct]

I went a little farther down in debugging by adding a message printing in /home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/opt/lsst/lsst-product-configs/ups/numpy.cfg :
def configure(self, conf, packages, check=False, build=True):
lsst.sconsUtils.log.info("Configuring package '%s'." % self.name)
try:
import numpy
except ImportError,msg:
print msg #here
return False
conf.env.AppendUnique(XCPPPATH=numpy.get_include().split())
return True
which I think should be added in general. This now tells me that the problem is that libgfortran.so.1 is not found. Indeed I have libgfortran.so.3 in my Linux Ubuntu 14.04 box, so I tried a soft link. But then numpy importing bombs again :
/home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_copy_string
So it looks like I need to rebuild numpy in miniconda. Is it possible, or has it been downloaded prebuilt? In the latter case, I guess I can replace the miniconda numpy by my system native one.... but that is a nice loophole for incompatibilities down the line....