Python: further migrate packages from 2 to 3 #131719

FRidh · 2021-07-27T18:37:26Z

Motivation for this change

Goal is to get rid of the python attribute.

Things done

Did not test with python3.

….nix

Not tested whether it works with python3.

not tested whether python3 works

but use python for the bindings.

not tested with 3

not tested with python3.

calling pythonPackages.

r-rmcgibbo · 2021-07-28T08:33:31Z

Result of nixpkgs-review pr 131719 at 238fa78 run on aarch64-linux 1

14 packages marked as broken and skipped:

python38Packages.ignite
python38Packages.pytorchWithCuda
python38Packages.spacy
python38Packages.spacy-transformers
python38Packages.tensorflowWithCuda
python38Packages.textacy
python39Packages.ignite
python39Packages.pytorchWithCuda
python39Packages.spacy
python39Packages.spacy-transformers
python39Packages.tensorflowWithCuda
python39Packages.textacy
rocm-opencl-icd
rocm-opencl-runtime

2 packages failed to build:

532 packages skipped due to time constraints:

ansible (ansible_2_11 ,python39Packages.ansible-core)
ansible-lint (python39Packages.ansible-lint)
ansible_2_10 (python39Packages.ansible-base)
apacheHttpdPackages.mod_tile (apacheHttpdPackages_2_4.mod_tile)
appdaemon
apprise (python39Packages.apprise)
azure-cli
beancount (python39Packages.beancount)
beets
beetsExternalPlugins.alternatives
...

30 packages built successfully:

coccinelle
flowblade
hammer
libsigrok
libsigrok_0_3
lkl
lyx
matrix-appservice-discord
maxima
maxima-ecl
mergerfs-tools
mlt
moosefs
neard
python38Packages.aprslib
python38Packages.mlt
python38Packages.mox3
python38Packages.requests-mock
python38Packages.subunit
python38Packages.testrepository
python39Packages.aprslib
python39Packages.mlt
python39Packages.mox3
python39Packages.requests-mock
python39Packages.sqlalchemy_migrate
python39Packages.subunit
python39Packages.testrepository
sigrok-cli
subunit
yder

3 suggestions:

warning: name-and-version
Did you mean to pass pname instead of name to mkDerivation?

Near pkgs/applications/science/math/maxima/default.nix:17:3:
```
   |
17 |   name = "${name}-${version}";
   |   ^
```
Near pkgs/applications/science/math/maxima/default.nix:16:10:
```
   |
16 |   inherit version;
   |          ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/development/python-modules/subunit/default.nix:17:20:
```
   |
17 |   inherit (subunit) name src meta;
   |                    ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/applications/science/math/maxima/default.nix:106:5:
```
    |
106 |     license = lib.licenses.gpl2;
    |     ^
```

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Result of nixpkgs-review pr 131719 at 238fa78 run on x86_64-linux 1

2 packages failed to build:

554 packages skipped due to time constraints:

ansible (ansible_2_11 ,python39Packages.ansible-core)
ansible-lint (python39Packages.ansible-lint)
ansible_2_10 (python39Packages.ansible-base)
apacheHttpdPackages.mod_tile (apacheHttpdPackages_2_4.mod_tile)
appdaemon
apprise (python39Packages.apprise)
azure-cli
beancount (python39Packages.beancount)
beets
beetsExternalPlugins.alternatives
...

32 packages built successfully:

coccinelle
flowblade
hammer
ja2-stracciatella
libsigrok
libsigrok_0_3
lkl
matrix-appservice-discord
maxima
maxima-ecl
mergerfs-tools
mlt
moosefs
neard
orcania
python38Packages.aprslib
python38Packages.mlt
python38Packages.mox3
python38Packages.requests-mock
python38Packages.sqlalchemy_migrate
python38Packages.subunit
python38Packages.testrepository
python39Packages.aprslib
python39Packages.mlt
python39Packages.mox3
python39Packages.requests-mock
python39Packages.sqlalchemy_migrate
python39Packages.subunit
python39Packages.testrepository
sigrok-cli
subunit
yder

27 suggestions:

warning: build-tools-in-build-inputs
doxygen is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/development/tools/libsigrok/default.nix:21:3:
```
   |
21 |   buildInputs = [ libzip glib libusb1 libftdi1 check libserialport
   |   ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/servers/neard/default.nix:34:5:
```
   |
34 |     license     = licenses.gpl2;
   |     ^
```
warning: build-tools-in-build-inputs
pkg-config is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/development/tools/misc/coccinelle/default.nix:12:3:
```
   |
12 |   buildInputs = with ocamlPackages; [
   |   ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/development/tools/parsing/hammer/default.nix:18:3:
```
   |
18 |   meta = with lib; {
   |   ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/development/compilers/llvm/rocm/clang.nix:68:5:
```
   |
68 |     maintainers = with maintainers; [ ];
   |     ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/applications/science/math/maxima/default.nix:106:5:
```
    |
106 |     license = lib.licenses.gpl2;
    |     ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/development/tools/parsing/hammer/default.nix:29:5:
```
   |
29 |     license = licenses.gpl2;
   |     ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/applications/virtualization/lkl/default.nix:65:5:
```
   |
65 |     license     = licenses.gpl2;
   |     ^
```
warning: build-tools-in-build-inputs
yarn is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/development/tools/yarn2nix-moretea/yarn2nix/default.nix:294:7:
```
    |
294 |       buildInputs = [ yarn nodejs rsync ] ++ extraBuildInputs;
    |       ^
```
warning: missing-phase-hooks
checkPhase should probably contain runHook preCheck and runHook postCheck.

Near pkgs/development/libraries/science/biology/mirtk/default.nix:23:3:
```
   |
23 |   checkPhase = ''
   |   ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/tools/filesystems/moosefs/default.nix:54:5:
```
   |
54 |     license = licenses.gpl2;
   |     ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/development/libraries/subunit/default.nix:21:3:
```
   |
21 |   meta = with lib; {
   |   ^
```
warning: build-tools-in-build-inputs
bc is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/applications/misc/lyx/default.nix:16:3:
```
   |
16 |   buildInputs = [
   |   ^
```
warning: build-tools-in-build-inputs
python3.pkgs.wrapPython is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/development/libraries/subunit/default.nix:15:3:
```
   |
15 |   buildInputs = [ check cppunit perl python3Packages.wrapPython ];
   |   ^
```
warning: missing-phase-hooks
buildPhase should probably contain runHook preBuild and runHook postBuild.

Near pkgs/servers/matrix-appservice-discord/default.nix:39:3:
```
   |
39 |   buildPhase = ''
   |   ^
```
warning: unclear-gpl
gpl2 is a deprecated license, please check if project uses gpl2Plus or gpl2Only and change meta.license accordingly.

Near pkgs/development/tools/misc/coccinelle/default.nix:37:5:
```
   |
37 |     license = lib.licenses.gpl2;
   |     ^
```
warning: name-and-version
Did you mean to pass pname instead of name to mkDerivation?

Near pkgs/applications/science/math/maxima/default.nix:17:3:
```
   |
17 |   name = "${name}-${version}";
   |   ^
```
Near pkgs/applications/science/math/maxima/default.nix:16:10:
```
   |
16 |   inherit version;
   |          ^
```
warning: build-tools-in-build-inputs
makeWrapper is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/applications/misc/lyx/default.nix:16:3:
```
   |
16 |   buildInputs = [
   |   ^
```
warning: build-tools-in-build-inputs
rsync is a build tool so it likely goes to nativeBuildInputs, not buildInputs.

Near pkgs/development/tools/yarn2nix-moretea/yarn2nix/default.nix:294:7:
```
    |
294 |       buildInputs = [ yarn nodejs rsync ] ++ extraBuildInputs;
    |       ^
```
warning: missing-phase-hooks
checkPhase should probably contain runHook preCheck and runHook postCheck.

Near pkgs/servers/matrix-appservice-discord/default.nix:45:3:
```
   |
45 |   checkPhase = ''
   |   ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/development/python-modules/subunit/default.nix:17:20:
```
   |
17 |   inherit (subunit) name src meta;
   |                    ^
```
warning: missing-patch-comment
Consider adding a comment explaining the purpose of this patch on the line preceeding.
Near pkgs/games/ja2-stracciatella/default.nix:35:5:
```
   |
35 |     ./remove-rust-buildstep.patch
   |     ^
```
warning: unclear-gpl
lgpl21 is a deprecated license, please check if project uses lgpl21Plus or lgpl21Only and change meta.license accordingly.

Near pkgs/development/libraries/mapnik/default.nix:97:5:
```
   |
97 |     license = licenses.lgpl21;
   |     ^
```
warning: unused-argument
Unused argument: fetchFromGitHub.
Near pkgs/development/compilers/llvm/rocm/clang.nix:2:3:
```
  |
2 | , fetchFromGitHub
  |   ^
```
warning: missing-phase-hooks
installPhase should probably contain runHook preInstall and runHook postInstall.

Near pkgs/applications/virtualization/lkl/default.nix:27:3:
```
   |
27 |   installPhase = ''
   |   ^
```
warning: maintainers-missing
Package does not have a maintainer. Consider adding yourself?

Near pkgs/games/ja2-stracciatella/default.nix:57:3:
```
   |
57 |   meta = {
   |   ^
```
error: no-flags-spaces
makeFlags cannot contain spaces, please use makeFlagsArray in Bash.

Near pkgs/applications/virtualization/lkl/default.nix:47:3:
```
   |
47 |   makeFlags = [
   |   ^
```

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

SuperSandro2000 · 2021-07-28T11:49:48Z

pkgs/development/libraries/mapnik/default.nix

@@ -1,6 +1,6 @@
 { lib, stdenv, fetchzip
 , boost, cairo, freetype, gdal, harfbuzz, icu, libjpeg, libpng, libtiff
-, libwebp, libxml2, proj, python, sqlite, zlib
+, libwebp, libxml2, proj, python3, python ? python3, sqlite, zlib


Suggested change

, libwebp, libxml2, proj, python3, python ? python3, sqlite, zlib

, libwebp, libxml2, proj, python3, sqlite, zlib

This is to separate the python used as build tool from that for which the binding is created.

oh, maybe we should add a comment for this. I am probably not the last one who stumbles upon this.

jonringer

LGTM

collares · 2021-07-31T23:09:16Z

In pkgs/applications/science/math/sage/default.nix, we do

  maxima-ecl = pkgs.maxima-ecl.override {
    ecl = pkgs.ecl.override {
      useBoehmgc = true;
    };
  };

ecl adds boehmgc to its propagatedBuildInputs when useBoehmgc == true. Unfortunately, this overridden version of maxima-ecl does not work anymore. Vanilla maxima-ecl, as expected, uses ecl to compile Lisp files. When ecl was compiled with useBoehmgc == true, this is done by generating C files that use boehmgc's headers and compiling those files with gcc. Since this PR landed, this last step (in the context of building maxima-ecl) fails to find the right headers:

;;;
;;; Compiling /build/maxima-5.44.0/src/maxima-package.lisp.
;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=2
;;;
;;; End of Pass 1.
;;; Internal error:
;;;   ** Error code 1 when executing
;;; (EXT:RUN-PROGRAM "gcc" ("-I." "-I/nix/store/v8k8wdlgcx6dl4a5v1rplsy89dyhlndj-ecl-21.2.1/include/" "-I/nix/store/sipixdr4j94crffvl069mi94v7kyvqzv-gmp-6.2.1-dev/include" "-I/nix/store/w0v6cyzyhws18dahnv94bb0l0h6yf7zv-libffi-3.4.2-dev/include" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-Dlinux" "-O2" "-c" "binary-ecl/maxima-package.c" "-o" "binary-ecl/maxima-package.o")):
;;; In file included from /nix/store/v8k8wdlgcx6dl4a5v1rplsy89dyhlndj-ecl-21.2.1/include/ecl/ecl.h:37,
;;;                  from /nix/store/v8k8wdlgcx6dl4a5v1rplsy89dyhlndj-ecl-21.2.1/include/ecl/ecl-cmp.h:36,
;;;                  from binary-ecl/maxima-package.c:5:
;;; /nix/store/v8k8wdlgcx6dl4a5v1rplsy89dyhlndj-ecl-21.2.1/include/ecl/config.h:79:11: fatal error: gc/gc.h: No such file or directory
;;;    79 | # include "gc/gc.h"
;;;       |           ^~~~~~~~~
;;; compilation terminated.

Adding lisp-compiler to maxima's buildInputs as well as nativeBuildInputs makes things succeed but does not feel like a correct solution. What is a good way to make maxima-ecl with useBoehmgc == true automatically propagate the boehmgc dependency to ecl's users? If anyone wants to reproduce this problem, just run:

nix-build -E 'with (import <nixpkgs> {}); maxima-ecl.override { ecl = ecl.override { useBoehmgc = true; }; }'

collares · 2021-08-01T01:46:37Z

Okay, I think #132269 fixes the above problem :)

arkivm · 2021-11-26T07:07:22Z

appleseed.studio still depends on python2 and there is an open issue for python3 support.
However, freebsd ports successfully migrated to python3 with these set of patches. It's a bit non-trivial though. Tagging appleseed maintainer @hodapp

FRidh added 30 commits July 27, 2021 20:37

blastem: stay with python2

d39767e

appleseed: use python3

1079157

cassandra: stay with python2

e029eba

Did not test with python3.

clasp: use python3

250544f

dtc: python is only used when called from python-packages.nix

6882ab1

osl: use python3

3bb6cb2

seexpr: stay with python2

95cdecf

gdcm: python is only used when called from python-packages.nix

8fab49c

fastnlo_toolkit: python is only used when called from python-packages…

03e4fa0

….nix

boost: python is only used when called from python-packages.nix

f887fbc

ja2-stracciatella: change to python3

edd356d

gem-config: stay with python2

9b2b860

hammer: clean up dependencies

ce7b1fc

coccinelle: change to python3

cf4f947

em-modules: stay with python2

41faf39

Not tested whether it works with python3.

k3d: stay with python2

9070c34

not tested whether python3 works

libsigrok-0-3-0 -> libsigrok_0_3

b793742

libsigrok: python bindings need additional work so remove python

9695d86

libsystemtap: python2 is needed

f969760

subunit: use python3

4e8d1a8

linphone: remove unused python parameter

4735ed6

libtorrent-rasterbar-{1_1_x,1_2_x}: stay with python2

af5e969

libtensorflow: refer to python3

857aba2

lkl: use python3

48d060a

lvtk: stay with python2

a313d4c

lyx: use python3

fce33c3

llvmPackages_rocm: use python3

e95ca01

mapnik: use python3 for tools

53d4f53

but use python for the bindings.

matrix-appservice-discord: use python3

c8dc922

openjfx11: stay with python2

196d3b8

FRidh requested a review from RaghavSood as a code owner July 27, 2021 18:37

github-actions bot added 6.topic: python 6.topic: ruby labels Jul 27, 2021

FRidh added 11 commits July 27, 2021 20:42

mirtk: use python3

dc04d77

mlt: requires python3

afa9c4a

moosefs: use strictDeps and python3

9c9f7fd

python3Packages.subunit: use postPatch

a949d53

munin: stay with python2

6331114

not tested with 3

mygpoclient: use python3 version

56a031d

neard: stay with python2

c7d0306

not tested with python3.

nano-wallet: use python3

80abab6

opencv4: use python3

f16520b

python interpreters: use correct python when

b3b3107

calling pythonPackages.

mininet: use python3

238fa78

FRidh force-pushed the python2-to-3 branch from 2c2e095 to 238fa78 Compare July 27, 2021 18:43

ofborg bot added 8.has: clean-up 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 501+ 10.rebuild-linux: 501-1000 labels Jul 27, 2021

SuperSandro2000 reviewed Jul 28, 2021

View reviewed changes

jonringer approved these changes Jul 28, 2021

View reviewed changes

FRidh merged commit 8dcfe52 into NixOS:master Jul 29, 2021

collares mentioned this pull request Aug 1, 2021

ecl: make sure boehmgc is available to gcc/linker #132269

Merged

11 tasks

arkivm mentioned this pull request Nov 26, 2021

appleseed: migrate to boost168 #147458

Closed

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: further migrate packages from 2 to 3 #131719

Python: further migrate packages from 2 to 3 #131719

FRidh commented Jul 27, 2021 •

edited

Loading

r-rmcgibbo commented Jul 28, 2021 •

edited

Loading

SuperSandro2000 Jul 28, 2021

FRidh Jul 28, 2021

SuperSandro2000 Jul 28, 2021

jonringer left a comment

collares commented Jul 31, 2021 •

edited

Loading

collares commented Aug 1, 2021

arkivm commented Nov 26, 2021

	, libwebp, libxml2, proj, python3, python ? python3, sqlite, zlib
	, libwebp, libxml2, proj, python3, sqlite, zlib

Python: further migrate packages from 2 to 3 #131719

Python: further migrate packages from 2 to 3 #131719

Conversation

FRidh commented Jul 27, 2021 • edited Loading

Motivation for this change

Things done

r-rmcgibbo commented Jul 28, 2021 • edited Loading

SuperSandro2000 Jul 28, 2021

Choose a reason for hiding this comment

FRidh Jul 28, 2021

Choose a reason for hiding this comment

SuperSandro2000 Jul 28, 2021

Choose a reason for hiding this comment

jonringer left a comment

Choose a reason for hiding this comment

collares commented Jul 31, 2021 • edited Loading

collares commented Aug 1, 2021

arkivm commented Nov 26, 2021

FRidh commented Jul 27, 2021 •

edited

Loading

r-rmcgibbo commented Jul 28, 2021 •

edited

Loading

collares commented Jul 31, 2021 •

edited

Loading