-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpython-virtual-envs.html
178 lines (158 loc) · 12.5 KB
/
python-virtual-envs.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
<!DOCTYPE html>
<html lang="en" prefix="og: http://ogp.me/ns#">
<head>
<link href="http://gmpg.org/xfn/11" rel="profile">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<!-- Metadata -->
<meta name="description" content="">
<meta property="og:description" content="">
<meta property="og:title" content="Python Virtual Envs" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://darrenwiens.github.io/python-virtual-envs.html" />
<meta property="og:image" content="https://darrenwiens.github.io/images/face.png" />
<!-- Enable responsiveness on mobile devices-->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1">
<title>Darren Wiens</title>
<!-- CSS -->
<link href="//fonts.googleapis.com/" rel="dns-prefetch">
<link href="//fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic|Abril+Fatface|PT+Sans:400,400italic,700&subset=latin,latin-ext" rel="stylesheet">
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/poole.css" />
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/hyde.css" />
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/syntax.css" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/fork-awesome.min.css" crossorigin="anonymous">
<!-- Feeds -->
<!-- Analytics -->
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-9664J8VT36"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-9664J8VT36');
</script>
<!-- Ads -->
<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-2681534008399266"
crossorigin="anonymous"></script>
</head>
<body class="theme-base-08">
<div class="sidebar">
<div class="container sidebar-sticky">
<div class="sidebar-about">
<h1>
<a href="/">
<img class="profile-picture" src="https://darrenwiens.github.io/images/face.png">
Darren Wiens
</a>
</h1>
<p class="lead"></p>
<p class="lead"> </p>
<p></p>
</div>
<ul class="sidebar-nav">
</ul>
<nav class="sidebar-social">
<a class="sidebar-social-item" href="http://twitter.com/dkwiens" target="_blank">
<i class="fa fa-twitter"></i>
</a>
<a class="sidebar-social-item" href="https://fosstodon.org/@dkwiens" target="_blank">
<i class="fa fa-mastodon"></i>
</a>
<a class="sidebar-social-item" href="http://github.com/darrenwiens" target="_blank">
<i class="fa fa-github"></i>
</a>
<a style="display:none" class="sidebar-social-item" href="https://darrenwiens.github.io/None">
<i class="fa fa-rss"></i>
</a>
<a rel="me" style="display:none" href="https://fosstodon.org/@dkwiens">Mastodon</a>
</nav>
</div>
</div> <div class="content container">
<div class="post">
<h1 class="post-title">Python Virtual Envs</h1>
<span class="post-date">Mon 18 April 2022</span>
<p><strong>Disclaimer: I fully admit I'm not a detail-oriented person, and as such, I'm satisfied when things work, not necessarily when things work the right way or even why they do so. Like most Python topics, there is more than one way to make or choose a Python virtual environment, and you should do what's right for you, and, ideally, leave me out of it.</strong></p>
<p>Let's set the scene. You're a relatively new Python user, happily writing amazing scripts that automate those button clicks that seem absurd in hindsight. You've ventured beyond core Python, pip installing third party modules with glee. You've even got multiple projects going, using a mix of shared and exclusive modules. It works great for a while, perhaps years! But, the time comes when one project requires X module version A, and another project requires X module version B. Without some fancy footwork, you can only have one version of each module installed at a time.</p>
<p align="center">
<img src="https://darrenwiens.github.io/images/classic_blunders.gif" />
</p>
<p>You've fallen victim to one Python's classic blunders: never install modules globally. Rather, use a <strong>virtual environment</strong>. No one tells you this until it's too late, sorry to say.</p>
<p>You can think of virtual environments as a self-contained bundle of Python plus a segregated set of modules. Unfortunately, or luckily, depending on your perspective, there are several options for creating virtual environments. This blog post is intended to serve as a starting point for avoiding <a href="https://en.wikipedia.org/wiki/Overchoice">choice overload</a> when it comes to creating a virtual environment. It is simply the way I do it, and you can do whatever you want in order to segregate your Python environments. You can find some basic advice at the end of each section, but just know that there are many ways to solve this problem, with a variety of tools.</p>
<p align="center">
<img src="https://darrenwiens.github.io/images/python_env_flowchart.png" />
</p>
<h2>1. Use recommended tool</h2>
<p>Especially when you're starting out, trust repo owners to help you choose the best environment for a project. There may be a good reason they recommend a certain method. If they recommend poetry, try to use poetry. If there is no recommendation, go ahead and choose something else.</p>
<h2>2. Use venv</h2>
<p><a href="https://docs.python.org/3/library/venv.html">venv</a> is what I assume most people mean when they say "virtual environment". The usage is like:</p>
<div class="highlight"><pre><span></span><code>$ python3 -m venv .venv <span class="c1"># create a new env in a directory called ".venv"</span>
$ <span class="nb">source</span> .venv/bin/activate <span class="c1"># activate the env in the ".venv" directory</span>
</code></pre></div>
<p>The first command creates a new environment, including, among other things, a Python binary, a known directory in which new modules will be installed (e.g. <code>.venv/lib/python3.8/site-packages/</code>), and a method to install new packages (<code>pip</code>). Once you <strong>activate</strong> the environment with the second command, all packages installed with <strong>that pip</strong> will be places in <strong>that site-packages</strong> directory.</p>
<p>venv is an excellent tool to spend time learning. It is my most used environment type, and good for situations in which:</p>
<ul>
<li>you want to create the environment quickly</li>
<li>you're not particular about exactly which version of Python will be used (although it is possible to control)</li>
<li>you don't need environment-specific environment variables (although this is achievable)</li>
<li>you're happy with manually activating the environment each time</li>
<li>you don't intend to reuse the environment elsewhere</li>
<li>all of the desired libraries are installable with pip</li>
</ul>
<h2>3. Use direnv</h2>
<p>If you like venv, you'll love <a href="https://direnv.net/">direnv</a>. Once you've installed direnv, the usage is like:</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">echo</span> <span class="s1">'layout_python3'</span> > .envrc <span class="c1"># create a file ".envrc" containing the text "layout_python3"</span>
$ direnv allow <span class="c1"># refresh the direnv to acknowledge the change</span>
</code></pre></div>
<p>The first command creates a <code>.envrc</code> file with a quick instruction for direnv that this is for python 3. The second command prompts direnv to acknowledge that this directory contains a <code>.envrc</code> and get to work building a new environment. In essence, a new venv is created in the directory, but also, because direnv recognizes this directory the environment will automatically be activated on entry and deactivated on exit. Handy! In addition, you can add <code>export</code> and other commands to the <code>.envrc</code>, allowing you to load and unload environment variables for this directory only.</p>
<p>direnv is great in situations where:</p>
<ul>
<li>you want to automatically activate/deactivate the environment on entry/exit</li>
<li>you want a simple way to use directory-specific environment variables</li>
<li>you want all the other benefits (and drawbacks) of venv</li>
</ul>
<h2>4. Use conda</h2>
<p><a href="https://docs.conda.io/en/latest/">conda</a> is another very popular environment manager (like venv) and also functions as a package manager (like pip). Once installed, conda usage is like:</p>
<div class="highlight"><pre><span></span><code>$ conda create -n env_name <span class="c1"># create a new conda environment called "env_name"</span>
$ conda activate env_name <span class="c1"># activate environment</span>
</code></pre></div>
<p>Similar to venv, the first command creates the environment and the second activates it. Unlike venv, conda environments are created and accessible <strong>globally</strong>, meaning you can easily activate conda environments wherever they are needed. New packages are then installed through <code>conda install</code> rather than pip, although you can also install packages using pip. The nice thing about conda environments is that all conda-installed packages are reconciled against each other to ensure compatibility. However, the annoying thing is that this reconciliation can take...</p>
<p>...f o r e v e r.</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I was going to use this in my slides but lately one cannot even `conda install gdal` from conda-forge :-( <a href="https://t.co/60XW5dIytN">pic.twitter.com/60XW5dIytN</a></p>— Filipe Fernandes (I am 😷 under that 🪣) (@ocefpaf) <a href="https://twitter.com/ocefpaf/status/753992589938860032?ref_src=twsrc%5Etfw">July 15, 2016</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>(Ignore the text in the tweet above, I'm pretty sure it works in conda now)</p>
<p>A major reason for geospatial developers to consider conda environments is that <a href="https://gdal.org/">GDAL</a> <a href="https://gdal.org/api/python.html">python bindings</a> are notoriously difficult to install correctly through pip. If you run into issues creating a functional environment containing GDAL python bindings, consider installing in a conda environment with <code>conda install -c conda-forge gdal</code>.</p>
<p>conda definitely has its place, and that is where:</p>
<ul>
<li>you don't mind how long it takes to create the environment</li>
<li>you want to ensure compatibility between all installed modules</li>
<li>you want to share a single environment between multiple projects</li>
<li>you want easy control over the python version (e.g. add <code>python=3.8</code> to environment creation command)</li>
</ul>
<h2>5. Use mamba</h2>
<p><a href="https://mamba.readthedocs.io/en/latest/index.html">mamba</a> is a newer spin-off of conda. I haven't used it much, but the idea of mamba is that it behaves like conda, only faster. You do need to have both conda and mamba installed in order to use mamba. To create a mamba environment:</p>
<div class="highlight"><pre><span></span><code>$ mamba create -n env_name <span class="c1"># create a new mamba environment called "env_name"</span>
$ conda activate env_name <span class="c1"># activate environment with conda</span>
</code></pre></div>
<p>As mentioned, I don't have much to say about mamba, but if your conda environment is taking too long to build, keep mamba in mind.</p>
<h2>Use something else entirely</h2>
<p>Other popular environment management tools that I have no real opinion about include:</p>
<ul>
<li><a href="https://python-poetry.org/">poetry</a></li>
<li><a href="https://pipenv.pypa.io/en/latest/">pipenv</a></li>
</ul>
<p>That's all I've got. Enjoy you virtual environments! Get in touch with me on <a href="https://twitter.com/dkwiens">Twitter</a> if you want to continue the discussion!</p>
<span class="post-tags">
Tags:
<ul>
<li><a href="https://darrenwiens.github.io/tag/python.html">python</a></li>
<li><a href="https://darrenwiens.github.io/tag/virtual-environments.html">virtual environments</a></li>
<li><a href="https://darrenwiens.github.io/tag/venv.html">venv</a></li>
<li><a href="https://darrenwiens.github.io/tag/conda.html">conda</a></li>
<li><a href="https://darrenwiens.github.io/tag/direnv.html">direnv</a></li>
<li><a href="https://darrenwiens.github.io/tag/mamba.html">mamba</a></li>
</ul>
</span>
</div>
</div>
</body>
</html>