-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.xml
552 lines (501 loc) · 63.7 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>AHR</title>
<link>https://alexhermida.dev/</link>
<description>Recent content on AHR</description>
<generator>Hugo -- gohugo.io</generator>
<language>en-en</language>
<copyright>2020 - @alexhermida</copyright>
<lastBuildDate>Tue, 09 May 2023 00:00:00 +0000</lastBuildDate>
<atom:link href="https://alexhermida.dev/index.xml" rel="self" type="application/rss+xml" />
<item>
<title>Building a CLI with SSO</title>
<link>https://alexhermida.dev/posts/building-cli-with-sso/</link>
<pubDate>Tue, 09 May 2023 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/posts/building-cli-with-sso/</guid>
<description><p>👋 Let&rsquo;s talk about adding SSO to our CLIs, something really useful for different reasons. Single Sign-On (SSO) is not only a common practice nowadays in organisations but is almost a must.</p>
<p>Although it looks challenging for many people at first glance, nowadays the integration has been simplified thanks to the different services that help to abstract you from the complexity of the different flows. When it comes to the benefits of using SSO, there are many but some are very specific for CLI applications.</p>
<ol>
<li>Simplifies the authentication process. CLI applications usually run locally in the user&rsquo;s computer which means they&rsquo;re probably already using SSO for other services with an active session.</li>
<li>Allow users to take advantage of a central identity provider (IdP), which could be any of the frequent mainstream social providers like Google, Microsoft, Github or any other over OIDC or SAML protocols.</li>
<li>Increases security as the user doesn&rsquo;t need to maintain long-lived secret keys, tokens or credentials for authentication.</li>
<li>Allow better integration within other systems and services in the organisation.</li>
</ol>
<p>those, among others, are some of the benefits but each use case it&rsquo;s different,
so what makes CLI session flow different?</p>
<h2 id="some-context">Some context</h2>
<p>Whereas in a regular application which lives in the cloud the authorisation flow is &ldquo;simpler&rdquo;, for a CLI application, where the source code lives in the user&rsquo;s computer and you can&rsquo;t exchange a secret, a relatively new flow was defined within the <a href="https://datatracker.ietf.org/doc/html/rfc8628"target="_blank" rel="noopener noreferrer">OAuth 2.0</a>
protocol. The &ldquo;Device authorisation grant&rdquo;.</p>
<p>This new grant defines a secondary &ldquo;device&rdquo; apart from the client where the application is executed. The user manually grants access through that &ldquo;device&rdquo;.</p>
<pre><code class="language-diagram" data-lang="diagram"> +----------+ +----------------+
| |&gt;---(A)-- Client Identifier ---&gt;| |
| | | |
| |&lt;---(B)-- Device Code, ---&lt;| |
| | User Code, | |
| Device | &amp; Verification URI | |
| Client | | |
| | [polling] | |
| |&gt;---(E)-- Device Code ---&gt;| |
| | &amp; Client Identifier | |
| | | Authorization |
| |&lt;---(F)-- Access Token ---&lt;| Server |
+----------+ (&amp; Optional Refresh Token) | |
v | |
: | |
(C) User Code &amp; Verification URI | |
: | |
v | |
+----------+ | |
| End User | | |
| at |&lt;---(D)-- End user reviews ---&gt;| |
| Browser | authorization request | |
+----------+ +----------------+
Figure 1: Device Authorization Flow
</code></pre><h2 id="coding-time">Coding time</h2>
<p>For learning purpose, I created a new CLI tool with a minimum configuration, and
for this specific example I used <strong>Auth0</strong> as the Identity Provider (IdP), but you can easily
configure it for use with other providers like <strong>Okta, Amazon Cognito or others.</strong></p>
<blockquote>
<p>If you want to run and test it you&rsquo;d need at least the following requirements:</p>
<ul>
<li>to create you own <strong>Navite application</strong>
in <a href="https://auth0.com/docs/get-started/auth0-overview/create-applications/native-apps"target="_blank" rel="noopener noreferrer">auth0</a>
</li>
<li>Enable social or a database connection</li>
<li><a href="https://auth0.com/docs/get-started/architecture-scenarios/mobile-api/part-2#create-the-api"target="_blank" rel="noopener noreferrer">Register your API</a>
</li>
</ul>
<p>With that, you&rsquo;d have the parameters needed for the OAuth flow, like client_id or audience.</p>
</blockquote>
<h3 id="request-device-code">Request device code</h3>
<p>As described in the flow diagram, the first step is to request a temporary device code that can be used to prompt the
user for device activation. For that, the IdP give us an endpoint where we can request that device code using
the audience and client_id parameters.</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">request_device_code</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">DeviceCodeResponse</span><span class="p">:</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;content-type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/x-www-form-urlencoded&#34;</span><span class="p">}</span>
<span class="n">payload</span> <span class="o">=</span> <span class="n">f</span><span class="s2">&#34;client_id={_APP_CLIENT_ID}&amp;audience={_AUDIENCE}&#34;</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
<span class="n">f</span><span class="s2">&#34;{_TENANT}/oauth/device/code&#34;</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span>
<span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
<span class="n">device_code_info</span> <span class="o">=</span> <span class="n">DeviceCodeResponse</span><span class="p">(</span><span class="o">**</span><span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">())</span>
<span class="n">_user_authorization_flow</span><span class="p">(</span><span class="n">device_code_info</span><span class="o">=</span><span class="n">device_code_info</span><span class="p">)</span>
<span class="k">return</span> <span class="n">device_code_info</span>
</code></pre></div><h3 id="user-device-activation">User device activation</h3>
<p>The <code>device code response</code> includes different information for enabling the user to activate and manage the CLI session,</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="nd">@dataclass</span>
<span class="k">class</span> <span class="nc">DeviceCodeResponse</span><span class="p">:</span>
<span class="n">device_code</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">user_code</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">verification_uri</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">expires_in</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">interval</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">verification_uri_complete</span><span class="p">:</span> <span class="nb">str</span>
</code></pre></div><p>For this use case, the most relevant is the <code>user_code</code> along with the <code>verification_uri</code>, the user can copy&amp;paste and go the
URL or just directly open <code>verification_uri_complete</code> in a browser to manually activate the CLI. For this step, the user
needs to be logged in with the appropriate credentials, either a social login (google, github, etc) or a database connection.</p>
<h3 id="request-access-token">Request access token</h3>
<p>The CLI needs to request and store the access token, meanwhile, the user is manually validating the device, and the CLI is making
polling to the token URL provided by the IdP on a defined interval.</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">_request_access_token</span><span class="p">(</span><span class="n">code</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;content-type&#34;</span><span class="p">:</span> <span class="s2">&#34;application/x-www-form-urlencoded&#34;</span><span class="p">}</span>
<span class="n">payload</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">f</span><span class="s2">&#34;grant_type=urn:ietf:params:oauth:grant-type:device_code&amp;&#34;</span>
<span class="n">f</span><span class="s2">&#34;device_code={code}&amp;client_id={_APP_CLIENT_ID}&#34;</span>
<span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">f</span><span class="s2">&#34;{_TENANT}/oauth/token&#34;</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">payload</span><span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
<span class="k">return</span> <span class="n">response</span>
<span class="k">def</span> <span class="nf">poll_user_verification</span><span class="p">(</span><span class="n">code</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">interval_seconds</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ReceivedCredentials</span> <span class="o">|</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">interval_seconds</span><span class="p">)</span>
<span class="n">valid_response</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">valid_response</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">_request_access_token</span><span class="p">(</span><span class="n">code</span><span class="p">)</span>
<span class="n">valid_response</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">return</span> <span class="n">ReceivedCredentials</span><span class="p">(</span><span class="o">**</span><span class="n">response</span><span class="o">.</span><span class="n">json</span><span class="p">())</span>
<span class="k">except</span> <span class="n">requests</span><span class="o">.</span><span class="n">HTTPError</span> <span class="k">as</span> <span class="n">http_error</span><span class="p">:</span>
<span class="k">if</span> <span class="n">http_error</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">403</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s2">&#34;Retrying device code credentials request&#34;</span><span class="p">)</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">interval_seconds</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">http_error</span>
</code></pre></div><p>Once the user has successfully authorised the CLI you get a 200 status code with the <code>access_token</code> in the payload along with
additional parameters depending on the API configuration.</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"><span class="nd">@dataclass</span>
<span class="k">class</span> <span class="nc">ReceivedCredentials</span><span class="p">:</span>
<span class="n">access_token</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">expires_in</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">token_type</span><span class="p">:</span> <span class="nb">str</span>
</code></pre></div><h3 id="call-your-api-with-the-credentials">Call your API with the credentials</h3>
<p>The <code>access_token</code>is a JWT (JSON Web Token) that follows the <a href="https://datatracker.ietf.org/doc/html/rfc7519"target="_blank" rel="noopener noreferrer">RFC7519</a>
and you
can validate and verify for providing access to your APIs and resources. For that, you can call your API passing the access_token
as a bearer token in the request header.</p>
<h2 id="sumary">Sumary</h2>
<p>The code above implements a basic solution for SSO in your CLI, of course, there are different missing pieces like <code>refresh_tokens</code>,
managing access through <code>scopes</code> or handling <code>profiles</code> which could help to understand better how this work in a corporate
environment.</p>
<p>For reference, if you want to follow up on the previous examples, all code snippets present in this post are available at:
<a href="https://github.com/alexhermida/sample_sso_cli">https://github.com/alexhermida/sample_sso_cli</a></p>
<p>Cheers!</p></description>
</item>
<item>
<title>Retrospective 2019-2020</title>
<link>https://alexhermida.dev/posts/retrospective-2019-2020/</link>
<pubDate>Wed, 02 Sep 2020 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/posts/retrospective-2019-2020/</guid>
<description><p>Hello!</p>
<p>Usually, this is the typical post people write in December or in January. But for me, it has more sense to do it now, not only because everybody is hoping to come back to some &ldquo;kind of&rdquo; normality (ending the summer, new school-year&hellip;) but also exactly one year ago I was starting a new adventure in Cambridge.</p>
<p>2018/2019 was a period of change, I ended a project of life, Initios, meanwhile, I&rsquo;d been working in an awesome project, Taiga. Also learning, preparing interviews and looking for a new challenge after many years. That challenge appeared, and I flew to Cambridge on the 31st of August 2019.</p>
<p>I did in the past, and I still do, different retrospective activities as a team so I&rsquo;ll give it a try for my personal scope 😁</p>
<h2 id="what-went-well">What went well?</h2>
<p>The Cambridge experience was remarkable, I&rsquo;ve been there whole September, and after that, I went every month for 4 to 5 days until last February. Beautiful place, with incredible history and lovely people.</p>
<p>Healx values and culture, I found what I would expect, extremely overturned with the company goals, you can feel the aim of having an impact on people.</p>
<p>I&rsquo;m surrounded by scientist and expertise people, but everyone is willing to share and always open to your doubts and to explain it better.</p>
<p><strong>Focus,</strong> last but not least. I think we all underestimate the power of focus even if we know deeply how important it is, I did it all my life. One project, shared goals, shared responsibility, stability, no side-projects. It worth it.</p>
<h2 id="what-didnt-go-so-well">What didn’t go so well?</h2>
<p>I feel that in some areas, like the domain, my progress is too slow. I&rsquo;m not aware of the best way of improving that, apart from participating more in the meetings and asking more questions. The reality is that I also want to deliver, learning technologies and the stack so it&rsquo;s an extra thing not easy to improve without specific actions.</p>
<p>The company needs to scale <em><strong>a lot</strong>,</em> some growth pain and not easy, nothing unexpected I think.</p>
<p>I guess COVID-19 it&rsquo;s the big thing, it affected all of us in multiple and different ways and here in Spain, it wasn&rsquo;t easy to cope with fifty days hard lockdown. Many things had changed last months and we still don&rsquo;t know how the &ldquo;new normality&rdquo; will be.</p>
<h2 id="what-have-i-learned">What have I learned?</h2>
<p>I learned new technologies, like React, TypesScript and GraphQL and I&rsquo;ve got used to functional programming paradigm.</p>
<p>I got into some new AWS tools. I had some experience working with EC2 and S3, but now I can say I feel somehow comfortable with things like ECS, Fargate, RDS or ELB.</p>
<p>I learn <strong>every</strong> day about the domain, I would say I need 6 more lives to have enough understanding.</p>
<p>I learned there are other ways to manage a team, especially the well-being of it. I worked in different places and I&rsquo;ve never felt so covered like in Healx.</p>
<h2 id="what-still-puzzles-me">What still puzzles me?</h2>
<p>My English; somehow I would expect my English would improve much more but that wasn&rsquo;t the case. I&rsquo;m aware my understanding and listening improved a lot but is not the case with my oral expression. Something to address and improve.</p>
<p>Still not comfortable with some of the stack, I&rsquo;m willing to keep improving and focusing.</p>
<h2 id="outcome">Outcome</h2>
<p>That&rsquo;s it, I think this is a good exercise, I came up with things I hadn&rsquo;t realised!</p>
<ul>
<li>Improve my English speaking and vocabulary</li>
<li>Keep focus</li>
<li>Keep learning</li>
</ul>
<p>Probably, it&rsquo;s time to get some actions from them. I guess I need a new post for my planning for 2020/2021 😏</p>
<p>Cheers!</p></description>
</item>
<item>
<title>Quick overview for securing a backend API with Google App Engine and Auth0.</title>
<link>https://alexhermida.dev/posts/create-app-engine-service-authenticated-with-auth0/</link>
<pubDate>Sat, 06 Jun 2020 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/posts/create-app-engine-service-authenticated-with-auth0/</guid>
<description><p>Google App Engine it&rsquo;s a good product for quick API deployment and very easy to integrate with Auth0 for autentication.
I&rsquo;ll go through the basics concepts for deploying an API backend in Python.</p>
<p>Auth0 is a cloud authentication and authorisation service which solves for you the complexity of identity managament.</p>
<p>This overview is focus in giving some additional explantions or key points that may help you with your specific use case.
It is complementary to what appears in the official &ldquo;How-to&rdquo; guides.</p>
<ul>
<li><a href="https://cloud.google.com/endpoints/docs/openapi/authenticating-users-auth0">https://cloud.google.com/endpoints/docs/openapi/authenticating-users-auth0</a></li>
<li><a href="https://auth0.com/docs/integrations/google-cloud-platform">https://auth0.com/docs/integrations/google-cloud-platform</a></li>
</ul>
<p>Prerequisites:</p>
<ul>
<li>Create a Google Cloud project (<a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">https://cloud.google.com/resource-manager/docs/creating-managing-projects</a>)</li>
<li>Enable billing for the project (<a href="https://cloud.google.com/billing/docs/how-to/modify-project">https://cloud.google.com/billing/docs/how-to/modify-project</a>)</li>
<li>Some Google services enabled (<a href="https://cloud.google.com/endpoints/docs/quickstart-endpoints#enabling_required_services">https://cloud.google.com/endpoints/docs/quickstart-endpoints#enabling_required_services</a>)</li>
</ul>
<p>For this <em>overview</em> will only be necessary to use two products from Google Cloud Platform (GCP): <a href="https://cloud.google.com/appengine"target="_blank" rel="noopener noreferrer">App Engine</a>
and <a href="https://cloud.google.com/endpoints"target="_blank" rel="noopener noreferrer">Endpoints</a>
.</p>
<p>An important thing to take into account is the differences between a <strong><em>flexible</em></strong> and a <strong><em>standard</em></strong> environment in Google App Engine. I&rsquo;ll be
focus on enabling it for a <em>flexible</em> environment. The <em>Endpoints</em> product for <em>Standard</em> environment is still in Beta and it requires many
additional configurations for making it work with Auth0.</p>
<ol>
<li>Deploying backend API</li>
</ol>
<p>You can deploy your API or application before the Endpoints configuration so you can test it without any authentication. There are
many boilerplates from the <a href="https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible"target="_blank" rel="noopener noreferrer">Google documantation</a>
</p>
<p>You could just create your basic sample with any framework like Flask, FastAPI or Django.</p>
<ul>
<li>
<p>main.py: entrypoint/executable</p>
<div class="highlight"><pre class="chroma"><code class="language-python" data-lang="python"> <span class="kn">from</span> <span class="nn">fastapi</span> <span class="kn">import</span> <span class="n">FastAPI</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">FastAPI</span><span class="p">()</span>
<span class="nd">@app.get</span><span class="p">(</span><span class="s2">&#34;/getText/{text}&#34;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_text</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">&#34;Text&#34;</span><span class="p">:</span> <span class="n">text</span><span class="p">}</span>
<span class="nd">@app.get</span><span class="p">(</span><span class="s2">&#34;/notAuthenticated&#34;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">not_authenticated</span><span class="p">():</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">&#34;Endpoint without authentication&#34;</span><span class="p">}</span>
<span class="nd">@app.get</span><span class="p">(</span><span class="s2">&#34;/&#34;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">root</span><span class="p">():</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">&#34;Root without authentication&#34;</span><span class="p">}</span>
</code></pre></div></li>
<li>
<p>requirements.txt: Requirements for your app. Take into account that GCloud uses only <code>requirements.txt</code> file
for the Python runtime. <a href="https://cloud.google.com/appengine/docs/flexible/python/using-python-libraries#declaring_and_managing_dependencies"target="_blank" rel="noopener noreferrer">Additional info</a>
</p>
<div class="highlight"><pre class="chroma"><code class="language-txt" data-lang="txt"> click==7.1.2
fastapi==0.55.1
gunicorn==20.0.4
h11==0.9.0
httptools==0.1.1
pydantic==1.5.1
starlette==0.13.2
uvicorn==0.11.5
uvloop==0.14.0
websockets==8.1
</code></pre></div></li>
<li>
<p>app.yml: It is your App Engine settings file for each service you deploy (<a href="https://cloud.google.com/appengine/docs/standard/python3/config/appref">https://cloud.google.com/appengine/docs/standard/python3/config/appref</a>)</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="w"> </span><span class="k">runtime</span><span class="p">:</span><span class="w"> </span>python<span class="w">
</span><span class="w"> </span><span class="k">env</span><span class="p">:</span><span class="w"> </span>flex<span class="w">
</span><span class="w"> </span><span class="k">runtime_config</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">python_version</span><span class="p">:</span><span class="w"> </span><span class="m">3</span><span class="w">
</span><span class="w"> </span><span class="k">entrypoint</span><span class="p">:</span><span class="w"> </span>gunicorn<span class="w"> </span>-w<span class="w"> </span><span class="m">4</span><span class="w"> </span>-k<span class="w"> </span>uvicorn.workers.UvicornWorker<span class="w"> </span>main<span class="p">:</span>app<span class="w">
</span><span class="w"> </span><span class="k">service</span><span class="p">:</span><span class="w"> </span>test-auth0<span class="w">
</span></code></pre></div></li>
</ul>
<p>After you already tested the running API you can procced to the Endpoints configuration.</p>
<ol start="2">
<li>Deploy Cloud Endpoints configuration</li>
</ol>
<p>Cloud Enpoints uses ESP (Extensible Service Proxy) which allows to serve the API&rsquo;s. We must deploy OpenAPI document to Service Management for configuring the endpoints.</p>
<p>OpenApi file for configuring the endpoints:</p>
<ul>
<li>openapi-appengine.yml</li>
</ul>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="k">swagger</span><span class="p">:</span><span class="w"> </span><span class="s1">&#39;2.0&#39;</span><span class="w">
</span><span class="w"></span><span class="k">info</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">title</span><span class="p">:</span><span class="w"> </span>Test<span class="w"> </span>auth0<span class="w">
</span><span class="w"> </span><span class="k">version</span><span class="p">:</span><span class="w"> </span><span class="m">1.0.0</span><span class="w">
</span><span class="w"></span><span class="k">host</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{GCLOUD_PROJECT_ID}.appspot.com&#34;</span><span class="w">
</span><span class="w"></span><span class="k">consumes</span><span class="p">:</span><span class="w">
</span><span class="w"></span>- <span class="s2">&#34;application/json&#34;</span><span class="w">
</span><span class="w"></span><span class="k">produces</span><span class="p">:</span><span class="w">
</span><span class="w"></span>- <span class="s2">&#34;application/json&#34;</span><span class="w">
</span><span class="w"></span><span class="k">schemes</span><span class="p">:</span><span class="w">
</span><span class="w"></span>- <span class="s2">&#34;http&#34;</span><span class="w">
</span><span class="w"></span>- <span class="s2">&#34;https&#34;</span><span class="w">
</span><span class="w"></span><span class="k">paths</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="s2">&#34;/getText/{text}&#34;</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">get</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Get text&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">operationId</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;getTextInUri&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">parameters</span><span class="p">:</span><span class="w">
</span><span class="w"> </span>- <span class="k">name</span><span class="p">:</span><span class="w"> </span>text<span class="w">
</span><span class="w"> </span><span class="k">in</span><span class="p">:</span><span class="w"> </span>path<span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span>Get<span class="w"> </span>text<span class="w">
</span><span class="w"> </span><span class="k">required</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="w"> </span><span class="k">type</span><span class="p">:</span><span class="w"> </span>string<span class="w">
</span><span class="w"> </span><span class="k">responses</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">200</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Success.&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">schema</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">type</span><span class="p">:</span><span class="w"> </span>string<span class="w">
</span><span class="w"> </span><span class="k">400</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Forbidden access&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">security</span><span class="p">:</span><span class="w">
</span><span class="w"> </span>- <span class="k">auth0_jwt</span><span class="p">:</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="w"> </span><span class="s2">&#34;/notAuthenticated&#34;</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">get</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Not authenticated endpoint&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">operationId</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;notAuth&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">responses</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">200</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Success.&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">400</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">description</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;Forbidden access&#34;</span><span class="w">
</span><span class="w"></span><span class="k">securityDefinitions</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">auth0_jwt</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">authorizationUrl</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://{AUTH0_DOMAIN}/authorize&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">flow</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;implicit&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">type</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;oauth2&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">x-google-issuer</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://{AUTH0_DOMAIN}/&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">x-google-jwks_uri</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://{AUTH0_DOMAIN}/.well-known/jwks.json&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">x-google-audiences</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;https://{AUTH0_API_ID}/&#34;</span><span class="w">
</span></code></pre></div><p>The file includes two endpoints, only the <code>getText</code> one is authenticated. Also, is worth mentioning that the root endpoint
is not available in Google Endpoints. Check <a href="https://cloud.google.com/endpoints/docs/openapi/openapi-limitations#operations_on_url_root_path_"target="_blank" rel="noopener noreferrer">limitations and unsupported features</a>
.</p>
<p>Deploy Endpoints service configuration:</p>
<p><code>gcloud endpoints services deploy openapi-appengine.yml</code></p>
<p>Additional info: <a href="https://cloud.google.com/endpoints/docs/openapi/architecture-overview">https://cloud.google.com/endpoints/docs/openapi/architecture-overview</a></p>
<p>You must add to your <code>app.yml</code> the configuration for link your proyect to the Cloud Endpoints service configuration</p>
<div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="k">endpoints_api_service</span><span class="p">:</span><span class="w">
</span><span class="w"> </span><span class="k">name</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{GCLOUD_PROJECT_ID}.appspot.com&#34;</span><span class="w">
</span><span class="w"> </span><span class="k">rollout_strategy</span><span class="p">:</span><span class="w"> </span>managed<span class="w">
</span></code></pre></div><p>You can do any update to your OpenApi without the need of reloading the service.</p>
<p>I hope this help you!</p>
<p>Cheers.</p></description>
</item>
<item>
<title>Start over</title>
<link>https://alexhermida.dev/posts/first-post/</link>
<pubDate>Thu, 07 May 2020 20:34:11 +0200</pubDate>
<guid>https://alexhermida.dev/posts/first-post/</guid>
<description><p>Hello!</p>
<p>Today, I start over with a personal blog after years of silence. I guess never is late for doing it. I have said <em>start over</em> because I had a blog in the past, but that&rsquo;s another story.</p>
<p>My aim&rsquo;s just to keep a personal reference for experiences and consolidate new knowledge while I share them with you. I&rsquo;ll be talking mainly about tech and related stuff.</p>
<p>If you came from the future and you found here valuable content or my future me does, it&rsquo;s more than enough.</p>
<p>Why do I write in English? Selfishly, to improve my communicating skills.</p>
<p>Cheers.</p></description>
</item>
<item>
<title>Scaling PostgreSQL from a developer point of view.</title>
<link>https://alexhermida.dev/posts/developing-with-postgres/</link>
<pubDate>Mon, 10 Jun 2019 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/posts/developing-with-postgres/</guid>
<description><p>As a developer, you have to deal with databases from time to time in your work-life, and if you are a web developer probably making queries, it is something you are facing daily.</p>
<p>Since I&rsquo;m not a database administrator nor a PostgreSQL expert I only want to give some small tips if you are dealing with scaling issues or starting with PostgreSQL and you want to know more about some underlying processes.</p>
<p>Things you must take into account when you add a query to your code:</p>
<ul>
<li>Indexes</li>
<li>Explain (planner)</li>
<li>Vaccum</li>
</ul>
<h2 id="indexes">Indexes</h2>
<p><code>CREATE INDEX</code></p>
<p>Probably this is one of the things that every developer take care of when is trying to improve a query or implementing some complex one. Regardless
<em>indexes</em> is one of the most important things in a query plan; it also can have counterparts.</p>
<ul>
<li>B-tree: Is the default option when you create an index and is recommended to handle equality, range queries using <a href="https://www.postgresql.org/docs/current/functions-comparison.html"target="_blank" rel="noopener noreferrer">basic comparison</a>
. It also works well with sorted data.</li>
<li>Hash: It can only handle equality comparisons although they are <a href="https://www.postgresql.org/docs/7.2/indexes-types.html"target="_blank" rel="noopener noreferrer">discouraged</a>
because the need of reindexing in case of database crash and there is no probe of performance improvements with B-tree. Although PostgreSQL 10 is crash-safe.</li>
<li>GiST (Generalized Search Tree): Give us a complex system for implementing different strategies depending on the data types. Are good for geometric data types and full-text search. It has <a href="https://www.postgresql.org/docs/current/gist-builtin-opclasses.html"target="_blank" rel="noopener noreferrer">built-in operator classes</a>
although
you can extend it.</li>
<li>SP-GiST (space-partitioned GiST): Similar to GiST s meat to allow the development of custom data types but supports partitioned search trees for use with unbalanced data structures.</li>
<li>GIN (Generalized Inverted Index): Are suitable for dealing with composite values like array values or JSONB.</li>
<li>BRIN (Block Range INdexes): For handling large datasets created sequentially. The index group adjacent datasets in the table with blocks. This type allows also to save a lot of disk space because it keeps an index entry for the block itself instead of the tuple.</li>
</ul>
<p>As you can see, some of the Indexes types, like GIN or GiST, are more worried about the knowledge of the data types rather than a database size or structure.</p>
<p>In general, the use of indexes is essential for excellent performance, and it is something that you have to take care of as a developer. To implement it, you must take into account:</p>
<ul>
<li>Lookups performed</li>
<li>The data type</li>
<li>Underlying data within the table</li>
</ul>
<p>With these three variables, you must decide what index you have to choose for improving your queries. How can you evaluate it in PostgreSQL? With <code>EXPLAIN</code>.</p>
<h2 id="planner">Planner</h2>
<p>Within the internal structure of PostgreSQL, you can find the &ldquo;Planner&rdquo;, which is in charge of looking for the best execution plan for a SQL query. In the <a href="https://www.postgresql.org/docs/9.5/planner-optimizer.html"target="_blank" rel="noopener noreferrer">official documentation</a>
there is much documentation regarding how the internals work. I only try to give some clues about how you can analyze a specific query.</p>
<p>After parsing the query, PostgreSQL will try to:</p>
<ol>
<li>Generate different execution plans</li>
<li>Calculate the cost of each plan</li>
<li>Select the best one to execute the query</li>
</ol>
<p>Since you can&rsquo;t guess your query behaviour <em>EXPLAIN</em> will be your main tool.</p>
<p><code>EXPLAIN</code> help you to view how the planner interprets queries and determines the optimal execution. Combining it with options like <em>ANALYZE, COSTS, or VERBOSE</em> gives you the plan chosen by the planner. When you execute <em>EXPLAIN</em> you get some result like:</p>
<pre><code>$ EXPLAIN ANALYZE select * from test WHERE num=164;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Gather (cost=1000.00..8325.47 rows=483 width=41) (actual time=10.448..260.448 rows=509 loops=1)
Workers Planned: 2
Workers Launched: 2
-&gt; Parallel Seq Scan on test (cost=0.00..7277.17 rows=201 width=41) (actual time=5.214..236.655 rows=170 loops=3)
Filter: (num = 164)
Rows Removed by Filter: 166497
Planning time: 6.670 ms
Execution time: 263.254 ms
(8 rows)
Time: 327,272 ms
</code></pre><p>Using the keyword <em>ANALYZE</em> you will have more detail but take care because the query will be executed!. For detail understanding of it you can go to the PosgreSQL <a href="https://www.postgresql.org/docs/9.5/using-explain.html"target="_blank" rel="noopener noreferrer">docs</a>
. I will only give some little advice about some key points.</p>
<ul>
<li>Check the total cost and time. The upper-level node includes all children costs.</li>
<li>Review what indexes are being used in every particular node.</li>
<li>Review access methods for every node (Sequential Scan, Index Scan, Bitmap Heap Scan)</li>
<li>Like in any programming language nest matters quite a lot.</li>
</ul>
<p>For analysis, you have some online tools:</p>
<ul>
<li><a href="https://tatiyants.com/pev/">https://tatiyants.com/pev/</a></li>
<li><a href="https://explain.depesz.com">https://explain.depesz.com</a></li>
</ul>
<p>Or you can also use some local tool, even some IDE&rsquo;s like PyCharm/Intellij has the feature of showing the query plan as a diagram.</p>
<p>You can find additional information at <a href="https://www.postgresql.org/docs/current/indexes-types.html">https://www.postgresql.org/docs/current/indexes-types.html</a></p>
<h2 id="vaccum">Vaccum</h2>
<p>Vaccum can be another key feature when you are scaling up. The Vaccum primary job is to reclaim storage space occupied by dead tuples so it can be available to be re-used by future data insertion within the same table.
I think that the latest versions of PostgreSQL do a good job on this, and if you have the autovacuum active it works very well. Although you can tweak the configuration if it&rsquo;s necessary, I would say if you are developer and not PostgreSQL expert, keep away of doing it.</p>
<p>Nevertheless, you always have the possibility of executing the process manually in specific cases. Vaccum can run in two different ways in PostgreSQL.</p>
<ol>
<li>Plain Vaccum</li>
</ol>
<p>When you run the <code>VACCUM</code> statement, reclaimed storage space is not given back to the operating system rather they are just defragmented within the same page but there is no exclusive lock and can run in parallel with normal read and writing.</p>
<ol>
<li>Full Vaccum</li>
</ol>
<p><code>VACCUM FULL</code> return the disk space to the operating system but creates an exclusive lock on each table while it is being processed.</p>
<p>I hope these brief notes can give you a better understanding of what it&rsquo;s behind the scenes.</p>
<p>Cheers!</p></description>
</item>
<item>
<title>Python tools</title>
<link>https://alexhermida.dev/talks/20181108-python-tools/</link>
<pubDate>Tue, 20 Nov 2018 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/talks/20181108-python-tools/</guid>
<description><p>Small talk about useful tools written in Python at
<a href="https://meetup.com/es-ES/VigoJUG/events/255662689/"target="_blank" rel="noopener noreferrer">VigoJUG</a>
meetup in November 8, 2018.</p>
<p>Slides: <a href="https://github.com/alexhermida/talks/tree/master/python-tools">https://github.com/alexhermida/talks/tree/master/python-tools</a></p>
</description>
</item>
<item>
<title>This summer master the mosquittos</title>
<link>https://alexhermida.dev/talks/20170720-python-mqtt-broker/</link>
<pubDate>Thu, 20 Jul 2017 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/talks/20170720-python-mqtt-broker/</guid>
<description><p>Lightning talk about <a href="http://mosquitto.org"target="_blank" rel="noopener noreferrer">MQTT broker</a>
and <a href="https://github.com/eclipse/paho.mqtt.python"target="_blank" rel="noopener noreferrer">Python Client</a>
.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe src="https://www.youtube.com/embed/Y-W9t8KWNz4" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" allowfullscreen title="YouTube Video"></iframe>
</div>
</description>
</item>
<item>
<title>Artesanos del futuro</title>
<link>https://alexhermida.dev/talks/20170221-artesanos-del-futuro/</link>
<pubDate>Tue, 21 Feb 2017 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/talks/20170221-artesanos-del-futuro/</guid>
<description><p>Charla realizada en el <a href="https://www.meetup.com/es-ES/GDGVigo/events/237396475/"target="_blank" rel="noopener noreferrer">High School Developers Parade Vigo 2017</a>
evento organizado por
el GDGVigo <a href="https://www.meetup.com/es-ES/GDGVigo/"target="_blank" rel="noopener noreferrer">GDGVigo</a>
.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe src="https://www.youtube.com/embed/-O7ZdnDQXUQ" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" allowfullscreen title="YouTube Video"></iframe>
</div>
</description>
</item>
<item>
<title>Python special methods for muggles</title>
<link>https://alexhermida.dev/talks/20160317-python-special-methods/</link>
<pubDate>Thu, 17 Mar 2016 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/talks/20160317-python-special-methods/</guid>
<description><p>Presentation about Python special methods for Python Vigo.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
<iframe src="https://www.youtube.com/embed/IY_6vY-M60Y" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" allowfullscreen title="YouTube Video"></iframe>
</div>
</description>
</item>
<item>
<title>About</title>
<link>https://alexhermida.dev/about/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://alexhermida.dev/about/</guid>
<description><p>Hello! I&rsquo;m Álex, a Software Engineer from Spain.</p>
<p>Nowdays, I work helping to find new treatments for rare diseases at <a href="https://healx.io"target="_blank" rel="noopener noreferrer">Healx</a>
an AI-driven company based in Cambridge. Previously, I worked in <a href="https://taiga.io"target="_blank" rel="noopener noreferrer">Taiga.io</a>
, I founded <a href="https://web.archive.org/web/20180829101008/http://www.initios.com/"target="_blank" rel="noopener noreferrer">Initios</a>
, a software company, and co-founded <a href="https://www.parkapp.com/"target="_blank" rel="noopener noreferrer">Parkapp</a>
, among other things.</p>
<p>I love Python language and the community around it, I had a broad experience in I.T projects, and I&rsquo;m also a computer security enthusiast. I like to work as a team facilitator.</p>
<p>One of the most valuable thing for achieving goals is communication, so I am a <a href="http://agilemanifesto.org/"target="_blank" rel="noopener noreferrer">Agile manifesto</a>
fan:</p>
<blockquote>
<p>&ldquo;Individuals and interactions over processes and tools.&rdquo;</p>
</blockquote>
<p>I love learning and improving my skills every day. I collaborate on local development groups inside <a href="http://vigotech.org/"target="_blank" rel="noopener noreferrer">VigoTech</a>
, an active member in <a href="https://aindustriosa.org"target="_blank" rel="noopener noreferrer">A Industriosa</a>
and I&rsquo;m one of the organizers of <a href="https://www.python-vigo.es/"target="_blank" rel="noopener noreferrer">Python Vigo</a>
and <a href="https://pyday2017.python-vigo.es/"target="_blank" rel="noopener noreferrer">PyDay Galicia</a>
.</p>
</description>
</item>
</channel>
</rss>