Add package archive fallback mechanism #6828

d12frosted · 2016-08-15T17:46:18Z

Seeing all these issues with package installation due to MELPA availability problems I decided to improve status check for all elpa archives and error about archive being unavailable instead of showing errors like 'package xxx is not available' in the future.

Also I removed timeout heuristic as it's not 100% accurate. I don't see any reason for it to be here once configuration-layer-check-archive-status is used.

Note that configuration-layer-check-archive-status is a little bit beefed (more than we need at this point), but it might become handy in future.

Oh, and I am not sure about the error message though. @syl20bnr, you might want to improve it. 😸

P. S. Hope I didn't miss anything.

P. P. S. Somewhat fixes #6836. Another related issue is #4453.

TheBB · 2016-08-15T17:48:35Z

core/core-configuration-layer.el

+               url)
+      (condition-case nil
+          (if (url-http-file-exists-p url)
+              (setq state 'avaialble)


My typing skills are weak 👀 Thanks for noticing this typo!

TheBB · 2016-08-15T17:50:08Z

If dotspacemacs-elpa-timeout isn't needed any more you should remove it (and the tests for it).

d12frosted · 2016-08-16T06:44:31Z

I have removed dotspacemacs-elpa-timeout as well as related test. Also I have updated error message in order to make it more clear.

Update 1 This PR is related to #6830 and #4453

Update 2 ~~test-retrieve-package-archives--catch-connection-errors also fails, gonna check it out.~~

d12frosted · 2016-08-18T18:32:19Z

Just pushed another commit that adds fallback functionality when one of the archives is down. You can test it by returning 'unavailable in configuration-layer-check-archive-status for any archive of your liking. For example,

(when (string-equal archive "melpa")
      (setq state 'unavailable))

So when any archive is not available, configuration-layer/retrieve-package-archives will fallback to configuration-layer-fallback-package-archive for unavailable archive.

As fallback I am using snapshot provided by @zilongshanren. Thanks to melpa/melpa#4156 I've found this nice 'mirror'. The good thing is that MELPA is going to create official mirror that people could use when MELPA is down.

P. S. I've made two commits for now - gonna squash them later on demand. Just wanna keep first commit for now.

d12frosted · 2016-08-18T18:58:53Z

core/core-configuration-layer.el

+    ("gnu" . unknown))
+  "Status of ELPA archives.")
+
+(defvar configuration-layer-fallback-package-archive


I am planning to convert it list of fallback archives.

d12frosted · 2016-08-20T11:03:40Z

Pushed another commit.

Instead of using one fallback url, use package-archives compatible list of fallback urls. If currently set url is not reachable, then try to connect to one of fallback urls. If all urls are unavailable - show an error with some hints and explanations.

Again, you might want to change error message. But I tried to make it as useful as possible. 😸

d12frosted · 2016-08-20T11:06:21Z

And a good thing, now https://github.com/syl20bnr/spacemacs-elpa is used as primary fallback archive 😸

d12frosted · 2016-09-13T13:48:54Z

core/core-configuration-layer.el

+    ("gnu" . unknown))
+  "Status of ELPA archives.")
+
+(defvar configuration-layer-fallback-package-archives


I wonder if we really need this. I mean, fallback archives can be added to the end of configuration-layer--elpa-archives, so instead of switching to values from configuration-layer-fallback-package-archives we can just remove first associated url and search for another. Once no urls are available for specific archive - error!

Instead of using one fallback url, use package-archives compatible list of fallback urls. If currently set url is not reachable, then try to connect to one of fallback urls. If all urls are unavailable - show an error with some hints and explanations.

Instead of storing fallback urls for package archives in separate variable and doing complicated operations in order to derive available url, store all urls (official and fallback ones) in `configuration-layer--elpa-archives`. This simplifies availability check and user configurations. Also get rid of `configuration-layer--elpa-archives-status` as it's really not used.

d12frosted · 2016-09-16T15:27:46Z

core/core-configuration-layer.el

+        (state 'unavailable))
+    (condition-case nil
+        (when (or (and (string-prefix-p "/" url)
+                       (file-readable-p url))


Not sure that this is the best way of checking that archive url is accessible. file-accessible-directory-p? I haven't tested situations when archive-contents file is readable, but all other files are not readable by current user.

@d12frosted Where can we find the documentation about file-readable-p capable of checking URLs except in the C source ? This function does not block for a long time when the URL is not reachable ? What kind of tests did you do to check that file-readable-p is not blocking for a long time ?

What kind of tests did you do to check that file-readable-p is not blocking for a long time ?

I mean, did you test regular timeout method and file-readable-p with the same input ?

d12frosted · 2016-09-16T15:29:27Z

Ok, I've made several changes. I simplified fallback mechanism, removed two redundant variables and improved error message. Oh and also I made it possible to use local package archive urls.

d12frosted · 2016-09-16T15:30:57Z

I spammed pretty much in this PR with commits. I'll squash them once @syl20bnr give the green light 😸 For now I wish to preserve history. 😸

ninrod · 2016-09-19T02:07:13Z

Just a quick note: this should close #7103 and provide a way for users behind corporate firewalls to use spacemacs in all it's glory.

d12frosted · 2016-09-19T09:18:13Z

@ninrod is GitHub available for you behind your firewall?

ninrod · 2016-09-19T13:19:27Z

@d12frosted yes it is available. We have a severely slowed down connection pipe to and from it though. So the local clone is like heaven.

But really, the local melpa solution is so awesome that I'm using it in home, even without a firewall.
With a single git clone you have the whole melpa, elpa and what not in a local folder that you control. No connection hiccups, I don't have to worry about melpa going down, reconfigurations are like instantaneous, I can tag a specific snapshot that I know for a fact that works with my config. What more can I ask? That should be the default. So robust, efficient, fast, configurable. Awesome.

d12frosted · 2016-09-19T13:44:50Z

@ninrod I asked about GitHub just because some of packages used by Spacemacs are downloaded using quelpa from GitHub directly. This fallback mechanism doesn't cover such packages. E. g. when GitHub is down (?!) it's impossible to start Spacemacs from scratch even if you have local copies of elpa package archives (namely GNU ELPA, MELPA, Stable MELPA and Org-mode package archive). I know that I already said this in other thread, but I just want to make it clear for others as well.

reconfigurations are like instantaneous

Indeed. Also it makes bootstrap testing much easier.

I can tag a specific snapshot that I know for a fact that works with my config

And this as well. Might be very useful for corporations, where they want to use specific stable version of Spacemacs with specific package archives snapshot (so no upstream changes break flow).

That should be the default. So robust, efficient, fast, configurable. Awesome.

Glad you like it 💃

ninrod · 2016-09-19T15:49:05Z

@d12frosted I should mention that I have the option of mirroring specific github repos inside an internal git server behind corp firewall. Out of curiosity, am I able to change those quelpa URLs to match my specific internal mirrors?

d12frosted · 2016-09-19T15:53:38Z

We are talking about some Emacs Lisp code, so yes, it's possible to change it's behaviour 😸 There might be better way, but you always can just brutally redefine github fetcher in quelpa. 😸

ninrod · 2016-11-04T23:35:03Z

let's do this!

github-actions · 2020-09-30T04:19:41Z

This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please let us know if this PR is still need merging!

TheBB reviewed Aug 15, 2016
View reviewed changes

d12frosted force-pushed the feature/elpa-status-check branch 4 times, most recently from 6bbe9d1 to fa4892e Compare August 16, 2016 06:42

d12frosted force-pushed the feature/elpa-status-check branch from fa4892e to 503676a Compare August 16, 2016 06:59

d12frosted changed the title ~~improve archive availability check~~ Add package archive fallback mechanism Aug 18, 2016

d12frosted reviewed Aug 18, 2016
View reviewed changes

d12frosted force-pushed the feature/elpa-status-check branch 2 times, most recently from 2f55ac1 to 3414922 Compare August 20, 2016 11:01

d12frosted force-pushed the feature/elpa-status-check branch from 3414922 to f699c63 Compare August 20, 2016 19:43

TheBB assigned syl20bnr Aug 21, 2016

TheBB added the Core label Aug 21, 2016

d12frosted reviewed Sep 13, 2016
View reviewed changes

d12frosted mentioned this pull request Sep 15, 2016

[question] how do I tell spacemacs to look for a local elpa mirror? #7103

Closed

d12frosted added 2 commits September 16, 2016 17:41

improve archive availability check

7996c8a

use fallback archive when any archive is down

493eb70

d12frosted force-pushed the feature/elpa-status-check branch from f699c63 to 4775fe4 Compare September 16, 2016 14:42

d12frosted force-pushed the feature/elpa-status-check branch from 4775fe4 to eded51e Compare September 16, 2016 14:54

d12frosted added 2 commits September 16, 2016 18:15

allow to use local package archive urls

595f5ab

d12frosted commented Sep 16, 2016

View reviewed changes

smile13241324 mentioned this pull request Oct 15, 2016

Autumnal Cleanup 2016 Reporting #7334

Closed

ninrod mentioned this pull request Jan 27, 2018

[SOLVED] Install from behind a restrictive firewall that does not allow emacs to connect to MELPA to get archives #10179

Closed

github-actions bot added the stale marked as a stale issue/pr (usually by a bot) label Sep 30, 2020

d12frosted closed this Nov 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add package archive fallback mechanism #6828

Add package archive fallback mechanism #6828

d12frosted commented Aug 15, 2016 •

edited

Loading

TheBB Aug 15, 2016

d12frosted Aug 15, 2016

TheBB commented Aug 15, 2016

d12frosted commented Aug 16, 2016 •

edited

Loading

d12frosted commented Aug 18, 2016

d12frosted Aug 18, 2016

d12frosted commented Aug 20, 2016

d12frosted commented Aug 20, 2016

d12frosted Sep 13, 2016

d12frosted Sep 16, 2016

syl20bnr Jan 15, 2017

syl20bnr Jan 15, 2017

d12frosted commented Sep 16, 2016

d12frosted commented Sep 16, 2016

ninrod commented Sep 19, 2016 •

edited

Loading

d12frosted commented Sep 19, 2016

ninrod commented Sep 19, 2016 •

edited

Loading

d12frosted commented Sep 19, 2016

ninrod commented Sep 19, 2016

d12frosted commented Sep 19, 2016

ninrod commented Nov 4, 2016

github-actions bot commented Sep 30, 2020

Add package archive fallback mechanism #6828

Add package archive fallback mechanism #6828

Conversation

d12frosted commented Aug 15, 2016 • edited Loading

TheBB Aug 15, 2016

Choose a reason for hiding this comment

d12frosted Aug 15, 2016

Choose a reason for hiding this comment

TheBB commented Aug 15, 2016

d12frosted commented Aug 16, 2016 • edited Loading

d12frosted commented Aug 18, 2016

d12frosted Aug 18, 2016

Choose a reason for hiding this comment

d12frosted commented Aug 20, 2016

d12frosted commented Aug 20, 2016

d12frosted Sep 13, 2016

Choose a reason for hiding this comment

d12frosted Sep 16, 2016

Choose a reason for hiding this comment

syl20bnr Jan 15, 2017

Choose a reason for hiding this comment

syl20bnr Jan 15, 2017

Choose a reason for hiding this comment

d12frosted commented Sep 16, 2016

d12frosted commented Sep 16, 2016

ninrod commented Sep 19, 2016 • edited Loading

d12frosted commented Sep 19, 2016

ninrod commented Sep 19, 2016 • edited Loading

d12frosted commented Sep 19, 2016

ninrod commented Sep 19, 2016

d12frosted commented Sep 19, 2016

ninrod commented Nov 4, 2016

github-actions bot commented Sep 30, 2020

d12frosted commented Aug 15, 2016 •

edited

Loading

d12frosted commented Aug 16, 2016 •

edited

Loading

ninrod commented Sep 19, 2016 •

edited

Loading

ninrod commented Sep 19, 2016 •

edited

Loading