cache.nixos.org is down

[YorikSar]

It seems us-east-1 region is experiencing some issues right now, and cache.nixos.org always returns "503 SlowDown" error.

Is cache.nixos.org hosted in one region only? Is it possible to duplicate it in different region (or different cloud) for more availability?

[vcunat]

The cache is in multiple regions. Europe seems OK to me, though I'm not sure if it will help to try cross-region downloads. FWIW:

$ dig cache.nixos.org +short
d3m36hgdyp4koz.cloudfront.net.
13.32.121.5
13.32.121.15
13.32.121.41
13.32.121.49
13.32.121.89
13.32.121.95
13.32.121.175
13.32.121.233

$ dig cache.nixos.org +short AAAA
d3m36hgdyp4koz.cloudfront.net.
2600:9000:20ac:1e00:10:dfa4:b0c0:93a1
2600:9000:20ac:2800:10:dfa4:b0c0:93a1
2600:9000:20ac:2c00:10:dfa4:b0c0:93a1
2600:9000:20ac:6c00:10:dfa4:b0c0:93a1
2600:9000:20ac:7400:10:dfa4:b0c0:93a1
2600:9000:20ac:ba00:10:dfa4:b0c0:93a1
2600:9000:20ac:ea00:10:dfa4:b0c0:93a1
2600:9000:20ac:fc00:10:dfa4:b0c0:93a1

[bbarker]

I was having a lot of trouble (I'm actually in the eastern US, assuming that matters):

$ dig cache.nixos.org +short
d3m36hgdyp4koz.cloudfront.net.
54.230.161.241
54.230.161.105
54.230.161.41
54.230.161.32
54.230.161.242
54.230.161.57
54.230.161.157
54.230.161.76

$ dig cache.nixos.org +short AAAA
d3m36hgdyp4koz.cloudfront.net.
2600:9000:2019:a800:10:dfa4:b0c0:93a1
2600:9000:2019:cc00:10:dfa4:b0c0:93a1
2600:9000:2019:4800:10:dfa4:b0c0:93a1
2600:9000:2019:ee00:10:dfa4:b0c0:93a1
2600:9000:2019:b400:10:dfa4:b0c0:93a1
2600:9000:2019:0:10:dfa4:b0c0:93a1
2600:9000:2019:4c00:10:dfa4:b0c0:93a1
2600:9000:2019:a00:10:dfa4:b0c0:93a1

[YorikSar]

Europe seems OK to me

Huh, I've been getting US servers from DNS because of active VPN all this time :)

I wonder if CloudFront can be configured to skip problem region.

[vcunat]

AWS status shows OK now. I don't think there's much to do about about such rare outages, except that in the long term we might manage to provide some redundancy, e.g. via NixOS/nix#1167 (comment)

[YorikSar]

@vcunat Oh, IPFS is already in works, huh? Cool, thanks!

[vcunat]

Noone serves nixpkgs binaries via IPFS currently, AFAIK, but I believe we will work it out in time. (There are just too many things to do.)

[YorikSar]

Is there some final conclusion on how to serve cache over IPFS? I've seen lots of discussion and didn't find a concrete proposal that is being implemented.

In my head it can be implemented as simple IPFS "directory" with pretty much same structure as being served through HTTP now.

Can you share current stats of the cache like overall size, number of files in it, average number of requests and network load?

[vcunat]

The link I posted is a complete implementation that has been tested, I believe, though it's not merged in the official nix repository. Here we discussed approximate sizes of the channels (I have no numbers first-hand).

[YorikSar]

@vcunat thanks. Diving into all these threads I feel totally lost. I guess I'm a teapot. The first thing that surprised me was 60G for total size of packages (really? just 60G? other systems require way more space for mirrors...), and it all went haywire from there. I guess learning stuff just shows me how much I don't know there :)

I think this discussion would be better suited for email thread than some random issue on GitHub, will start a thread there.

[bbarker]

Well, I don't know, maybe it is just me but I feel github issues are more friendly to newcomers that happen on the discussion post facto.

…

On Sat, Sep 16, 2017 at 1:35 PM, Yuriy Taraday ***@***.***> wrote: @vcunat <https://github.com/vcunat> thanks. Diving into all these threads I feel totally lost. I guess I'm a teapot. The first thing that surprised me was 60G for total size of packages (really? just 60G? other systems require way more space for mirrors...), and it all went haywire from there. I guess learning stuff just shows me how much I don't know there :) I think this discussion would be better suited for email thread than some random issue on GitHub, will start a thread there. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29389 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA37jgErY9z12ApFzpbtW39hmHAfFqp3ks5sjAbpgaJpZM4PYElv> .

-- Brandon Barker [email protected]