-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathconfigure-s3-for-copc.html
158 lines (141 loc) · 8.72 KB
/
configure-s3-for-copc.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
<!DOCTYPE html>
<html lang="en" prefix="og: http://ogp.me/ns#">
<head>
<link href="http://gmpg.org/xfn/11" rel="profile">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<!-- Metadata -->
<meta name="description" content="">
<meta property="og:description" content="">
<meta property="og:title" content="Configure S3 for COPC" />
<meta property="og:type" content="article" />
<meta property="og:url" content="https://darrenwiens.github.io/configure-s3-for-copc.html" />
<meta property="og:image" content="https://darrenwiens.github.io/images/face.png" />
<!-- Enable responsiveness on mobile devices-->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1">
<title>Darren Wiens</title>
<!-- CSS -->
<link href="//fonts.googleapis.com/" rel="dns-prefetch">
<link href="//fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic|Abril+Fatface|PT+Sans:400,400italic,700&subset=latin,latin-ext" rel="stylesheet">
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/poole.css" />
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/hyde.css" />
<link rel="stylesheet" href="https://darrenwiens.github.io/theme/css/syntax.css" />
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/css/fork-awesome.min.css" crossorigin="anonymous">
<!-- Feeds -->
<!-- Analytics -->
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=G-9664J8VT36"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'G-9664J8VT36');
</script>
<!-- Ads -->
<script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-2681534008399266"
crossorigin="anonymous"></script>
</head>
<body class="theme-base-08">
<div class="sidebar">
<div class="container sidebar-sticky">
<div class="sidebar-about">
<h1>
<a href="/">
<img class="profile-picture" src="https://darrenwiens.github.io/images/face.png">
Darren Wiens
</a>
</h1>
<p class="lead"></p>
<p class="lead"> </p>
<p></p>
</div>
<ul class="sidebar-nav">
</ul>
<nav class="sidebar-social">
<a class="sidebar-social-item" href="http://twitter.com/dkwiens" target="_blank">
<i class="fa fa-twitter"></i>
</a>
<a class="sidebar-social-item" href="https://fosstodon.org/@dkwiens" target="_blank">
<i class="fa fa-mastodon"></i>
</a>
<a class="sidebar-social-item" href="http://github.com/darrenwiens" target="_blank">
<i class="fa fa-github"></i>
</a>
<a style="display:none" class="sidebar-social-item" href="https://darrenwiens.github.io/None">
<i class="fa fa-rss"></i>
</a>
<a rel="me" style="display:none" href="https://fosstodon.org/@dkwiens">Mastodon</a>
</nav>
</div>
</div> <div class="content container">
<div class="post">
<h1 class="post-title">Configure S3 for COPC</h1>
<span class="post-date">Fri 01 April 2022</span>
<p><strong>Disclaimer: I'm pretty basic as far as security/AWS devops goes, so you should consider this <em>a way</em> to host cloud optimized point cloud files, definitely not <em>the way</em>.</strong></p>
<p>First off, I highly recommend Bert Temme's excellent <a href="https://bertt.wordpress.com/2022/03/29/cloud-optimized-point-cloud-copc/">blog post</a> about creating <a href="https://copc.io/">cloud optimized point cloud</a> (COPC) files using <a href="https://pdal.io/">PDAL</a> (version 2.4+ to take advantage of COPC reader/writer).</p>
<p>This blog post explains how I created a COPC from <a href="https://www2.gov.bc.ca/gov/content/data/geographic-data-services/lidarbc">LiDAR BC</a>, configured <a href="https://docs.aws.amazon.com/s3/index.html">AWS S3</a> for storage/hosting, and displayed the file on the <a href="https://viewer.copc.io/">COPC Viewer</a>.</p>
<h2>Get the data</h2>
<p>I downloaded LAZ data from <a href="https://www2.gov.bc.ca/gov/content/data/geographic-data-services/lidarbc">LiDAR BC</a>. It's a nice, basic GUI for data download. Find the tile, download the data. In my case, I downloaded a <strong>177 MB</strong> LAZ file.</p>
<p><img alt="LiDAR BC Download" src="https://darrenwiens.github.io/images/lidarbc.gif"></p>
<h2>Convert LAZ to COPC</h2>
<p>I converted my LAZ file to COPC with the following <a href="https://pdal.io/">PDAL</a> Python script:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">pdal</span>
<span class="n">json</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2">[</span>
<span class="s2"> {</span>
<span class="s2"> "type":"readers.las",</span>
<span class="s2"> "filename":"/path/to/input.laz"</span>
<span class="s2"> },</span>
<span class="s2"> {</span>
<span class="s2"> "type":"writers.copc",</span>
<span class="s2"> "filename":"/path/to/output.copc.laz"</span>
<span class="s2"> }</span>
<span class="s2">]</span>
<span class="s2">"""</span>
<span class="n">pipeline</span> <span class="o">=</span> <span class="n">pdal</span><span class="o">.</span><span class="n">Pipeline</span><span class="p">(</span><span class="n">json</span><span class="p">)</span>
<span class="n">pipeline</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
</code></pre></div>
<p>The resulting file ballooned somewhat to <strong>499 MB</strong>, but hey, it's cloud optimized.</p>
<h2>Upload to AWS S3 and Configure for Display</h2>
<p>You can use whatever hosting or cloudprovider you like, but this is how I set up <a href="https://docs.aws.amazon.com/s3/index.html">AWS S3</a>.</p>
<p>First step, upload.</p>
<p>Second step, configure CORS. I will explain CORS in a future blog post<sup><a href="#cors">1</a></sup>, but suffice it to say that by configuring CORS on the S3 bucket, we can restrict the methods allowed to interact with files (in this case, <code>GET</code>) and the domains from which they can originate (in this case, <code>https://viewer.copc.io</code>). The CORS configuration I added to my S3 bucket, under the Permissions tab, was:</p>
<div class="highlight"><pre><span></span><code>[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
"GET"
],
"AllowedOrigins": [
"https://viewer.copc.io"
],
"ExposeHeaders": [
"x-amz-server-side-encryption",
"x-amz-request-id",
"x-amz-id-2"
],
"MaxAgeSeconds": 300
}
]
</code></pre></div>
<h2>Generate a Presigned URL</h2>
<p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html">S3 Presigned URLs</a> are magical things that allow you to temporarily attach your own permissions to an S3 URL, so someone else can do something with a file. In this case, I want a viewer to be able to download chunks of a point cloud by pointing it to a URL, but I don't want to make my file completely public for anyone to download, and I don't want to set up authorization between the viewer and my file. I get the added peace of mind that the url will expire in a certain amount of time, so absolutely no one will be able to use the URL later. Generate a presigned URL in your preferred method (CLI or GUI).</p>
<h2>View the COPC</h2>
<p><a href="https://hobu.co/">Hobu</a> has kindly provided a COPC viewer <a href="https://viewer.copc.io/">here</a>. The UX is about as straighforward as possible: paste your presigned URL where it says "Add a Point Cloud". In the process of making the animation below, I downloaded approximately <strong>250 MB</strong> through the browser.</p>
<p><img alt="LiDAR BC Download" src="https://darrenwiens.github.io/images/mr_pg_copc.gif"></p>
<p>That is all. As always get in touch with me on <a href="https://twitter.com/dkwiens">Twitter</a> if you want to talk more about this! I'd be very interested to know better strategies for configuring S3 for hosting/serving files like this, or if there are other ways to further optimize COPCs.s</p>
<p><a name="cors">1</a>: I will never explain CORS. You can Google it, though.</p>
<span class="post-tags">
Tags:
<ul>
<li><a href="https://darrenwiens.github.io/tag/aws.html">aws</a></li>
<li><a href="https://darrenwiens.github.io/tag/s3.html">s3</a></li>
<li><a href="https://darrenwiens.github.io/tag/copc.html">copc</a></li>
</ul>
</span>
</div>
</div>
</body>
</html>