forked from projectvacuum/vcycle
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathvcycle.conf.5
320 lines (269 loc) · 12.6 KB
/
vcycle.conf.5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
.TH vcycle.conf 5 "Jan 2014" "vcycle.conf" "vcycle Manual"
.SH NAME
.B vcycle.conf
\- vcycle configuration file
.SH DESCRIPTION
.B vcycled
is a daemon which implements VM lifecycle management on IaaS Cloud systems
in a way inspired by Vac. vcycled reads its configuration from
.B /etc/vcycle.conf
and .conf files in
.B /etc/vcycle.d
The configuration files use the Python ConfigParser syntax, which is similar
to MS Windows INI files. The files are divided into sections, with each section
name in square brackets. Each section contains
a series of option=value pairs. Sections with the same name are merged
and if options are duplicated, later values overwrite values given
earlier.
For ease of management, any configuration file ending in .conf in the
directory /etc/vcycle.d will be read, in
alphanumeric order by name, and then /etc/vcycle.conf is read if present.
.SH [SPACE ...] SECTIONS
One [space ...] section must exist for each tenancy, project, or account in which
VMs will be created, with the Vcycle space name for the space given in the section
name, such as [space project1.example.com]. The Vcycle admin has a free choice
of what names to give each space, but the name must only consist of lowercase
letters, numbers, periods, and hyphens so it can be used as the DNS name of
a virtual CE elsewhere in the system.
.B api
is required and specifies the lowercase name of the API to use when contacting
the IaaS service associted with this space.
.B max_machines
prevents new machines from being created if the number of machines for
this space in any state exceeds the given limit.
.B gocdb_sitename
gives the GOCDB site name to use when writing APEL
accounting record files to /var/lib/vcycle/apel-outgoing and
/var/lib/vcycle/apel-archive. Please use your official site name to avoid
the risk of misnamed records getting into the central APEL database.
If gocdb_sitename is not given, then records are only written to
apel-archive and the vcycle space name is used as a placeholder in the
files.
.B https_port
gives the port number used to contact the Vcycle HTTPS server from
within the VMs. This only needs to be changed if there is an intervening
firewall. Default 443.
.SH OPENSTACK SPACE SECTIONS
OpenStack spaces are enabled with
.B api = openstack
and the following options are then available.
.B url
is the URL of the identity (KeyStone) endpoint for this OpenStack service.
.B tenancy_name
is the tenancy or project name.
.B username
is a user with access to the tenancy.
.B password
is the encoded password for the user. Vcycle will decrease the value of
each character of the string by one to obtain the true password. This
encoding is to avoid revealing memorable passwords when casually viewing
or editing configuration files.
When creating VMs in OpenStack spaces, Vcycle will create machinefeatures,
jobfeatures, and joboutputs metadata keys with the URLs of the
corresponding directories for the VM on the Vcycle machine's HTTP(S)
server.
.SH MACHINE / JOB FEATURES HTTP(S) SERVER
Vcycle will create the machinefeatures and jobfeatures files according
to the WLCG MJF protocol in machinefeatures and jobfeatures subdirectories
of the VMs directory in /var/lib/vcycle/machines . Using the vcycle-httpd.conf
Apache configuration file supplied in /usr/share/doc/vcycle-VERSION, these
files are publish via HTTP and HTTPS.
In addition, if x509dn is set for the VM's machinetype, then the script vcycle-cgi
will allow the VM to write heartbeat, logging, and shutdown_message files
to the VM's $JOBOUTPUTS directory.
.SH [MACHINETYPE ... ...] SECTIONS
One [machinetype ... ...] section must exist for each machinetype in each space, with
the space name and name of the machinetype given in the section name, such as
[machinetype project1.example.com example].
A machinetype name must only consist of lowercase letters, numbers, and hyphens.
Each of these sections contain option=value pairs that are specific to
that machinetype. The same machinetype name can appear in different spaces and will
be managed separately.
.B flavor_name
gives the name of a flavor previously defined in the IaaS system which
represents a particular CPU, memory, and disk geometry.
.B hs06
gives the total HEPSPEC06 power of each virtual machine created for this
flavor in this machinetype. This is used as the value $MACHINEFEATURES/hs06
supplied to the VM, when calculating target shares, and
when writing APEL accounting records. Defaults to 1.
.B cpu_per_machine
gives the number of processors of each virtual machine created for this flavor
in this machinetype. This is used as the value of $MACHINEFEATURES/phys_cores
and $MACHINEFEATURES/log_cores supplied to the VM. If the api plugin can
positively determine the number from metadata about the flavor, it will be
used in preference to the value given here. Default 1.
.B target_share
gives the desired share of the capacity available in this space for this
machinetype. The shares do not need to add up to 1.0, and if a share is not given
for a machinetype, then it is set to 0. Vcycle consults these shares
when deciding which machinetype to start as VM capacity becomes available.
Shares are weighted by the hs06 value of the machinetype.
.B backoff_seconds
is the delay after a VM of this machinetype aborts. If a VM aborts, then no new
VMs of this type will be created for this amount of time. This can be used
to prevent the unnecessary creation of many VMs when no work is available,
and avoid overloading the matcher or task queue of the VO.
.B fizzle_seconds
is used in three places within the backoff procedure and in two
other parts of vcycle:
.br
(1) First, if a VM finishes
without producing a shutdown message code and has lasted less than
fizzle_seconds, then it is treated as aborted.
.br
(2) Secondly, after the
backoff_seconds time has expired for a VM abort, once at least one VM has
been started in this vcycle space, then no more new VMs can be started for
another fizzle_seconds.
.br
(3) Additionally, when writing the accounting log files, any VMs which
run for less than fizzle_seconds are excluded.
.br
(5) Finally, the heartbeat file
checking is only carried out once an initial period of fizzle_seconds
has passed.
.B accounting_fqan
is used to specify a FQAN to include when writing APEL accounting
records, to associate usage with particular experiments.
.B max_machines
prevents new machines from being created if the number of machines for
this machinetype in any state exceeds the given limit.
.B max_wallclock_seconds
gives the maximum lifetime of a VM. vcycle will create
$MACHINEFEATURES/shutdowntime inside the VM using this value to
communicate it to the VM. vcycle will destroy the VM if it is still
running after this amount of time. Default 86400.
.B heartbeat_file
allows the machinetype to nominate a file which will be created in
the $JOBOUTPUTS directory before fizzle_seconds has passed. If this
file is not created by then and maintained for the lifetime of the VM,
the VM will be destroyed.
.B heartbeat_seconds
gives the frequency at which the heartbeat_file must be updated after
fizzle_seconds has passed. If the file is not updated for
heartbeat_seconds then the VM will be destroyed. If heartbeat_seconds
is 0, then only the existence of the file will be checked. Default 0.
.B x509dn
is an optional X.509 DN which will be used by the vcycle-cgi script to
control writing to VMs' $JOBOUTPUTS directories on the local HTTPS
server.
.B log_joboutputs
can be set to True to enable recording of all the files from
local $JOBOUTPUTS directories for VMs, to subdirectories of
/var/lib/vcycle/joboutputs when the VMs finish or are killed. The
subdirectories are in a hierarchy of the space name, machinetype name,
and then hostname of the VM. Default False.
.B joboutputs_days
sets the expiration time in days for per-VM directories created under
/var/lib/vcycle/joboutputs.
.B remote_joboutputs_url
sets a base URL on a remote HTTPS server to which VMs of this machinetype
can write. The value of $JOBOUTPUTS will be the VM
name chosen by Vcycle appended as a directory name to the URL given
by this option.
For the remaining options, if the file name begins with '/', then it
will be used as an absolute path; otherwise the path will be interpreted
relative to the machinetype's subdirectory of /var/lib/vcycle/spaces/SPACE/machinetypes/MACHINETYPE/files
where SPACE is the parent space name and MACHINETYPE is the name of
this machinetype.
.B remote_joboutputs_cert
and
.B remote_joboutputs_key
give filesnames of an X.509 client
certificate and key to use when requesting
$JOBOUTPUTS/shutdown_message and any heartbeat file in $JOBOUTPUTS. If
both are contained in the same file then the same value can be given
to both options.
.B root_image
is the path to the image file from which the VM will boot.
This can also be a remote HTTP or HTTPS URL which vcycle
will cache in /var/lib/vcycle/imagecache. The remote server must supply a
Last-Modified timestamp and vcycle will re-request the image each time a
VM starts using an If-Modified-Since request to minimise network load.
Alternatively, the images may be files in the local filesystem. If
root_image ends in .iso , then the image will be declared as ISO format
(a CD-ROM image), otherwise as a raw HDD image.
.B cernvm_signing_dn
is used to specify a regular expression to match the DN of an X.509
certificate used to verify the authenticity of the root image. Vcycle
attempts to obtain the certificate and signature from a CernVM Signature
Block at the end of the image file, verifies the
certificate using the CA files in /etc/grid-security/certificates, and
compares the certificate DN to cernvm_signing_dn. If this option is
given, all these verification steps must be satisified for the image
to be used. As of 2015, CernVM images are signed with a DN matching
the regular expression /CN=cvm-sign01\\.cern\\.ch$
.B root_public_key
is the file name of a public key which Vcycle will set up on the IaaS
system and supply to the VMs to allow root ssh access. Setting this
option to /root/.ssh/id_rsa.pub will give access from the factory machine.
.B user_data
is the path of a contextualization file provided by the VO and perhaps
modified by the site. If the path is a remote HTTP or HTTPS URL, vcycle
will fetch it over the network each time a VM is started. However the
file is obtained, vcycle will apply a series of default and locally defined
##user_data___## substitutions to it. See USER_DATA SUBSTITUTIONS below
for a list of the default substitutions.
.B user_data_option_XXX
and
.B user_data_file_XXX
are locally defined substitutions which will be applied to the user_data
file before the VM is started. user_data_option_XXX takes the string to
be substituted. user_data_file_XXX takes the relative or absolute path to
a file whose contents will be substituted for the pattern in the
user_data file.
.B user_data_proxy_cert
and
.B user_data_proxy_key
are the locations of files containing X.509 certificate(s) and an RSA
private key, all in PEM format, which will be used to make a limited
X.509 proxy. The same file can be given for both options if desired.
.B legacy_proxy
can be set to True to generate Globus legacy proxies rather than RFC 3820
proxies. Default False.
.SH USER_DATA SUBSTITUTIONS
Before the user_data file is used in starting a VM, several pattern based
substitutions are performed by vcycle. These patterns are in the form
##user_data___##. String values given to the option user_data_option_XXX
replace patterns of the form ##user_data_option_XXX##. The contents of
the files given to user_data_file_XXX options replace patterns of the
form ##user_data_file_XXX##. In both cases XXX are arbitrary strings
consisting of letters, numbers, and underscores.
The pattern ##user_data_x509_proxy## is replaced by the proxy created if the
user_data_proxy_cert and user_data_proxy_key options are given.
In addition, the following substitutions are performed automatically by
vcycle using data it holds internally:
.br
.B ##user_data_space##
is the vcycle space name.
.br
.B ##user_data_machinetype##
and
.B ##user_data_vmtype##
(deprecated)
are the name of the machinetype of this VM.
.br
.B ##user_data_machine_hostname##
and
.B ##user_data_vm_hostname##
(deprecated)
are the hostname given to the VM by Vcycle.
.br
.B ##user_data_manager_version##
and
.B ##user_data_vmlm_version##
(deprecated)
have the form "Vcycle v.v.v" where v.v.v is the Vcycle version.
.br
.B ##user_data_manager_hostname##
and
.B ##user_data_vmlm_hostname##
(deprecated)
are the hostname of the machine on which the Vcycle daemon is running.
.SH AUTHOR
Andrew McNab <[email protected]>
vcycled is part of vcycle: http://www.gridpp.ac.uk/vcycle/
.SH "SEE ALSO"
.BR vcycled(8),
.BR vcycle-cgi(8)