From 9bf0ff362a45b0a312c51a3118c74180500ebbcb Mon Sep 17 00:00:00 2001 From: Phil Dibowitz Date: Mon, 27 Mar 2023 18:15:08 -0700 Subject: [PATCH] sync fb_init with upstream Summary: This one was tricky. I basically just excluded default.rb and site_settings.rb Test Plan: --- cookbooks/fb_init/README.md | 105 ++++++++++++++++++------ cookbooks/fb_init/attributes/default.rb | 26 ++++-- cookbooks/fb_init/recipes/firstboot.rb | 21 +++++ 3 files changed, 124 insertions(+), 28 deletions(-) create mode 100644 cookbooks/fb_init/recipes/firstboot.rb diff --git a/cookbooks/fb_init/README.md b/cookbooks/fb_init/README.md index 12db344..1d38150 100644 --- a/cookbooks/fb_init/README.md +++ b/cookbooks/fb_init/README.md @@ -9,27 +9,33 @@ Requirements Attributes ---------- +* node['fb_init']['firstboot_os'] +* node['fb_init']['firstboot_tier'] Usage ----- -This cookbook includes all of the opensource Facebook cookbooks. You may not be -ready to use them all, or you may want to include additional stuff, so adjust to -taste. +This cookbook includes all of the open source Facebook cookbooks. You may not +be ready to use them all, or you may want to include additional stuff, so +adjust to taste. We've gone ahead and included some extra "HERE: " comments on where we've put other cookbooks that are internal to give you a better idea of our full -runlist. We hope to be able to release more of these as time allows and where +run-list. We hope to be able to release more of these as time allows and where appropriate. -It is highly recommended you read through the README.md in the root of GitHub -repo as well as Philosophy.md. The general idea though is that cookbooks are -ordered least specific to most specific. This allows a small core team to make -APIs and defaults and then let individual service owners' cookbooks at the end -overwrite whatever they ened to. This also ensures that all things the service -owner chooses not to bother with are setup to sane settings by the core group -at your site. +### Philosophy -The idea is that your runlists will look include this first, then everything +It is highly recommended you read through the +[README.md](https://github.com/facebook/chef-cookbooks/blob/main/README.md) +in the root of GitHub repo as well as +[Philosophy.md](https://github.com/facebook/chef-utils/blob/main/Philosophy.md). +The general idea though is that cookbooks are ordered least specific to most +specific. This allows a small core team to make APIs and defaults and then let +individual service owners' cookbooks at the end overwrite whatever they need +to. This also ensures that all things the service owner chooses not to bother +with are setup to sane settings by the core group at your site. + +The idea is that your run-lists will look include this first, then everything else. This cookbook should be every "core cookbook" that provides APIs for everyone else. @@ -45,18 +51,71 @@ It's useful to think of things in a 3-pass system: cookbooks the core OS team writes that should be applicable in general to all machines unless someone has a more specific desire. Owners then can include other cookbooks that are more specific - maybe for a specific - cluster, location, type of service. Finally the last cookbooks should be the - most specific ones for that service or machine which gets the final say. - Anytime someone removes a node assignment the next-most-specific setting - will take precdent. + cluster, location, type of service. Finally the last cookbooks should be + the most specific ones for that service or machine which gets the final + say. Anytime someone removes a node assignment the next-most-specific + setting will take precdent. * Consume APIs Everyone who uses any API is generally the cookbook that provides that API, - so APIs must be consumed only at runtime: templates, ruby_blocks, providers, - etc. + so APIs must be consumed only at runtime: templates, ruby_blocks, + providers, etc. -We have provided an early recipe called site-settings.rb in which you can set +We have provided an early recipe called `site-settings.rb` in which you can set the defaults for your organization for all settings Facebook cookbooks provide. -For example, setting all the most reasoanble sysctl settings here is advisable -- then let others override them in their later cookbooks. Assuming fb_init is -the first thing in your runlist, this is basically the first thing, so any -other cookbooks in your runlist will have time to overwrite them. +For example, setting all the most reasonable sysctl settings here is advisable +- then let others override them in their later cookbooks. Assuming `fb_init` is +the first thing in your run-list, this is basically the first thing, so any +other cookbooks in your run-list will have time to overwrite them. + +### Firstboot + +Sometimes it's necessary to distinguish the very first Chef run after +provisioning from subsequent ones, e.g. to do some run-once setup. It's worth +noting this is usually the wrong approach: it is much better to write +idempotent Chef code instead. Alternatively, it can be useful to *prevent* +something from happening during the first run (though again, this should be +last resort). + +To do this, the provisioning infrastructure can drop files on disk to tell Chef +it's being run in a specific phase. We choose to split the provisioning process +in two phases: +* an *OS run*, which is the first Chef run after provisioning -- it takes care + of setting up the basic OS services on the machine, but doesn't do any + application-specific customization +* a *tier run*, which is the first Chef run after the OS run (i.e. the second + Chef run after provisioning, assuming the OS run was successful) -- it takes + care of setting up any application-specific settings and services needed; for + example, in the case of a webserver this is where we would install and + configure Apache + +On the Chef side, we then provide two attributes to expose the phase: +* `node['fb_init']['firstboot_os']` is true on all runs before the first time + successful *OS run* happens; this checks for `/root/firstboot_os` + (or `/var/root/firstboot_os` on macOS) +* `node['fb_init']['firstboot_tier']` is true on all runs before the first time + successful *tier run* happens; this checks for `/root/firstboot_tier` + (or `/var/root/firstboot_os` on macOS). + +These attributes are also wrapped in `fb_helpers` by convenience node methods: +* `node.fistboot_os?` for `node['fb_init']['firstboot_os']` +* `node.firstboot_tier?` for `node['fb_init']['firstboot_tier']` +* `node.firstboot_any_phase?` to encompass both + +The reason these are implemented as attributes, and not just node methods, is +so they can be consumed by Chef handlers for reporting purposes. + +In `/etc/chef/client.rb`, we also setup a special run-list for the *OS run*: + +```ruby +if File.exists?('/root/firstboot_os') + puts 'INFO: firstboot_os mode, running with limited run_list' + base = '/etc/chef/run-list-base.json' + File.write(base, '{"run_list":["role[fb_base]"]}') unless File.exist?(base) + json_attribs '/etc/chef/run-list-base.json' +else + json_attribs '/etc/chef/run-list.json' +end +``` + +Note that for this to work properly, it's important to ensure the flag files +are deleted once the relevant phase has been successfully completed. diff --git a/cookbooks/fb_init/attributes/default.rb b/cookbooks/fb_init/attributes/default.rb index 3f43c07..e732cd1 100644 --- a/cookbooks/fb_init/attributes/default.rb +++ b/cookbooks/fb_init/attributes/default.rb @@ -1,14 +1,30 @@ # vim: syntax=ruby:expandtab:shiftwidth=2:softtabstop=2:tabstop=2 +# +# Copyright (c) 2018-present, Facebook, Inc. +# All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. # This stuff should go in an ohai plugin or some-such shorthostname = node['hostname'] trimmed_hostname = shorthostname.tr('0-9', '') pieces = trimmed_hostname.split('-') - env = 'prod' tier = pieces[1] -default['tier'] = tier -default['env'] = env - -default['fb_init'] = {} +default['fb_init'] = { + 'firstboot_os' => File.exist?('/root/firstboot_os'), + 'firstboot_tier' => File.exist?('/root/firstboot_tier'), + 'tier' => tier, + 'env' => env, +} diff --git a/cookbooks/fb_init/recipes/firstboot.rb b/cookbooks/fb_init/recipes/firstboot.rb new file mode 100644 index 0000000..736d818 --- /dev/null +++ b/cookbooks/fb_init/recipes/firstboot.rb @@ -0,0 +1,21 @@ +# +# Cookbook Name:: fb_init_sample +# Recipe:: firstboot +# +# Copyright (c) 2019-present, Facebook, Inc. +# All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# HERE: put any non-OS specific firstboot logic here