Support migration with more sophisticated disk changes

[jschmid1]

Initial setup:

disk1:
  1: data(filestore)
  2: journal
disk2:
  1: data(filestore)
  2: journal

and you want to convert to bluestore with a dedicated wal/db

disk1:
  1: data(bluestore)
disk2:
  1: WAL
  2: db

means not only a change in format but also in layout. Consequently the pillar data changes to have only one entry instead of the initial two. When parsing from the grains, which represent the old structure, we will most likely end up with KeyError as the corresponding disk now only serves as db/wal device.

These options come to my mind:

• a) Change the way we generate diffs between actual <-> desired
• b) Add exclusive db/wal/journal devices to the profile-X/stack.../*.yml
• c) Adapt OSDConfig to tolerate the absence of a disk in the pillar for special occasions

[swiftgist]

We are likely missing the comparison of grains to the pillar configuration. Will is_incorrect behave correctly if we tolerate a KeyError? If an OSD is no longer listed in the pillar, then its configuration is incorrect and should be removed. The redeploy would take care of it if we got that far.

Likely need to study the different paths and make sure not to impact the normal creation process.

[jschmid1]

Will is_incorrect behave correctly if we tolerate a KeyError?

I tried to sketch the path we take but got interrupted by other tasks. I'll try again betimes.

[jschmid1]

@swiftgist
After giving the potential kinks some thoughts, I'd advocate for only allowing substantial layout changes when the 'per node' path is selected. For most layout changes we can definitely compute the resulting changes.. but opportunities to mess things up are omnipresent in this field :/

Thinking of disk sizes that need to be checked if the resulting xxx_size is bigger than the accumulated current partitions + the 'about to be created' one. Rather substantial changes in the OSDConfig class, messing with the reformatting of grains and pillar and I bet much more when diving into it.

In the 'per node' branch we can first purge all disks before redeploying them, which makes things way easier. Most cases will be covered with the 'per node' option either, I assume.

TLDR; Although probably doable, only support more sophisticated cluster layout changes with the 'per node' approach.

[jschmid1]

targeted with #433