-
Notifications
You must be signed in to change notification settings - Fork 0
/
paper.tex
770 lines (667 loc) · 33.1 KB
/
paper.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
\documentclass[english,]{IEEEtran}
% \usepackage{draftwatermark}
% \SetWatermarkText{DRAFT}
% \SetWatermarkScale{0.25}
\usepackage{lmodern}
\usepackage{amssymb,amsmath}
\usepackage{ifxetex,ifluatex}
\usepackage{fixltx2e} % provides \textsubscript
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\else % if luatex or xelatex
\ifxetex
\usepackage{mathspec}
\else
\usepackage{fontspec}
\fi
\defaultfontfeatures{Ligatures=TeX,Scale=MatchLowercase}
\fi
% use upquote if available, for straight quotes in verbatim environments
\IfFileExists{upquote.sty}{\usepackage{upquote}}{}
% use microtype if available
\IfFileExists{microtype.sty}{%
\usepackage{microtype}
\UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts
}{}
\usepackage[unicode=true]{hyperref}
\hypersetup{
pdftitle={SkiffOS: Minimal Cross-compiled Linux for Embedded Containers},
pdfkeywords={Container, Embedded Linux, Robotics, Single
Board Computer},
pdfborder={0 0 0},
breaklinks=true}
\urlstyle{same} % don't use monospace font for urls
\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
\usepackage[shorthands=off,main=english]{babel}
\else
\usepackage{polyglossia}
\setmainlanguage[]{english}
\fi
\usepackage{csquotes}
\usepackage[backend=biber,style=ieee,natbib=true]{biblatex}
\renewcommand{\bibfont}{\footnotesize}
\addbibresource{paper.bib}
\usepackage{color}
\usepackage{fancyvrb}
\newcommand{\VerbBar}{|}
\newcommand{\VERB}{\Verb[commandchars=\\\{\}]}
\DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\}}
% Add ',fontsize=\small' for more characters per line
\newenvironment{Shaded}{}{}
\newcommand{\AlertTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{#1}}}
\newcommand{\AnnotationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{#1}}}}
\newcommand{\AttributeTok}[1]{\textcolor[rgb]{0.49,0.56,0.16}{#1}}
\newcommand{\BaseNTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{#1}}
\newcommand{\BuiltInTok}[1]{#1}
\newcommand{\CharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{#1}}
\newcommand{\CommentTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textit{#1}}}
\newcommand{\CommentVarTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{#1}}}}
\newcommand{\ConstantTok}[1]{\textcolor[rgb]{0.53,0.00,0.00}{#1}}
\newcommand{\ControlFlowTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{#1}}}
\newcommand{\DataTypeTok}[1]{\textcolor[rgb]{0.56,0.13,0.00}{#1}}
\newcommand{\DecValTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{#1}}
\newcommand{\DocumentationTok}[1]{\textcolor[rgb]{0.73,0.13,0.13}{\textit{#1}}}
\newcommand{\ErrorTok}[1]{\textcolor[rgb]{1.00,0.00,0.00}{\textbf{#1}}}
\newcommand{\ExtensionTok}[1]{#1}
\newcommand{\FloatTok}[1]{\textcolor[rgb]{0.25,0.63,0.44}{#1}}
\newcommand{\FunctionTok}[1]{\textcolor[rgb]{0.02,0.16,0.49}{#1}}
\newcommand{\ImportTok}[1]{#1}
\newcommand{\InformationTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{#1}}}}
\newcommand{\KeywordTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{\textbf{#1}}}
\newcommand{\NormalTok}[1]{#1}
\newcommand{\OperatorTok}[1]{\textcolor[rgb]{0.40,0.40,0.40}{#1}}
\newcommand{\OtherTok}[1]{\textcolor[rgb]{0.00,0.44,0.13}{#1}}
\newcommand{\PreprocessorTok}[1]{\textcolor[rgb]{0.74,0.48,0.00}{#1}}
\newcommand{\RegionMarkerTok}[1]{#1}
\newcommand{\SpecialCharTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{#1}}
\newcommand{\SpecialStringTok}[1]{\textcolor[rgb]{0.73,0.40,0.53}{#1}}
\newcommand{\StringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{#1}}
\newcommand{\VariableTok}[1]{\textcolor[rgb]{0.10,0.09,0.49}{#1}}
\newcommand{\VerbatimStringTok}[1]{\textcolor[rgb]{0.25,0.44,0.63}{#1}}
\newcommand{\WarningTok}[1]{\textcolor[rgb]{0.38,0.63,0.69}{\textbf{\textit{#1}}}}
\usepackage{longtable,booktabs}
\usepackage{supertabular}
\let\longtable\supertabular
\let\endlongtable\endsupertabular
\let\endhead\relax
% Fix footnotes in tables (requires footnote package)
\IfFileExists{footnote.sty}{\usepackage{footnote}\makesavenoteenv{long table}}{}
\IfFileExists{parskip.sty}{%
\usepackage{parskip}
}{% else
\setlength{\parindent}{0pt}
\setlength{\parskip}{6pt plus 2pt minus 1pt}
}
\setlength{\emergencystretch}{3em} % prevent overfull lines
\providecommand{\tightlist}{%
\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}
\setcounter{secnumdepth}{5}
% Redefines (sub)paragraphs to behave more like sections
\ifx\paragraph\undefined\else
\let\oldparagraph\paragraph
\renewcommand{\paragraph}[1]{\oldparagraph{#1}\mbox{}}
\fi
\ifx\subparagraph\undefined\else
\let\oldsubparagraph\subparagraph
\renewcommand{\subparagraph}[1]{\oldsubparagraph{#1}\mbox{}}
\fi
% set default figure placement to htbp
\makeatletter
\def\fps@figure{htbp}
\makeatother
\title{SkiffOS: Minimal Cross-compiled Linux for Embedded Containers}
\author{
\IEEEauthorblockN{Christian Stewart}
\IEEEauthorblockA{%
\\
Aperture Robotics LLC. \\
}
\date{}
\begin{document}
\maketitle
\begin{abstract}
Embedded Linux processors are increasingly used for real-time computing
tasks such as robotics and Internet of Things (IoT). These applications
require robust and reproducible behavior from the host OS, commonly
achieved through immutable firmware stored in read-only memory. SkiffOS
addresses these requirements with a minimal cross-compiled GNU/Linux
system optimized for hosting containerized distributions and
applications, and a configuration layering system for the Buildroot
embedded cross-compiler tool which automatically re-targets system
configurations to any platform or device. This approach cleanly
separates the hardware support from the applications. The host system
and containers are independently upgraded and backed-up Over-the-Air
(OTA).
\end{abstract}
\begin{IEEEkeywords}
Container;
Embedded Linux;
Robotics;
Single Board Computer\end{IEEEkeywords}
\hypertarget{introduction}{%
\section{Introduction}\label{introduction}}
Single Board Computer (SBC) vendors distribute "board support packages"
(BSPs) containing a mutable root filesystem with a package manager. Each
platform requires a unique combination of kernel, bootloader, and
firmware to operate, as a result BSPs typically lack portability.
Most Linux distributions use binary package managers to install and
upgrade applications and system files, managed by distribution
maintainers. Application dependencies co-exist with system dependencies,
often creating version conflicts resulting in out-of-date and
potentially vulnerable system software \autocite{foss}. Package
availability and versioning can vary across processor architectures.
Package managers operate on a mutable filesystem with imperative
management commands (i.e.~install, uninstall, upgrade) \autocite{nixos}.
Each modification to the system configuration diverges from the known
initial state, and can result in unpredictable changes in system
behavior such as broken boot, inaccessible ssh, or malfunctioning
drivers.
Declarative configuration models allow for reproducible behavior from
the OS. Embedded Linux systems such as IoT sensors, industrial
automation, TV set-top boxes, webcams, kiosks, and other products often
bundle a GNU/Linux system into read-only firmware based on declarative
configurations. This guarantees reproducible behavior, but comes with
significant disadvantages in usability: software typically cannot be
installed or upgraded without a system re-build and re-flash.
\hypertarget{skiffos-firmware-for-hosting-containers}{%
\section{SkiffOS: Firmware for Hosting
Containers}\label{skiffos-firmware-for-hosting-containers}}
Buildroot is a project providing a Makefile and KConfig-based tool for
automated cross-compilation of GNU/Linux systems, including toolchains,
kernels, drivers, userspaces, and bootloaders \autocite{buildroot}. It
includes over 2500 software packages providing support for all major
system utilities, and provides an easy to understand structure and
configuration language allowing for rapid development across a diverse
set of hardware configurations \autocite{buildrootrt}.
SkiffOS builds a minimal cross-compiled system for hosting containers
across a diverse set of compute platforms, with minimal variance in the
user experience. It emulates traditional firmware approaches with an
immutable operating system image which produces reproducible behavior,
while providing the utility of package-manager based GNU/Linux
distributions with containerized environments attached to persistent
storage.
Configuration layers bundle configuration files, Buildroot extension
packages, system files, build hooks, and installation/utility scripts
into named packages. Configurations are supplied for numerous target
systems and virtualization environments. Hardware-specific performance
optimizations are enabled at build time, producing a system image tuned
for the target computer. Makefile targets are available for formatting
and installing the system to boot media. Multiple system configurations
can be compiled in parallel with "workspaces."
The SkiffOS Git repository can be embedded in extension projects as a
sub-module and extended with out-of-tree configuration packages and
overrides. Buildroot is referenced as a sub-module within the SkiffOS
Git tree. Checksumming, package version pinning, and reproducible
offline builds are used to ensure that a given SkiffOS commit will
always produce identical output.
Existing system userspaces can be imported and used directly as
container images Package managers can then be used to install and manage
software independently from the host system. Multiple distributions and
containerized applications can be run in parallel. Workloads can be
defined as container images to enhance portability and reproducibility
\autocite{reworkflow}. Container management platforms such as Kubernetes
and Docker Swarm can be used to remotely deploy and monitor workloads
\autocite{autoveh}.
\hypertarget{implementation}{%
\section{Implementation}\label{implementation}}
SkiffOS is available under the MIT license, and references Buildroot as
a sub-module under the GPLv2 license, with a patch series providing
additional features and bug fixes. Changes are frequently submitted
upstream to the Buildroot mailing list.
As of Release 2020.11.7 the target support table is:
\begin{longtable}[]{@{}lll@{}}
\toprule
System & Config Package & Kernel \\ \addlinespace
\midrule
\endhead
Apple Macbook (Intel) & apple/macbook & 5.11.2 \\ \addlinespace
BananaPi M1 & bananapi/m1 & 5.11.2 \\ \addlinespace
BananaPi M1+/Pro & bananapi/m1plus & 5.11.2 \\ \addlinespace
BananaPi M2+ & bananapi/m2plus & 5.11.2 \\ \addlinespace
BananaPi M3 & bananapi/m3 & 5.11.2 \\ \addlinespace
Docker Container & virt/docker & N/A \\ \addlinespace
Intel x86/64 & intel/x64 & 5.11.2 \\ \addlinespace
Msft Windows (WSL) & virt/wsl & N/A \\ \addlinespace
NVIDIA Jetson Nano & jetson/nano & 4.9.140 \\ \addlinespace
NVIDIA Jetson TX2 & jetson/tx2 & 4.9.140 \\ \addlinespace
Odroid C2 & odroid/c2 & tb-5.9.16 \\ \addlinespace
Odroid C4 & odroid/c4 & tb-5.9.16 \\ \addlinespace
Odroid HC1/2, XU3/4 & odroid/xu & tb-5.9.16 \\ \addlinespace
Odroid U & odroid/u & tb-5.9.16 \\ \addlinespace
OrangePi Lite & orangepi/lite & 5.11.2 \\ \addlinespace
OrangePi Zero & orangepi/zero & 5.11.2 \\ \addlinespace
PcDuino 3 & pcduino/3 & 5.11.2 \\ \addlinespace
PcEngines APU2 & pcengines/apu2 & 5.11.2 \\ \addlinespace
Pi 0 & pi/0 & 5.4.72 \\ \addlinespace
Pi 1 & pi/1 & 5.4.72 \\ \addlinespace
Pi 3 (and 1/2) & pi/3 & 5.4.72 \\ \addlinespace
Pi 4 & pi/4 & 5.4.72 \\ \addlinespace
Pine64 H64 & pine64/h64 & 5.8.0 \\ \addlinespace
PinePhone & pine64/phone & megi-5.9.11 \\ \addlinespace
Pinebook Pro & pine64/book & 5.11.2 \\ \addlinespace
Qemu (VM) & virt/qemu & 5.11.2 \\ \addlinespace
Virtualbox (VM) & virt/qemu & 5.11.2 \\ \addlinespace
Rockpro64 & pine64/rockpro64 & 5.9.0 \\ \addlinespace
\bottomrule
\end{longtable}
Legal information and licenses for all dependencies can be bundled
together with the \texttt{make\ legal-info} command. Some board support
packages include proprietary binary blobs (typically firmware) and are
denoted as "Proprietary" in the produced licensing information bundle.
\hypertarget{configuration-layers}{%
\subsection{Configuration Layers}\label{configuration-layers}}
The SkiffOS base configuration, builds a minimal host operating system
with hardware support, OpenSSH server, and system management tools.
Enabling additional configuration layers adds use-case specific
functionality. Configurations are organized into logical units called
"layers" with the following structure:
\begin{itemize}
\tightlist
\item
\texttt{cflags}: additional target compiler flags
\item
\texttt{buildroot}: buildroot configuration fragments
\item
\texttt{buildroot\_ext}: buildroot extensions
\item
\texttt{buildroot\_patches}: buildroot package patches
\item
\texttt{extensions}: utility commands
\item
\texttt{hooks}: scripts hooking pre/post build steps
\item
\texttt{kernel}: kernel configuration fragments
\item
\texttt{kernel\_patches}: kernel .patch files
\item
\texttt{root\_overlay}: root overlay files
\item
\texttt{metadata}: metadata files
\begin{itemize}
\tightlist
\item
\texttt{commands}: targets in "extensions" makefile
\item
\texttt{dependencies}: comma-separated layers
\item
\texttt{description}: single-line description
\item
\texttt{unlisted}: if exists, hidden from "help"
\end{itemize}
\item
\texttt{resources}: support files
\item
\texttt{scripts}: used by extensions and/or hooks
\item
\texttt{uboot}: u-boot configuration fragments
\item
\texttt{uboot\_patches}: u-boot .patch files
\end{itemize}
Configuration layers can override options set by previous layers, making
it simple to re-target configurations to various compute platforms by
merging with the desired hardware configuration layer. The set of
desired configuration layers is defined as an ordered comma-separated
list, for example:
\texttt{SKIFF\_CONFIG=pi/4,core/gentoo}
SkiffOS can be extended \autocite{skiffext} by adding it as a sub-module
of a project repo. Project configurations can then be specified in
additional configuration layers.
\hypertarget{persistent-data-partition}{%
\subsection{Persistent data partition}\label{persistent-data-partition}}
The persist partition contains a \texttt{skiff} directory with:
\begin{itemize}
\tightlist
\item
\texttt{connections}: network-manager connections
\item
\texttt{core}: skiff-core configuration and state
\item
\texttt{docker}: docker state and storage
\item
\texttt{etc}: configuration tree overlay
\item
\texttt{hostname}: system hostname
\item
\texttt{journal}: systemd-journald system logs
\item
\texttt{keys}: public keys for access to "root" user
\item
\texttt{ssh}: ssh server keys and persistent configuration
\end{itemize}
System startup scripts mount the persist partition and create the file
structure at boot time. Most configurations will create a memory
swapfile to avoid failure due to an out-of-memory condition. OpenSSH is
configured for public-key access with the persist "keys" directory or in
the OS image.
The persist partition is automatically resized to fit the remainder of
the available storage space on first boot. Bootloaders such as u-boot
are sometimes copied to the beginning of the storage media. Some systems
use more complex partition layouts mandated by the hardware vendor.
Skiff and Buildroot's flexible configuration language supports this
variance between target systems.
\hypertarget{over-the-air-ota-upgrades}{%
\subsection{Over-the-air (OTA)
upgrades}\label{over-the-air-ota-upgrades}}
The SkiffOS system typically consists of fewer than five files,
including the root filesystem squashfs/initramfs, kernel image, and
kernel modules squashfs. The kernel and immutable boot system can be
atomically upgraded by replacing these files at run-time. The
\texttt{push\_image.sh} script is provided, using \texttt{rsync} and
\texttt{ssh} to upload the updated files to a running system. In some
cases, the firmware and/or bootloader is also upgraded by the script.
\texttt{./scripts/push\_image.sh\ root@my-device-ip}
\hypertarget{kconfig-configuration-fragments}{%
\subsection{Kconfig configuration
fragments}\label{kconfig-configuration-fragments}}
Buildroot organizes software components into "packages." Packages can be
enabled and configured using the Kconfig language. The available options
can be explored using the \texttt{make\ menuconfig} or
\texttt{make\ xconfig} configuration menus. Configuration options are
specified in "fragments" that are merged together into a single
".config" file and provided to the Buildroot build system. The Buildroot
build system manages merging together kconfig and u-boot configuration
fragments and applying patches.
Example buildroot configuration fragment:
\begin{verbatim}
BR2_LINUX_KERNEL_DEFCONFIG="versatile"
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.11.2"
\end{verbatim}
Buildroot and the Linux kernel use the Kconfig configuration language.
The available options can be explored with \texttt{make\ br/menuconfig}
and \texttt{make\ br/kernel-menuconfig} configuration menus.
Configuration fragments are merged together in \texttt{SKIFF\_CONFIG}
order followed by lexicographic filename order.
This example kernel configuration fragment enables the ext3/ext4
filesystem:
\begin{verbatim}
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_POSIX_ACL=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_EXT4_POSIX_ACL=y
\end{verbatim}
The \texttt{m} option denotes a feature built as a kernel module instead
of "built-in."
\hypertarget{buildroot-extensions}{%
\subsection{Buildroot extensions}\label{buildroot-extensions}}
Configuration layers can extend Buildroot with packages in the
\texttt{buildroot\_ext} directory. The \texttt{buildroot\_patches}
directory contains patches for Buildroot packages, i.e.~fixes for
platform errata. For convenience, kernel and uboot patches can also be
specified in the \texttt{kernel\_patches} and \texttt{uboot\_patches}
directories.
\hypertarget{skiffos-extensions}{%
\subsection{SkiffOS extensions}\label{skiffos-extensions}}
The \texttt{extensions} configuration layer directory contains a
Makefile which implements custom commands made available to the user in
the help screen:
\begin{verbatim}
cmd/pi/common/format: Format a SD card.
cmd/pi/common/install: Install to a SD card.
\end{verbatim}
Commands are prefixed by their layer name and can be executed as
Makefile targets, for example, \texttt{make\ cmd/pi/common/install}, and
are declared in the \texttt{metadata/commands} text file:
\begin{verbatim}
format Format a SD card and install bootloader.
install Installs to a formatted SD card.
\end{verbatim}
\hypertarget{root-filesystem-overlay}{%
\subsection{Root filesystem overlay}\label{root-filesystem-overlay}}
The \texttt{root\_overlay} trees are copied to the target filesystem
image at the end of the build in the order that the layers were
specified. This is used to add additional configuration files or scripts
to the root SkiffOS system image. For example, the \texttt{pi/common}
layer adds configuration under the \texttt{etc} directory, and firmware
configuration under the \texttt{usr} directory.
\hypertarget{temporary-local-overrides}{%
\subsection{Temporary local overrides}\label{temporary-local-overrides}}
Configuration overrides can be specified in the \texttt{overrides}
directory tree, which contains additional configuration layers
implicitly added to the build. For example, temporary local buildroot
overrides for all workspaces can be declared in
\texttt{overrides/buildroot}, and kernel configuration fragments
affecting the \texttt{pi4} workspace alone can be declared in
\texttt{overrides/workspaces/pi4/kernel}.
\hypertarget{skiff-core-containerized-environments}{%
\section{Skiff Core: Containerized
Environments}\label{skiff-core-containerized-environments}}
Skiff Core includes the Docker containerization runtime and a Go program
for automating the creation and initial setup of containerized
environments. It can be enabled with the \texttt{skiff/core}
configuration layer. User sessions are routed to the container assigned
to their account. Multiple OS distributions can be installed
simultaneously on a single machine. Existing userspaces including
vendor-provided software images can be imported as container images.
The \texttt{skiff-core} binary is configured as the user shell, and
intercepts incoming SSH sessions to redirect them to the corresponding
container. The container setup process is displayed if the container(s)
are not yet ready. The usual userspace init daemon, typically
\texttt{systemd}, is run as PID 1, and standard approaches for defining
\texttt{systemd} services and the \texttt{systemctl} CLI tool function
identically to when running without containerization.
The "core" system can be updated or rolled-back independently from the
"root" operating system. Mountpoints are used to mount user home
directories and other temporary paths so that "docker export" and
"docker save" include system files only. Containers are portable between
machines of similar architecture without target-specific configuration.
The container system is configured with a YAML file:
\begin{Shaded}
\begin{Highlighting}[]
\FunctionTok{containers}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{core}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{image}\KeywordTok{:}\AttributeTok{ skiffos/skiff{-}core{-}gentoo:latest}
\AttributeTok{ }\FunctionTok{mounts}\KeywordTok{:}
\AttributeTok{ }\KeywordTok{{-}}\AttributeTok{ /dev:/dev}
\AttributeTok{ }\KeywordTok{{-}}\AttributeTok{ /etc/resolv.conf:/etc/resolv.conf:ro}
\AttributeTok{ }\KeywordTok{{-}}\AttributeTok{ /mnt/persist/data:/home}
\AttributeTok{ }\KeywordTok{[}\AttributeTok{...}\KeywordTok{]}
\FunctionTok{users}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{core}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{container}\KeywordTok{:}\AttributeTok{ core}
\AttributeTok{ }\FunctionTok{containerUser}\KeywordTok{:}\AttributeTok{ core}
\AttributeTok{ }\FunctionTok{auth}\KeywordTok{:}\AttributeTok{ }\KeywordTok{\{}\FunctionTok{copyRootKeys}\KeywordTok{:}\AttributeTok{ }\CharTok{true}\KeywordTok{\}}
\FunctionTok{images}\KeywordTok{:}
\AttributeTok{ skiffos/skiff{-}core{-}gentoo}\FunctionTok{:latest}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{pull}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{policy}\KeywordTok{:}\AttributeTok{ ifnotexists}
\AttributeTok{ }\FunctionTok{registry}\KeywordTok{:}\AttributeTok{ quay.io}
\AttributeTok{ }\FunctionTok{build}\KeywordTok{:}
\AttributeTok{ }\FunctionTok{source}\KeywordTok{:}\AttributeTok{ /opt/skiff/coreenv/base}
\end{Highlighting}
\end{Shaded}
Several OS distribution configurations are available as configuration
layers. If a pre-built image is unavailable, or the \texttt{pull}
section of \texttt{skiff-core.yaml} is empty, the system will instead
build the included \texttt{Dockerfile}.
\begin{longtable}[]{@{}lll@{}}
\toprule
OS Distribution & Config Package & Website \\ \addlinespace
\midrule
\endhead
Gentoo &
\href{https://github.com/skiffos/skiffos/tree/2020.11.7/configs/core/gentoo}{core/gentoo}
& \href{https://gentoo.org}{gentoo.org} \\ \addlinespace
Manjaro &
\href{https://github.com/skiffos/skiffos/tree/2020.11.7/configs/core/manjaro}{core/manjaro}
& \href{https://manjaro.org}{manjaro.org} \\ \addlinespace
NixOS &
\href{https://github.com/skiffos/skiffos/tree/2020.11.7/configs/core/nixos/}{core/nixos}
& \href{https://nixos.org}{nixos.org} \\ \addlinespace
Ubuntu &
\href{https://github.com/skiffos/skiffos/tree/2020.11.7/configs/skiff/core/}{skiff/core}
& \href{https://ubuntu.com}{ubuntu.com} \\ \addlinespace
\bottomrule
\end{longtable}
Additional images optimized for specific use cases are available:
\begin{longtable}[]{@{}ll@{}}
\toprule
Image & Description \\ \addlinespace
\midrule
\endhead
gentoo-lto-exwm & LTO, Apps, Emacs X11 Workflow \\ \addlinespace
gentoo-kde & KDE Desktop w/ apps \\ \addlinespace
gentoo-lto & O3, Graphite, Link-time optimize \\ \addlinespace
nasa-cfs & Flight systems framework \\ \addlinespace
nasa-fprime & Flight systems framework \\ \addlinespace
pinephone-neon & KDE Neon for PinePhone \\ \addlinespace
-manjaro-kde & Manjaro KDE for PinePhone \\ \addlinespace
\bottomrule
\end{longtable}
The NASA Fprime\autocite{fprime} and NASA cFS\autocite{nasacfs} images
are currently based on Ubuntu. To pull, for example, the
\texttt{gentoo-lto} image, which includes the Gentoo core image with
\texttt{gentooLTO} optimizations, the Docker CLI command is:
\texttt{docker\ pull\ skiffos/skiff-core-gentoo-lto:latest}
When using "Skiff Core," the Docker container engine is used, which
leverages Linux namespaces and not virtualization or emulation,
implementing "OS-level virtualization as opposed to hardware
virtualization" \autocite{virtperf}. The PID namespace is used to allow
running the usual system initialization and management daemons as the
privileged PID 1 within the container.
The results of \autocite{iotcontainer} show "an almost negligible impact
of the {[}Docker container{]} layer in terms of performance, if compared
to native execution." While throughput is not significantly affected,
the results of \autocite{autoveh} show that namespacing can cause a
measurable signal processing delay. To mitigate potential impacts of
processing latency, most container isolation is disabled.
\hypertarget{comparison-with-existing-approaches}{%
\section{Comparison with Existing
Approaches}\label{comparison-with-existing-approaches}}
This section discusses several of the current approaches in use today,
and their disadvantages when applied to embedded Linux and/or robotics
development:
\hypertarget{binary-package-distributions}{%
\subsubsection{Binary Package
Distributions}\label{binary-package-distributions}}
Traditional board-support packages use binary package distribution
systems such as Debian's Advanced Package Tool (APT). Several key
disadvantages of this approach are lack of hardware-specific
optimizations, reliance on third-party infrastructure for builds and
maintenance, inability to reproduce a system from source code, and
difficulties with portability. The rolling nature of package upgrades
introduces unpredictable behavior when upgrading a system, particularly
when a long time has passed since the previous upgrade \autocite{foss}.
Current widely-used package management tools install system files,
firmware, and user applications together into a single mutable "root"
filesystem. Imperative install, remove, and upgrade commands instruct
the package manager to perform operations on the system. Interrupted
package manager operations might leave some files in a partially written
state. If the affected (now corrupted) files are critical to system boot
and/or reachability, the only recourse to fix the system may be to
remove the boot media from the machine, connect it to a different
device, and fix the issue manually.
SkiffOS addresses the concern of always having reproducible boot-up and
reachability behavior in embedded systems by booting to an immutable
root filesystem image, mounting persistent storage, and running user
applications and operating systems inside lightweight Linux containers.
This mitigates the risks of mid-upgrade power brownout: the immutable
portion of the system is always reachable, allowing the user can connect
to the container and fix the problem without physical intervention.
Optimization of compute performance and reduction of energy usage are
important considerations for resource-constrained embedded devices (the
"power budget"). For most robotics applications, the "power budget" is a
significant factor in determining maximum range/endurance. Compilation
of the operating system and applications from source allows fine-tuning
of build output to the hardware to make maximum use of energy saving
optimizations.
Backing up the system by creating a full bit-for-bit copy is a
time-consuming process which often leads to infrequent backups and, as a
result, occasional data loss due to storage card failure. SkiffOS
provides an alternate approach, in which existing OS distributions can
be imported as containers. Containers are easy to back up, restore,
roll-back, and are portable between machines of similar architecture.
Any vendor-provided BSP can be imported as a container and used without
sacrifices to workflow or compatibility.
\hypertarget{nixos}{%
\subsubsection{NixOS}\label{nixos}}
NixOS \autocite{nixos} argues that imperative package managers
destructively update the state of the system, leading to "inability to
roll back changes easily, to run multiple versions of a package
side-by-side, to reproduce a configuration deterministically on another
machine, or to reliably upgrade a system." NixOS includes a declarative
package manager which performs modifications according to a
specification of target system state. Nix can perform atomic upgrades
with the ability to roll-back.
SkiffOS is designed around the same principles of immutability and
declarative configuration, but provides these guarantees through
compilation of the system in advance, loading an immutable and ephemeral
root system at boot-up. It also addresses the other issues described by
the NixOS paper, including running multiple systems side-by-side in
Linux containers, and reproducing a system configuration
deterministically at a later time.
NixOS breaks the Filesystem Hierarchy Standard (FHS) \autocite{fhs}
declared by the Linux foundation and followed by most distributions,
instead using a flat symbolic link-based structure. This causes
incompatibility with software not compiled by Nix. On the other hand,
Buildroot \autocite{buildroot} (and therefore SkiffOS) follow the FHS
for compatibility with existing glibc-based binaries.
SkiffOS focuses on providing a minimal "shim" to abstract away the
differences between hardware (particularly Single-Board Computers) such
that the containers are portable to new platforms. It does not
distribute complex packages like web browsers, graphical interfaces, and
other user applications. This responsibility is left to the
distributions running in Linux containers attached to persistent storage
and/or customization of the cross-compiled system.
NixOS has been integrated with SkiffOS as a "core" container
configuration. This approach uses Buildroot to manage the hardware
specifics, and NixOS to handle declarative system configuration and
rolling application upgrades in containers. Software not compiled by Nix
can still be run in a concurrent container.
\hypertarget{buildroot}{%
\subsubsection{Buildroot}\label{buildroot}}
The Buildroot \autocite{buildroot} project provides a automated system
cross-compiler. Users typically download a Buildroot release, create a
configuration by hand, and store this along with their other project
files. It focuses on providing a toolset for Embedded Linux developers
to produce system images for use with a single target platform and/or
product.
SkiffOS extends Buildroot to simplify this process with an easy to
understand configuration layering architecture, which merges together
the platform support configs with the selected components to configure
the build automatically. The configuration layers can be stored
externally to the SkiffOS tree. Existing Buildroot projects can be
adapted as configuration layers and ported to any of the available
target configurations.
\hypertarget{yocto-project}{%
\subsubsection{Yocto Project}\label{yocto-project}}
The Yocto Project \autocite{yocto} is closely comparable to Buildroot,
and is primarily focused on the python-based "bitbake" tool, which is
derived from Gentoo's "portage," with a focus on cross-compilation of
complex embedded Linux systems. This is contrasted with Buildroot's
focus on simplicity, using the Makefile and Kconfig architecture
\autocite{yoctobr}. Yocto system configurations are fragmented into
"overlay" repositories from various sources, while Buildroot focuses on
a consolidated and curated package tree with strict and opinionated code
style. Yocto has many more packages than Buildroot.
Buildroot \autocite{buildroot} forms a linear commit history with
release checkpoints, and Skiff releases pin the version of the Buildroot
sub-module along with the versions of the device firmware, kernels, and
board support files. A given Git checkout is reproducible forever
\autocite{buildrootrt}. Yocto setups may suffer from loss of
availability of overlay repositories, or de-synchronization between
changes to the overlay repositories and changes to the core system
configurations.
\hypertarget{conclusion}{%
\section{Conclusion}\label{conclusion}}
SkiffOS offers reproducible system behavior, offline compilation,
immutable host system for containers attached to persistent storage,
board-specific build re-targeting via config layers, and Over-The-Air
upgrade/downgrade. The host system and/or containers can be easily
customized for specific use cases with cross-platform configuration
layers. The "Skiff Core" workflow provides an easy way to import any
existing Linux userspace such that users will likely not realize they
are now working in a container. Background workloads can run in parallel
Linux containers, independently isolated from the user workspace(s)
including resource management and quality-of-service (QoS).
SkiffOS is available under the MIT license at:
\url{https://github.com/skiffos/skiffos}
Buildroot is available under the GPLv2 license at:
\url{https://buildroot.org}
\clearpage
\printbibliography
\end{document}