A collection of topics related to (Embedded) Linux that a professional in this field should be familiar with.
The general chapters and main topics associated to the individual subchapters are meant to represent the broad knowledge that can be expected from any Embedded Linux professional. For some topics, detailed knowledge could be expected from more advanced professionals - the corresponding sections can be expanded with a click.
Beyond this point, there is only a rough guidance on what level of detail could be expected in relation to a professional's experience grade, represented by the level of indentation.
Contents
Some helpful learning resources are attached to the individual topics in the respective details section.
General concepts & properties of any Linux based system (including desktop or server distributions). A solid foundation here allows to reuse as many OS mechanisms as possible and to avoid building conceptually broken software.
Resources:
The Linux Kernel
How the kernel operates and interacts with the applications on the system.
Subtopics:
- scheduling
- signals
Kernel- & Userspace
How the two separate runtime contexts differ and interact.
Subtopics:
- system calls
Everything is a File
How the system's entities are represented to the user.
Subtopics:
- file descriptors
- symlinks, hardlinks
Processes & Threads
How application logic is executed.
User & Access Management
How access to and ownership of data and execution logic is handled.
Subtopics:
- Access Control Lists (ACL)
- Passwords & keys
- Real & effective IDs
- Users (
id,whoami,sudo,su) - Ownership (
chown,chmod)
System Startup & Shutdown
How the system enters & quits operation.
From 1st stage to 2nd stage bootloader, kernel (+ initramfs) and finally the system manager (systemd) and userspace.
System Root
How the filesystem is structured, visible and protected against manipulation.
Subtopics:
- rootfs
chroot
Block Device Filesystems
How to choose and work with the filesystems on disc.
Subtopics:
- types (
ext4,squashfs,ubifs) - utilities
mkfsfsck
- mounting (
mount,umount)
Special Filesystems
How to use the filesystems exposing kernel features or virtually extending the system.
Subtopics:
procfssysfstmpfsoverlayfs
Partition Table
How to define and interpret the system's partition table.
Subtopics:
- formats (
msdos,gpt) - utilities (
parted)
Mounting
How to add filesystems to the rootfs.
Subtopics:
- formats (
msdos,gpt) - utilities (
parted)
Event Handling
How to detect and react on (file) events.
Subtopics:
ppoll,select
Interprocess Communication (IPC)
How to use built-in features to communicate between processes.
Subtopics:
- sockets
- pipes
- shared memory API (
shm_open,mmap, ...) - message queues
- signals
Orchestration
How to manage dependencies, resources and the lifetime of applications on the system.
Subtopics:
systemd- units
- overrides
- utilities (
systemctl,journalctl,systemd-run,systemd-analyze, ...)
cgroups
Monitoring
How to monitor applications on the system.
Subtopics:
- Software watchdog (
systemd_notify)
Device Rules
How to integrate external devices.
Subtopics:
udev
Networking & Protocols
How to interconnect multiple devices in a network.
Subtopics:
- ISO/OSI protocol layers and common protocols (IPv4, IPv6, TCP, UDP, IEEE 802.11)
- Configuring network interfaces in Linux (netctl, NetworkManager, ConnMan, iwd, wpa_supplicant, systemd-networkd, ...)
- Troubleshooting (
ping,traceroute,ssh,netstat,nmap,dig, Wireshark)
Topics mostly relating to embedded devices and their specific product use case.
General resources:
Bootloader
How to interact with and/or extend the behavior of the bootloader.
Subtopics:
U-Boot- Repository structure
- Configuring / patching u-boot
- u-boot environments
fw_*envtools
bareboxgrub- grub.env
- (
efi)
Device Trees
How to enable hardware features in the operating system.
Resources:
Drivers
How to use (or implement) drivers for extended system features.
Resources:
Peripherals
How to interact with peripherals on the hardware board.
Subtopics:
- HSM / TPM
- Flash memory
- eMMC
- NAND/NOR
Constrained Devices Support
How to use the reduced feature set of embedded-specific tools.
Subtopics:
busybox
Cryptography
How to implement security features with existing libraries.
Subtopics:
openssllibsodiumwolfssl- Mbed TLS
Binaries
How binaries are assembled in Linux.
Subtopics:
- static & dynamic linking
RPATH,LD_PRELOADand other environment variables- cross-compilation, canadian cross concept
- troubleshooting (
ldd,readelf,objdump,file)
Flashing tools
How to deploy software on the actual hardware.
System testing
What frameworks to use when testing with hardware-in-the-loop.
Knowledge helpful when working with Linux-based systems.
File operations
How to use the built-in tools.
Subtopics:
ls,cat,head,tail,less,echogrep,awk
Editors
How to view and edit files on the device.
Subtopics:
vi/vimnano
Scripting
How to implement complex logic using the native command line.
Subtopics:
- sh / bash
- zsh, hsh, ...
- interoperability (e.g. avoiding bashisms)
Process Management
How to use the built-in process related tools.
Subtopics:
fuser,lsofkill
Logging
How to leverage built-in logging features.
Subtopics:
dmesg- syslog (
logger) journald
Application Analytics
How to get performance metrics, profiles and call graphs for applications.
Subtopics:
perf,gprof,hotspotvalgrind,kcachegrind, ...
System Inspection
How to gather information about the operating system and applications.
Subtopics:
- Disk usage (
df,du) - Runtime stats (
ps,top)
Debugging
How to analyse the system in case of unexpected behavior.
Subtopics:
coredumpkdump
Partial solutions or technical approaches for use cases commonly seen in embedded Linux products.
Yocto
How the package-based image build system works and when to choose it.
Resources:
Buildroot
How the distribution build system works and when to choose it.
Package Managers
How to manage static application versioning and dependencies on the device.
Subtopics:
dpkg/aptipkg/opkgdnf/rpm
Update Clients
How to handle updates on the device.
Subtopics:
raucswupdate
Resources:
- Introductions here, here or here
- RAUC documentation
- SWUpdate documentation
Incremental Update Support
How to reduce update transmission bandwidth.
Subtopics:
casyncOSTree
Read-only Rootfs
How to reduce the number of writes to the rootfs and protect it against undesired manipulation.
Subtopics:
- systemd requirements
- pre-init & overlays
Resources:
TODO: Extend security section.