1. Historical context

1945 - 1955

You directly loaded instructions into memory and let it execute your “code” (processor instructions). Nothing else ran on the machine.

This process was labor intensive: a qualified operator loaded your program, dumped the memory contents, remove any external media, reset the machine and load the next job.

1956 - 1959

Computers became faster, but a lot of time was spend on managing jobs.

resident monitors: very small programs, which always resided in memory, and monitored what the state of the current job was.
Jobs were loaded in series (batches). When the current job finished, memory would be dumped and the next one would automatically be loaded and started.

1960s

Peripherals (tapes, punch cards, …) were extremely slow.
Multiprogramming: the current job is waiting for a peripheral, another job would be started.

Travel reservation system by American Airlines: where travel agents would search, price and book services. A computer system now had to support:

• User accounts with concurrent access
• Fast response time and ease of use
• Storage organization, also following the rise of disks
• Data communication links (often phone lines)

More and more business started using computers, thanks to minicomputers, so demand for OS software increased.

1964 - 1969

IBM’s System/360 line of computers, each with expansion capabilities, backwards compatibility, all under one instruction set and operating system.

Multics, an influential operating system that was designed for a General Electric mainframe. Some of it’s novel ideas include, but are not limited to:

• Time-sharing: sharing resources of a singular machine among many users (via multiprogramming and multi-tasking)
• Single level store (aka virtual memory): Processes would write into contiguous memory, but in reality it is scattered and managed by the operating system.
• Protection rings: Allowing different levels of access to different resources (programs). In general, this also often requires a CPU protected mode.
• Hierarchical file system: Arbitrary directories with subdirectories, rather than just having files next to each other or having directories which could only contain files.
• Command processor (that is separated from the OS): More on that later

1969: Death of Multics and birth of UNIX

Ultimately Multics grew too large, becoming unusable and unmaintainable.
In the beginning, programming was done mostly by Bell Labs, with Ken Thompson being one of the developers.

Ken Thompson image by National Inventors Hall of Fame
Ken Thompson and Dennis Ritchie in 1973

Thompson still had some desire to work on operating systems, after Bell Labs pulled out. The tools he made while rewriting his own video game on a PDP-7 lead to a whole operating system: UNIX.

Based, expanded on and improved upon many of Multic’s ideas, UNIX became the father of modern OSs.

2. Structure of UNIX-likes

Ignoring standards and specifications, in my opinion, from the point of a user, a UNIX-like OS has the following components:

1. shell: A programmable replaceable command-line interpreter, with utilities for managing the whole system and support for pipelines and I/O redirection
2. UNIX-style file system: Filesystem as a single rooted tree, objects (nodes) in the file system are inodes and an inode can be (at least) either a regular file, directory or devices. Permissions per users and group.
3. kernel: Program that manages all communication and operations between the hardware and software

Command-line shells

Command-line shells operate solely with text, often only with ASCII characters. Usually connecting to a computer (to the shell) is said to be done with either a “terminal” (“console”) or “teleprinter” (“teletype”).

Video displays became widely available in the late 1970s, so before that access was done with a “teleprinter”, a typewriter-printer combo.

Teletype Model 33 by Arnold Reinhold

With the advent of computer displays, teleprinters were replaced with terminals, video display-keyboard combo. Thanks to the widely popular VT100, almost all terminals support ANSI escape codes.

DEC VT100 by Jason Scott

Today we don’t usually use such specialised hardware, but the names persist with slightly different meanings.

Virtual Terminal (Console, tty): on some modern UNIX-likes (like Linux), the kernel/OS provides special devices, where a console is directly implemented/simulated with the connected computer and display.

• systems that don’t provide it usually expose another way to connect a console (now external hardware and/or software is required), like a serial port

Terminal Emulator: a special program that emulates a virtual terminal (within another display system)

Pseudoterminal (pty): a program that sits between a terminal and a shell (or other program), and makes them both think they’re directly connected

• ssh, a program to remotely connect to a device’s shell, establishes a pseudoterminal. Your local terminal thinks it’s connected to a local shell and vice versa with the remote shell, however the pty in between handles the true communication.

2.2. UNIX-style file system

• Rooted singular tree of directories (the root folder of which is called “root”, denoted with a “/”)
• Each tree object (node) is an i(ndex)node, containing type, ownership, access, internal data and any other metadata, but not name!
• Directory inodes (files) contain a table with rows of names and inode numbers, mapping a file name to it’s contents. Each row is called a hard link. Every directory has a hard link to itself, called “.”, and a hard link to it’s parent, called “..”.

• FIFO special: aka named pipe, a special type of regular file, where at any moment only one process can read from it and one can write to it. It takes the concept of shell pipes to inter-process communication.

  [Process]     --write->     [FIFO]     x-write--     [Process]
  [   A   ]     x--read--     [    ]     --read-->     [   B   ]
  

• block special: a device which is randomly accessible, like a hard drive or cdrom drive. Think of a C++ array (pseudocode):

  BlockSpecial[438] = 1 ; var = BlockSpecial[438]
  

• character special: devices which are accessed via serial streams of input or output, like keyboards, mice, graphics cards and teletypes.
  Think of a C++ stream (pseudocode):

  CharacterSpecial << 1 ; CharacterSpecial >> var
  

Side-step into users and groups

A user in UNIX is essentially a small collection of data, most important of it being a unique ID (number), a name (string), a group id (number; files created by the user are in that group) and a password.

Groups are also small collections of data, but much simpler, comprised only of a unique ID (number), a name (string) and a list of users that are “in” the group. Their main purpose is to simplify access control.

User with ID 0 is called root, it is the “system administrator”, all actions made by the system itself are done as that user. Every user has their own “home” folder (under “/home/USERNAME/”, except for root, which is “/root/”), in which they store personal files, as well as user-specific configuration files.

• file permissions: control the freedom of a user to read (navigate for directories), change and execute files
  File permissions are split into three sets: what the owner can do, what group members can do and what anyone else can do.
  Each set contains a mix of three permissions: read, write and execute, and each set stores some combination of them.

  Each is represented by a bit in the order above (101 means you can read, cannot write, can execute) and the overall permissions are often shown as a three digit number or character string.
  Example: 111101100 will be shown as 754 or rwxr-xr--

• link count: the count of hard links that point to the inode. A file is deleted only when there are no hard links left to it. Every directory inode has at least two - one from parent and one from itself (“.”), though the latter isn’t counted. Child and parent directories also add to the count.

• User ID of owner, Group ID, file size and timestamps

2.3. Kernel

The kernel has the core functionality of the operating system and bridges the gap between programs and hardware. Some important subsystems include scheduling, file, device, process and memory management.

To preserve the everything is a file methodology, devices and processes can be handled as files (inside /dev and /proc). They (most of the time, in modern kernels) aren’t actual files on a hard drive, but “virtual” files, where operations on them are handled in a different manner than normal by the kernel.

Two of the important things a kernel does is manage processes and memory.

3. Linux from the inside

Enough theory, time to have some fun and learn Linux!

We’ll be using Linux Minimal Live (just a bootable ISO) because it’s:

• very small, comprised of only the Linux kernel, GNU C library and Busybox
  
• made by a fellow Bulgarian

We’re going to explore and look around the following stuff:

• Shells: What a modern shell would often support on a Linux based OS
• Files and directories: Naming conventions and commands to work with them
• Common configuration files: Relevant configuration files, their purpose and syntax
• System management: Other commands with which to modify your system

3.1. Shells

Variables and data types

Before talking about the different features newer shells support, we’ll roughly cover variables and data types (mostly bash specifics).

A variable, also called parameter, is created with the syntax (NO spaces between the =):

name=value

Where name is either a combination of letters (upper and lower case), numbers and the underscore character, which cannot begin with a number, or one of select few positional and special parameters.

Mixing types and operations

name=value

Parameter expansion

With parameter expansion, actions are done with the general variable (entity) or on strings and arrays values. Operations on integers are done with arithmetic expansion.

Parameter expansion is started with the character $ and either a name or curly braces. Retrieving a value is done with the forms $name, ${name} and ${name[index]} for arrays.

Some parameter expansions in bash:

• ${#parameter} length of characters
• ${parameter:offset} and ${parameter:offset:length} return substring from index offset
• ${parameter/pattern/string} return a string, where every match on pattern is replaced by string
• ${name[@]} a (space separated) sequence of indices from 0 to length (for defined values) of array name

Arithmetic expansion

Arithmetic expansions allow evaluation of arithmetic expressions. They’re in the form $(( expression )).

Some useful expressions are:

• ++name, name++, --name, name++ postfix and prefix incrementation and decrementation
• +, -, *, /, % normal arithmetic operators, ** exponentiation
• ~, <<, >>, &, ^, | bitwise operators
• ==, !=, >, >=, <, <= logical comparison
• !, &&, || logical not, and, or
• expr ? exprTrue : exprFalse conditional
• =, +=, …, <<=, … assignment operators
• expr1 , expr2 consecutive expressions and (expr) bracing

Expressions work with both variable names and integers.

• #: Everything between a # and the end of the line denotes a comment

• compound commands: Structures for flow control
  In all examples, test-command is any command (or combination with control operators), and logic is based on exit code.

  • conditional

  if test-command ; then commands-if-true
  elif test-command ; then commands-if-true
  else commands-if-true
  fi
  

  elif statements can appear multiple times, both elif and else are optional

•  

  case word in (pattern | ...) commands-if-true ;; ... esac
  

  Where the beginning brace can be ommited.
  Patterns are according to bash’s pattern matching

  • loops

  while test-command; do commands-if-true; done
  

  for name in words...; do commands-if-true; done
  

  If in words... is ommited, positional arguments are used

Everyday file system commands

• mkdir: create directories
• rm: remove a (hardlink to) file or directory
• mv: move file (or directory) to another directory
• cp: copy a file
• ln: create hard (or soft, with -s) links to files (inodes)
• ls: list directory contents

Other useful file system commands

• shred: overwrite file contents
• dd: do many things with files, like modifying and duplicating data across devices and partitions, (basic) data recovery and generating files
• du: estimate file usage
• stat: display status for file or file system
• chwon: change owner of an inode
• chgrp: change group of an inode
• chmod: change file modes
• find: search for a file inside a directory (recursively)

Everyday text (file) commands

• echo: prints a given string to standard output
• cat: sequentially prints files to standard output
• less: view files with support for scrolling
• head: print the first lines (or characters) of a file
• tail: print the last lines (or characters) of a file

Other useful text (file) commands

• grep: searches for matches in a file, using simplified regular expressions. egrep for extended regex and fgrep for string match (as-is).
• sed: filter and transform text from files, according to regex
• tr: translate, map (more than one to one for example) and delete characters
• cut: remove parts of text from each line
• strings: find embedded text inside binary files
• truncate: shrink or extend the size of a file
• wc: count words inside file
• diff: compare files and show their differences

User and group management

• useradd: creates a new user (with a password)
• userdel: remove a user and all of his files
• usermod: modify a user account, including his name and groups
• passwd: change password of a user. With -l user could be locked (impossible to log in)

• groupadd: create a new group
• groupdel: remove a group
• groupmod: modify an existing group

• w: show who is logged in and what they are doing
• who: show who is logged in
• whomai: print username
• login: establish a new connection to the system

• /etc: system configuration files

  • /etc/group: the data for all groups is stored here, in the format

  GROUPNAME:PASSWORD:GID:USERS...
  

  • /etc/passwd: the data for all users is stored here, in the format

  USERNAME:PASSWORD:UID:GID:USERINFO:HOMEDIR:SHELL
  

  • /etc/shadow: stores other user data, especially the password as an encrypted string, in the format

  USERNAME:PASSWORD:LASTCHANGED:MINBETWEENCHANGES:MAXBETWEENCHANGES:WARNEXPIRE:INACTIVEDAYS:EXPIRE
  

4. Software licensing

It is important to note that most software in Linux, including the kernel itself, is under a variety of open licenses. An open license is a license which allows others to reuse the original work, under some restrictions.

For software, such a license is applied to the source code from which the original application was made. Some commonly used ones, with a (NON-LEGALLY BINDING) summary include:

• MIT: the author is not responsible for anything that the code does
• BSD-3-Clause: use the same license (copyleft), the author is not responsible for anything that the code does
• GNU GPL: disclose the original project and use the same license (copyleft)

This is important, since pretty much everything can be freely modified by anyone (for the better or worse, but generally better).

5.1. Bootloader

The computer boot process is a relay race.

In a “standard” BIOS-MBR boot configuration:

BIOS -> Master Boot Record -> Active Parition -> Bootloader ->
-> Boot menu (optional) -> Kernel -> Everything else

• BIOS: a small chip with code that is ran upon power up
• MBR: small amount of space at the very start of the disk, that contains some (bootstrap) code and partition information. The BIOS hands off execution to the bootstrap code.
• Active partition: MBR loads and runs code from the beginning of the partition (boot sector), which starts the bootloader
• Bootloader: the first complex program, which manages kernels, shows a menu, allows for debugging and so on

5.3. Package manager

Programs are just executable files, usually placed in a specific folder, like /usr/bin.

• my Debian server has ~1000 installed packages
• my current desktop Gentoo setup has ~1500 installed packages
• my old Fedora setups have reached ~3000 packages on multiple occasions

Updating all of them means replacing all of the binaries, means tracking where each one came from, installing a new one might require installing multiple others and so on.

On modern systems, managing binaries is automated by an application called a package manager.

5.4. Window systems

A widow system is your graphical shell: the graphical way to interact with your computer (the kernel). It’s main components are:

• Window manager: the overall system with which you create, remove, resize and work with windows. They are generally separated into two types:
  • Stacked: this is what you know, windows are just rectangles floating about, and can be put one over each other
  • Tiled: windows take up a predefined (but changeable) portion of the screen
• Compositor: adds an unseen window buffer, allowing additional processing, before a window is shown. This powers transparency, animation effects and more.
• Session manager: graphical application that manages the window managers (outside WMs), think of it as a login window