Constructing a Cloud

Jump to Examples

We have come a long way in storing and interpreting data. Storing digital data in DNA, compared to the floppy disk that was around when I was born, shows the huge loop humanity has made in the field of storing data.

Nevertheless, no data really hangs in the cloud. The data is stored in at least one machine that is connected to the network humanity has built. A huge amount of data travels through the air at any point in time (how much data is generated each day?), but all of it is stored somewhere, on a physical machine, with a physical owner. "The cloud" is a very successful name for "other people's computers, reachable over a network". Once you know that, building a small one yourself stops looking like magic.

A Bit of Cloud History

The idea is older than the name. In 1961 John McCarthy suggested that computing might one day be sold like a public utility, like water or electricity. That was the era of time-sharing: one expensive mainframe, many terminals, and users who paid for the time they used (CTSS was demonstrated the same year). In 1967 IBM's CP-40 and later VM/370 (1972) brought virtualization – one physical machine pretending to be many. That is still the core trick of every cloud today. Around the same time Licklider was already dreaming of a network of computers that would make such sharing global.

The personal computer in the 1980s moved the computing back to the desk, and the client–server model of the 1990s split the work between a small local machine and a bigger remote one. In telephone and network diagrams of that period, the part that somebody else manages was drawn as a cloud – which is probably where the symbol comes from. The phrase "cloud computing" is often traced to a 1996 internal document at Compaq, though it only became widespread in the mid-2000s. In the late 1990s, grid computing tried to connect many computers into one big resource, mostly for science – SETI@home used volunteers' idle PCs and DEISA linked European supercomputing centres. In 1999 Salesforce started delivering business software through a web browser, long before anyone said "SaaS".

The turning point was 2006, when Amazon launched S3 (storage, March) and EC2 (virtual machines, August) – you could rent a server by the hour with a credit card and an API call. Behind it was the maturing of open-source virtualization: Xen (2003) and KVM (in the Linux kernel since 2007). Google's App Engine (2008) and Microsoft's Azure (announced 2008, generally available 2010) followed. In 2010 OpenStack – started by Rackspace and NASA – made it possible to build a private cloud with open-source software, and in 2011 NIST published the definition that is still quoted today (see the next section).

The next shift was lighter packaging. The Raspberry Pi arrived in 2012 and made a real Linux server cost as little as a dinner. Docker (2013) made containers easy to use, Kubernetes (open-sourced in 2014 by Google) made it possible to run thousands of them, and AWS Lambda (2014) introduced "serverless" – you hand over only a function, not even a machine. The CNCF was founded in 2015 to look after this growing ecosystem. After that came a counter-movement: the GDPR (2018) put a legal weight on where personal data lives, Gaia-X started as a European data-sovereignty initiative, edge computing moved computation back toward the devices, and self-hosting became a hobby again.

A short timeline:

Some resources for history:

• Cloud computing (Wikipedia)
• Time-sharing (Wikipedia)
• Timeline of cloud computing (Wikipedia)
• NIST SP 800-145 – The NIST Definition of Cloud Computing
• Computer History Museum

What Makes Something a Cloud?

The NIST definition lists five essential characteristics: on-demand self-service (you get resources without calling someone), broad network access (you reach them over a network from many kinds of devices), resource pooling (many users share the same hardware), rapid elasticity (you can grow and shrink) and measured service (usage is metered). A Raspberry Pi on a shelf will not be elastic, but it can be self-service, reachable and pooled between the people in your household – and that is already plenty for a home cloud.

Service models and deployment models (the usual vocabulary):

Services and Protocols – What Is Possible and What Is Used

A cloud is not one program. It is a set of services, and each service speaks one or more protocols. The tables below are my own selection, not a complete list: each row says what you can do and what is actually used in practice (and, where it matters, what is better avoided). As in the Computer Networks article, the protocols are mostly open documents – the RFCs – which is why software from different authors can work together at all.

Reaching and securing the machine:

Storing and distributing data:

Ways data is being distributed between people and devices:

Running and watching the machines:

A few observations. First, almost everything is HTTP over TLS now, even things that used to have their own protocol; that is why a single reverse proxy can serve a dozen services. Second, the old plaintext protocols (Telnet, FTP, POP3 without TLS, HTTP for logins) are still supported by a lot of software "for compatibility" – it is up to you to turn them off. Third, a self-hosted cloud is mostly glue: DNS, certificates, identity and backup matter more than which file-sharing application you choose.

Some resources for services and protocols:

• RFC Editor
• IANA – Service Name and Transport Protocol Port Number Registry
• Awesome-Selfhosted – a large list of self-hostable software
• My Self-hosting & Home Server resources
• CNCF Cloud Native Landscape
• Home Assistant – Examples

Examples: Building a Small Cloud

The smallest useful cloud is one machine, one container runtime and one or two services. Take a Raspberry Pi 4 running Home Assistant, coupled with your smartphone, and you have a smart-home environment with a touch display; there are many examples for various elements. With a minimum of router management, you can display and manage these elements from anywhere in the world where there is internet. Soon also on other planets: NASA planned a 4G mobile network on the Moon (the article is from 2020; the plans have since shifted in time). Below are three variations on the same idea. The commands are a starting point, not a tested recipe for your exact hardware – always check the current documentation of the distribution and of each project before you run anything.

Debian family or Red Hat family? (my own judgement, not test results):

Example 1 – Raspberry Pi with Raspberry Pi OS / Debian: Home Assistant, Nextcloud and remote access

This is the "home cloud" most people start with: smart-home control and private file storage on one small machine.

1. Prepare the storage. Use an SSD over USB (or NVMe on a Pi 5 with a suitable board) for the data. SD cards wear out quickly under constant writes from databases. If you must use an SD card, buy a good one and keep the data elsewhere.
2. Write the system image with the Raspberry Pi Imager: choose the 64-bit Lite version (no desktop is needed), and in the advanced options set the host name, a user, SSH with your public key, and the Wi-Fi or leave it on Ethernet (strongly preferred for a server).
3. Boot, connect and update:

   ssh youruser@cloudpi.local
   sudo apt update
   sudo apt full-upgrade
   sudo apt install unattended-upgrades
   sudo reboot

4. Give it a fixed address. Either a DHCP reservation in the router (easiest, and documented in the Computer Networks article) or a static configuration on the Pi. Everything else – DNS names, port forwarding, firewall rules – depends on a stable address.
5. Install a container runtime. Follow the official Docker instructions for Debian (the repository changes over time, so copy from the current page rather than from this article), or install Podman from the distribution packages. Add your user to the docker group only if you understand that it is effectively root access.
6. Home Assistant as a container. Create a directory and a file compose.yaml:

   services:
     homeassistant:
       image: ghcr.io/home-assistant/home-assistant:stable
       container_name: homeassistant
       restart: unless-stopped
       network_mode: host        # needed for device discovery on the LAN
       volumes:
         - ./config:/config
         - /etc/localtime:/etc/localtime:ro

   Start it with docker compose up -d and open http://cloudpi.local:8123. If you want add-ons and a managed system instead, use the dedicated Home Assistant OS image – but then the Pi is a dedicated appliance, not a general server.
7. Nextcloud with a proper database, in a second Compose project:

   services:
     db:
       image: postgres:16
       restart: unless-stopped
       environment:
         POSTGRES_DB: nextcloud
         POSTGRES_USER: nextcloud
         POSTGRES_PASSWORD: change-this-password
       volumes:
         - ./db:/var/lib/postgresql/data
   
     app:
       image: nextcloud:stable
       restart: unless-stopped
       depends_on:
         - db
       ports:
         - "8080:80"
       environment:
         POSTGRES_HOST: db
         POSTGRES_DB: nextcloud
         POSTGRES_USER: nextcloud
         POSTGRES_PASSWORD: change-this-password
       volumes:
         - ./nextcloud:/var/www/html

   After docker compose up -d open http://cloudpi.local:8080 and create the admin account. Put it behind a reverse proxy with HTTPS before using it from outside the house (see the services table above).
8. Firewall. A simple start with ufw:

   sudo apt install ufw
   sudo ufw default deny incoming
   sudo ufw allow 22/tcp
   sudo ufw allow 8080/tcp
   sudo ufw allow 8123/tcp
   sudo ufw enable

   A well-known trap: ports published by Docker are added to the firewall by Docker itself and can bypass ufw rules. Check what is really reachable with a scan from another computer, or publish ports only on the local address (127.0.0.1:8080:80) behind a reverse proxy.
9. Reach it from outside – with WireGuard instead of open ports. Install WireGuard (sudo apt install wireguard), generate keys with wg genkey and wg pubkey, and create /etc/wireguard/wg0.conf:

   [Interface]
   Address = 10.8.0.1/24
   ListenPort = 51820
   PrivateKey = SERVER_PRIVATE_KEY
   
   [Peer]
   # your smartphone
   PublicKey = PHONE_PUBLIC_KEY
   AllowedIPs = 10.8.0.2/32

   Enable it with sudo systemctl enable --now wg-quick@wg0, forward one UDP port (51820) on the router to the Pi, and configure the phone with the mirror image of this file. When connected, the phone can open Home Assistant and Nextcloud as if it were at home. If your home address changes, add a dynamic DNS name.
10. Back it up and test the restore. Use restic or BorgBackup to copy the config, db and nextcloud directories to another disk and to an encrypted remote location. A backup that was never restored is a hope, not a backup.

Example 2 – Fedora / AlmaLinux / Rocky / RHEL: Podman, Quadlet and Cockpit

The same idea in the Red Hat way of doing things: containers managed by Podman as ordinary systemd services (with Quadlet), a firewall with zones (firewalld), SELinux left enabled, and a web console. This is also what you would meet on a company server, so it is good practice.

1. Install the system. On a Raspberry Pi start with the image the distribution offers for your exact Pi model (Fedora Server, or the Raspberry Pi images of AlmaLinux / Rocky) and check the hardware notes; on an old PC use the normal installer from a USB stick (see the Reviving an Old Computer article). Create a user with SSH key access.
2. Update and install the tools:

   sudo dnf upgrade --refresh
   sudo dnf install podman cockpit firewalld
   sudo systemctl enable --now firewalld cockpit.socket
   sudo dnf install dnf-automatic     # optional: automatic updates (configure it in /etc/dnf/automatic.conf)

   The Cockpit web console is then on https://hostname:9090 – services, logs, storage, updates and containers in a browser.
3. Open only what is needed:

   sudo firewall-cmd --permanent --add-service=cockpit
   sudo firewall-cmd --permanent --add-port=8080/tcp
   sudo firewall-cmd --reload
   sudo firewall-cmd --list-all

4. Describe a container as a systemd unit. Create /etc/containers/systemd/nextcloud.container:

   [Unit]
   Description=Nextcloud
   After=network-online.target
   
   [Container]
   Image=docker.io/library/nextcloud:stable
   PublishPort=8080:80
   Volume=nextcloud-data:/var/www/html
   
   [Service]
   Restart=always
   
   [Install]
   WantedBy=multi-user.target

   Then run sudo systemctl daemon-reload and sudo systemctl start nextcloud.service. Quadlet generates the service for you; logs are in journalctl -u nextcloud. For production use, add a PostgreSQL container in the same way instead of the default SQLite. If you prefer rootless containers (safer), place the file in ~/.config/containers/systemd/ and run loginctl enable-linger youruser so that it keeps running after you log out.
5. Leave SELinux on. When a container cannot read a directory you mounted, do not switch SELinux off; add the :Z option to that bind mount (named volumes are labelled automatically) and read sudo ausearch -m avc -ts recent to see what was blocked.
6. Optional extras from the same ecosystem: FreeIPA (LDAP, Kerberos and certificates in one), Cockpit machines for KVM virtual machines, Ansible to repeat the whole setup on the next machine.

Example 3 – Several small machines as a cluster with k3s

Two or three Raspberry Pis (or old PCs) with the same operating system, connected to the same switch, can become a small Kubernetes cluster using k3s, a lightweight distribution that runs well on modest hardware. It is not necessary for a home, but it is the best way to learn how the "big" clouds schedule work, replace a failed node and keep services alive.

1. Prepare the nodes: the same Linux on all of them, fixed addresses, host names (node1, node2, node3), synchronised time, and 64-bit systems. Read the k3s requirements for your distribution; on the Red Hat family the SELinux policy package for k3s is needed, and on some Raspberry Pi systems a kernel setting for memory control groups.
2. Install the server node. The installation script is the documented way, but it is always wise to download it and read it before you run it:

   curl -sfL https://get.k3s.io -o k3s-install.sh
   less k3s-install.sh
   sudo sh k3s-install.sh
   sudo k3s kubectl get nodes

3. Join the other nodes with the token from the server (sudo cat /var/lib/rancher/k3s/server/node-token):

   sudo K3S_URL=https://node1:6443 K3S_TOKEN=THE_TOKEN sh k3s-install.sh

4. Deploy something – for example Nextcloud or a simple web server with a Helm chart or a YAML manifest – and watch it move to another node when you unplug the first one. Persistent storage across nodes is the difficult part: look at Longhorn, NFS from a separate machine, or keep stateful services pinned to one node.
5. Remember the single point of failure. A cluster with one server node still has a single brain, and a cluster on one power strip and one switch is not highly available. Real redundancy needs several server nodes, separate power and a documented recovery procedure.

There is also a variation for those who prefer virtual machines: install Debian or a Red Hat-family system on an old PC, enable KVM with libvirt (managed from the Cockpit machines page), or install Proxmox VE directly. You then create small virtual servers for DNS, mail, files and so on and snapshot them before every change. The principle is the same as in 1972 on the IBM mainframe, only the machine is quieter.

Keeping the Cloud Healthy

• Expose as little as possible. One VPN port and perhaps one HTTPS reverse proxy are better than ten open admin panels. Never put SSH with password login, databases or the Docker socket on the Internet.
• Update regularly. unattended-upgrades on Debian, dnf-automatic on the Red Hat family, and a routine for updating container images (pin versions, read the release notes, then update). Old images are the usual way a home cloud gets compromised.
• Encrypt and authenticate. TLS everywhere, SSH keys, strong unique passwords with a password manager, two-factor authentication where the service supports it. Encrypt disks of portable or remotely hosted machines (LUKS).
• Follow the 3-2-1 rule for backups: three copies, on two different kinds of media, one of them in another place. Test the restore at least once.
• Know the law. If you store other people's personal data (family, clients, colleagues), the GDPR may apply to you too; see the Security Considerations article.
• Watch power and heat. A Raspberry Pi needs a proper power supply and some cooling, and a power cut can corrupt a disk – a small UPS or at least a clean shutdown routine helps.

A Few More Thoughts

A cloud is a very practical idea: share expensive machines, pay for what you use, reach your data from anywhere. It is also a trade-off between convenience and control, and the two sides are not the same for everyone. Renting a cloud means that someone else handles the hardware, the updates and the availability – and that you accept their prices, their rules and their location. Building your own gives you control and knowledge, but you become the administrator, the security team and the person who restores the backup on a Sunday evening.

For most people the sensible answer is somewhere in the middle: let a commercial cloud do what it does best, and keep at home what is personal, small or simply fun to run. A Raspberry Pi will never be a data centre, but after you have built a small cloud on it, the big ones stop being mysterious. They are the same ideas – virtualization, containers, protocols, backups, updates – only repeated a few million times.