Mnemonics
These cover the material where ordering or an exact set is the thing that fails under pressure — not comprehension. Each one is collapsed by default: try to produce the sequence before you open it.
Troubleshooting 30%
Pod is Pending — the four causes
CTAP — Capacity, Taints, Affinity, PVC
Show the sequence
| C | Capacity: 'Insufficient cpu/memory' — requests exceed what's allocatable |
|---|---|
| T | Taints: 'had untolerated taint' |
| A | Affinity: 'didn't match Pod's node affinity/selector' |
| P | PVC unbound, or no events at all → the scheduler itself is down |
Why: The scheduler writes the failed predicate verbatim into the FailedScheduling event, so describe answers this in one read. No events at all is the tell that nothing is scheduling anything.
Static Pods
File on disk, kubelet runs it, no scheduler, no kubectl delete.
Show the sequence
| Where | /etc/kubernetes/manifests/ (staticPodPath in /var/lib/kubelet/config.yaml) |
|---|---|
| Who | the kubelet, directly — the API server is not involved |
| Restart | there is none; edit the file and the kubelet recreates the Pod |
| Delete | move the file out of the directory; kubectl delete just recreates it |
Why: This is exactly why a dead control plane is repairable: kube-apiserver, etcd, scheduler and controller-manager are static Pods, so they can be fixed from disk with no working API server.
Cluster Architecture 25%
kubeadm upgrade order
DUPA-KRU — Drain, Unhold, Plan, Apply, Kubelet, Restart, Uncordon
Show the sequence
| D | Drain the node (--ignore-daemonsets) |
|---|---|
| U | Unhold + install the new kubeadm, then hold again |
| P | Plan — kubeadm upgrade plan |
| A | Apply — 'upgrade apply vX.Y.Z' on the FIRST control-plane node; 'upgrade node' on every other node |
| K | Kubelet + kubectl: unhold, install, hold |
| R | Restart — daemon-reload && systemctl restart kubelet |
| U | Uncordon the node |
Why: kubeadm is only an installer: the binary upgrade does nothing until 'upgrade apply' rewrites the static Pod manifests, and kubeadm never touches the kubelet package, which is why K and R are separate steps you do by hand.
etcd restore — the two edits
Restore, then change BOTH: the flag and the volume
Show the sequence
| 1 | etcdctl snapshot restore <file> --data-dir=/var/lib/etcd-restore |
|---|---|
| 2 | etcd.yaml: --data-dir=/var/lib/etcd-restore (the container's view) |
| 3 | etcd.yaml: volumes[etcd-data].hostPath.path=/var/lib/etcd-restore (the host's view) |
| 4 | Wait — the kubelet recreates the static Pod on file change. Restart nothing. |
Why: --data-dir is a path inside the container and the hostPath volume is what maps it to disk. Change one without the other and the restore appears to succeed while etcd serves the old data.
etcdctl's three mandatory TLS flags
CA, Cert, Key — Can't Connect Keyless
Show the sequence
| --cacert | /etc/kubernetes/pki/etcd/ca.crt |
|---|---|
| --cert | /etc/kubernetes/pki/etcd/server.crt |
| --key | /etc/kubernetes/pki/etcd/server.key |
| --endpoints | https://127.0.0.1:2379 |
Why: etcd speaks mutual TLS only. Omit any of the three and the rejection looks exactly like an outage. If you blank on the paths, grep them out of /etc/kubernetes/manifests/etcd.yaml.
Which RBAC pair?
Namespaced resource → Role. Cluster-scoped resource → ClusterRole. Reusable definition, one namespace → ClusterRole + RoleBinding.
Show the sequence
| Role + RoleBinding | namespaced resources, one namespace |
|---|---|
| ClusterRole + ClusterRoleBinding | cluster-scoped resources (nodes, PVs, namespaces, CRDs) or every namespace |
| ClusterRole + RoleBinding | define once, grant only inside one namespace |
| Role + ClusterRoleBinding | does not exist — invalid |
Why: The binding's kind decides the scope of the grant; the role's kind decides where the definition can live. That's why the third combination is legal and the fourth is not.
RBAC verbs
GLW-CUP-D-E — Get List Watch, Create Update Patch, Delete, and the odd ones
Show the sequence
| Read | get, list, watch |
|---|---|
| Write | create, update, patch |
| Remove | delete, deletecollection |
| Odd | * (all), plus non-resource URLs via nonResourceURLs |
Why: 'get' fetches one object by name and does NOT imply 'list'. A role with get but not list makes 'kubectl get pods' fail while 'kubectl get pod web' works — a classic partially-correct answer.
Why drain refuses
DEF — DaemonSets, EmptyDir, Free-standing pods
Show the sequence
| D | DaemonSet Pods can't be evicted → --ignore-daemonsets |
|---|---|
| E | emptyDir data would be destroyed → --delete-emptydir-data |
| F | Free-standing (uncontrolled) Pods won't come back → --force |
| + | Hangs forever on 'would violate the disruption budget' → a PodDisruptionBudget, not a bug |
Why: Each refusal names its own cause in the error text. The PDB case is the one that hangs instead of erroring, which is why it reads as a broken command.
Services and Networking 20%
NetworkPolicy: one dash changes the meaning
Two dashes = OR. One dash = AND.
Show the sequence
| OR | - podSelector: {...}\n- namespaceSelector: {...} → web Pods, or anything in that namespace |
|---|---|
| AND | - podSelector: {...}\n namespaceSelector: {...} → web Pods that are in that namespace |
| Deny-all | podSelector: {} selects every Pod; policyTypes with no matching rules denies that direction |
| Additive | multiple policies are UNIONed — there is no deny rule in NetworkPolicy |
Why: Entries in the 'from' list are separate peers (OR); selectors inside a single entry must all match the same peer (AND). Indentation is the only thing distinguishing them.
Cluster DNS names
service.namespace.svc.cluster.local — always four labels after the name
Show the sequence
| Service | <service>.<namespace>.svc.cluster.local |
|---|---|
| StatefulSet Pod | <pod>.<headless-service>.<namespace>.svc.cluster.local |
| Pod | <pod-ip-with-dashes>.<namespace>.pod.cluster.local |
Why: Test with the FQDN. Short names depend on the search list in the Pod's /etc/resolv.conf, so a short-name failure may mean 'wrong namespace', not 'DNS is broken'.
Workloads and Scheduling 15%
Taint effects
NoSchedule keeps them out. PreferNoSchedule asks nicely. NoExecute throws them out.
Show the sequence
| NoSchedule | no new Pods without a toleration; existing Pods stay |
|---|---|
| PreferNoSchedule | soft — the scheduler avoids the node if it can |
| NoExecute | as NoSchedule, and evicts running Pods that don't tolerate it |
Why: 'Make the existing Pods leave' is NoExecute. NoSchedule alone looks like it did nothing, because it only affects future scheduling.
The three probes
Liveness restarts. Readiness removes from endpoints. Startup buys time.
Show the sequence
| livenessProbe | fails → container restarted |
|---|---|
| readinessProbe | fails → Pod removed from Service endpoints, NOT restarted |
| startupProbe | disables the other two until it first succeeds |
Why: A failing readiness probe presents as a networking bug — the Service has no endpoints — because unready Pods are excluded from endpoints. Without a startupProbe, a slow-booting app is killed by its own liveness probe in a loop that looks like a crash.
Storage 10%
Why a PVC binds (or doesn't)
SAC — Size, Access modes, Class. All three, or nothing binds.
Show the sequence
| S | PV capacity >= PVC request |
|---|---|
| A | PV accessModes is a superset of the PVC's |
| C | storageClassName matches EXACTLY — including the empty string |
Why: storageClassName: "" means 'no class, static binding only' and is not the same as omitting the field, which means 'use the default class'. With no default class, omitting it leaves the PVC Pending forever.
Access modes
RWO is per NODE, not per Pod. That's why RWOP exists.
Show the sequence
| RWO | ReadWriteOnce — read-write by Pods on one node (several Pods on that node is fine) |
|---|---|
| ROX | ReadOnlyMany — read-only from many nodes |
| RWX | ReadWriteMany — read-write from many nodes; needs NFS/CephFS-class backing |
| RWOP | ReadWriteOncePod — exactly one Pod cluster-wide |
Why: The 'Once' in ReadWriteOnce counts nodes. Reading it as 'one Pod' is the most common wrong answer about storage.
Rendered from data/mnemonics.json, which is also what the PDF build reads — one
source, two outputs.