Rook - General information
Warning
Even if ipv4_enabled
and ipv6_enabled
,
only one IP family will be used for the Ceph daemons
whereas IPv4 is preferred over IPv6.
Todo
needs updates (differentiate between on OpenStack and on Bare Metal)
For the usage of RWX (ReadWriteMany) volumes, a distributed storage system is needed. Ceph is a distributed storage system which allows to supply normal block devices (ReadWriteOnce volumes or real disks) in differing formats via network. In particular, it can supply them as ReadWriteMany volumes via CephFS.
We configure the cluster to use Cinder CSI volumes as backing storage. The volumes are already replicated on our OpenStack’s Ceph level, so we set the pool size to 1 (= one replica only) in the Rook cluster.
Cinder volumes are numbered in rook (e.g.
cinder-2-ceph-data
). The numbering does not always correspond to the OSD number using that volume! This is important when debugging OSD issues and when removing OSDs and their storage especially.To access ceph tools, run:
$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
Rook stores information about the provisioning state of volumes in ConfigMap objects while the provisioning hasn’t completed yet. When removing a provisioning job half way through, it is important to also clean up the corresponding ConfigMap object, otherwise the operator will hang.
The ceph mons may use the
local-storage
StorageClass which is a fancy version ofhostPath
and has the advantage of “binding” a pod to a node.local-storage
works through a controller that presents disks (or bind-mounts of directories, as in our case) as PVs to K8s. The controller also attaches PVCs to these PVs.Note that it’s usually safe to delete a ceph mon and its PVC if at least one healthy mon remains. To be on the safe side, make sure the quorum is
> floor(mons / 2).