Kubernetes on OpenStack: Terraform, Kubespray & Cilium BGP

Scope: this is the layer that sits on top of the OpenStack cloud described in OpenStack Lab Deployment — how Kubernetes clusters get provisioned as OpenStack VMs with Terraform, installed with Kubespray, and wired into the same MikroTik router via Cilium’s BGP control plane (plus L2 announcements) instead of NodePort/NAT tricks. Real values from this deployment (ASNs, CIDRs, CRDs) are included throughout; credentials and private keys are intentionally omitted.

1. Architecture at a glance

Two Kubernetes clusters live on top of the same OpenStack cloud, each on its own provider VLAN (see the sibling article’s VLAN table): k8s project a and k8s project b. Both are provisioned the same way: Terraform creates the OpenStack VMs (Nova instances, ports on the cluster’s provider network, Cinder volumes), then Kubespray installs Kubernetes on top over SSH. Cilium is the CNI on both, in kube-proxy-replacement mode, with its BGP control plane peering directly with the MikroTik router — no external LoadBalancer, no NodePort/NAT — pod and Service IPs get real routes.

2. Provisioning the VMs with Terraform, then Kubespray

Each cluster has its own Terraform workspace (Nova instances sized from the flavor list, ports on that cluster’s provider VLAN, a Cinder boot volume per node) feeding a generated Ansible inventory that Kubespray installs against. The real post-install sequence, run once the nodes are up and Kubespray has finished:

bash config-kubectl.sh
bash script-cni.sh              # Cilium: helm install + BGP/LB-IP-pool CRDs (see §3-5)
bash script-ingress-haproxy.sh  # ingress-nginx/HAProxy ingress controller
bash script-cert-manager.sh     # see §7
bash script-dns-cm-autoscaler.sh
bash script-registry-secret.sh
bash script-csi-setup.sh        # Ceph RBD/CephFS CSI, backed by the same OpenStack-hosted Ceph

The ingress HAProxy here is a completely different deployment from the OSA-internal HAProxy in the sibling article — that one fronts the OpenStack control-plane APIs (Keystone, Nova, etc.); this one is a Kubernetes IngressClass controller fronting workloads running inside the cluster.

3. Cilium install

Both clusters run the same chart, same major settings, different only in their per-cluster ASN/CIDR:

helm upgrade --install cilium oci://quay.io/cilium/charts/cilium --version 1.19.1 \
  --namespace kube-system -f cni/cilium/cilium-values.yaml --wait --timeout=5m

kubectl apply -f cni/cilium/crds/bgp-cluster-config.yaml
kubectl apply -f cni/cilium/crds/bgp-peer-config.yaml
kubectl apply -f cni/cilium/crds/bgp-advertisement.yaml
kubectl apply -f cni/cilium/crds/loadbalancer-ip-pool.yaml

Non-default values on top of the chart’s defaults:

• kubeProxyReplacement: true, kube_proxy_remove: true — Cilium fully replaces kube-proxy
• autoDirectNodeRoutes: true, tunnel mode disabled — routing is direct, not VXLAN-encapsulated, since the underlying OpenStack provider VLAN already gives L2 reachability between nodes
• bgpControlPlane.enabled: true — the CRD-driven BGP control plane (§4-5), the actual mechanism used to reach the MikroTik router
• l2announcements.enabled: true on k8s project a’s Helm values (§6) — see that section for the honest caveat
• ipam.mode: kubernetes, enable-ipv4-masquerade: true with an explicit non-masquerade CIDR list (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) so traffic to any RFC1918 range — including the other cluster and the OpenStack tenant networks — isn’t SNAT’d

4. BGP: Cilium ↔ MikroTik peering

Same AS-numbering convention as the OpenStack article: router-side ASN = node-side ASN + 2. Cilium peers directly with the MikroTik’s per-cluster BGP instance from the sibling article’s §3 — no OpenStack-side BGP dragent/Neutron BGP speaker involved, this is a straight eBGP session from each node to the router IP on that cluster’s provider VLAN:

CiliumBGPClusterConfig (k8s project b shown; project a is identical with the ASNs/IP above swapped in):

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  bgpInstances:
  - name: instance-65025
    localASN: 65025
    localPort: 179
    peers:
    - name: 10.15.0.1
      peerASN: 65027       # the MikroTik
      peerAddress: 10.15.0.1
      peerConfigRef:
        name: cilium-peer

CiliumBGPPeerConfig — the timers and multihop setting are the same on both clusters:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: cilium-peer
spec:
  timers:
    connectRetryTimeSeconds: 5
    holdTimeSeconds: 15
    keepAliveTimeSeconds: 5
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 20
  ebgpMultihop: 5
  transport:
    peerPort: 179
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: bgp

CiliumBGPAdvertisement — pod CIDR and every LoadBalancer/ExternalIP Service address get advertised, tagged with a BGP community per ASN so the router-side filter chain (sibling article §3) can still enforce “only this cluster’s own prefixes”:

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisements
  labels:
    advertise: bgp
spec:
  advertisements:
  - advertisementType: PodCIDR
    attributes:
      communities:
        standard: ["65026:100"]
  - advertisementType: Service
    attributes:
      communities:
        standard: ["65025:100"]
    service:
      addresses:
      - LoadBalancerIP
      - ExternalIP        # only if you need manual static IP control
    selector:
      matchExpressions:
      - {key: somekey, operator: NotIn, values: ["never-used-value"]}   # empty selector = advertise all LB Services

ClusterIP Services are deliberately never advertised over BGP — pod-to-pod and pod-to-ClusterIP traffic stays on Cilium’s own eBPF datapath; advertising ClusterIPs externally would defeat the point of them being cluster-internal.

5. LoadBalancer IP pools

Each cluster gets its own CiliumLoadBalancerIPPool, carved out of that cluster’s own pod/service supernet (also the block accepted by the router-side filter in the sibling article):

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: default-pool
spec:
  allowFirstLastIPs: "No"
  blocks:
  - cidr: 10.16.128.0/18   # k8s project b; 10.13.128.0/18 on k8s project a

6. L2 announcements

L2 announcements and BGP solve different problems and are normally either/or per environment: L2 announcements make a Service’s external IP answer ARP/NDP on the local segment (useful when there’s no upstream router able to do BGP), while BGP gets the IP a real route into the network. Since the MikroTik here does speak BGP, BGP (§4) is the actual mechanism carrying traffic on both clusters.

Honest caveat: k8s project a’s Helm values do set l2announcements.enabled: true at the chart level (cilium_l2announcements: true in that cluster’s Kubespray inventory), but no CiliumL2AnnouncementPolicy custom resource exists anywhere in this repo for either cluster. Enabling the feature flag alone doesn’t announce anything — without a policy CR scoping which Services/interfaces to announce, this is a no-op today, effectively dormant. It’s flagged here as a real gap rather than glossed over: turning it into an active fallback (for example, restricted to a specific interface on the provider VLAN, only for Services carrying a particular label) is a follow-up, not something already running.

7. cert-manager + Route53 DNS-01

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.19.4 --namespace cert-manager --create-namespace \
  --set crds.enabled=true --wait --timeout=5m

DNS-01 via Route53 rather than HTTP-01 — one wildcard certificate covers every vhost in a single order, and it works for hosts that only resolve on the internal LAN:

# 1. Secret holding the AWS secret access key (never commit this value)
kubectl -n cert-manager create secret generic route53-credentials \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>'

# 2. ClusterIssuer referencing it — access key ID is not secret, secret key is
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: you@example.com
    privateKeySecretRef: { name: letsencrypt-dns-account-key }
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyID: <AWS_ACCESS_KEY_ID>
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key
---
# 3. One wildcard Certificate covers every host under the domain
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-wildcard
  namespace: haproxy-controller
spec:
  secretName: example-wildcard-tls
  issuerRef: { name: letsencrypt-dns, kind: ClusterIssuer }
  commonName: "*.example.com"
  dnsNames: ["example.com", "*.example.com"]

The wildcard secret is set as the ingress controller’s defaultTLSSecret, so every Ingress resource gets TLS with no per-host tls: block. The IAM user backing this only needs route53:ChangeResourceRecordSets/GetChange/ListResourceRecordSets scoped to the one hosted zone.

Summary of the full chain

1. OpenStack up and its provider VLANs routed via BGP (sibling article)
2. Terraform provisions the cluster’s VMs on that cluster’s own provider VLAN
3. Kubespray installs Kubernetes over SSH against the generated inventory
4. Cilium: kube-proxy replacement, BGP control plane peering to the MikroTik with the same router-AS-plus-2 convention (§3-4)
5. Per-cluster LoadBalancer IP pool, carved from that cluster’s own routed CIDR (§5)
6. L2 announcements flagged on but not yet backed by a policy — dormant, noted as a gap (§6)
7. Ingress HAProxy + cert-manager + Route53 DNS-01 (§7), Ceph CSI, and the cluster is ready for workloads