We faced the bug when one of the cluster nodes is unable to mount PVC (NFS) with error:
71s Warning FailedMount pod/grafana-5b7b4f4dc7-9r4zf MountVolume.SetUp failed for volume "pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6" : rpc error: code = Internal desc = error mounting NFS volume x.x.x.x/trident_pvc_4c0caf5e_ac71_4dc2_a9d1_329913b244a6 on mountpoint /opt/rke/var/lib/kubelet/pods/cd1b1ad1-f380-4fb2-9e9b-eff4806121b4/volumes/kubernetes.io~csi/pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6/mount: exit status 32
After investigation we have discovered that NFS export policy on SVM is missing this node’s IP (policy had 16 entries where cluster consists of 17 nodes).
trident-node / trident-controller did not produce any useful error messages regarding ‘publishing’ volume to the node.
SVM also did not complain about any problem.
Issue was manually resolved by storage team - missing node was manually added to export policy - after that POD was immediately able to mount PVC
Environment
kind: tridentbackendconfigs.trident.netapp.io for NFS share is using both parameters: autoExportPolicy: true
and
autoExportCIDRs: with subnet class /24 where k8s storage interfaces are
- Trident version: v24.10
- Kubernetes version: v1.30.6
- Container runtime: docker://26.1.0
- Kubernetes orchestrator: Rancher (custom cluster)
- OS: Flatcar Container Linux by Kinvolk 4081.2.0
- NetApp backend types: ONTAP AFF (ONTAP 9.12.1P12)
Expected behavior
Complete export list of all k8s worker nodes
Any advice ? what to do if the problem occurs again ? (any tshooting commands that can be used ?)
We faced the bug when one of the cluster nodes is unable to mount PVC (NFS) with error:
71s Warning FailedMount pod/grafana-5b7b4f4dc7-9r4zf MountVolume.SetUp failed for volume "pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6" : rpc error: code = Internal desc = error mounting NFS volume x.x.x.x/trident_pvc_4c0caf5e_ac71_4dc2_a9d1_329913b244a6 on mountpoint /opt/rke/var/lib/kubelet/pods/cd1b1ad1-f380-4fb2-9e9b-eff4806121b4/volumes/kubernetes.io~csi/pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6/mount: exit status 32After investigation we have discovered that NFS export policy on SVM is missing this node’s IP (policy had 16 entries where cluster consists of 17 nodes).
trident-node / trident-controller did not produce any useful error messages regarding ‘publishing’ volume to the node.
SVM also did not complain about any problem.
Issue was manually resolved by storage team - missing node was manually added to export policy - after that POD was immediately able to mount PVC
Environment
kind: tridentbackendconfigs.trident.netapp.io for NFS share is using both parameters: autoExportPolicy: true
and
autoExportCIDRs: with subnet class /24 where k8s storage interfaces are
Expected behavior
Complete export list of all k8s worker nodes
Any advice ? what to do if the problem occurs again ? (any tshooting commands that can be used ?)