Skip to content

NFS export policy is missing one k8s node's IP #965

@ptrkmkslv

Description

@ptrkmkslv

We faced the bug when one of the cluster nodes is unable to mount PVC (NFS) with error: 



71s Warning FailedMount pod/grafana-5b7b4f4dc7-9r4zf MountVolume.SetUp failed for volume "pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6" : rpc error: code = Internal desc = error mounting NFS volume x.x.x.x/trident_pvc_4c0caf5e_ac71_4dc2_a9d1_329913b244a6 on mountpoint /opt/rke/var/lib/kubelet/pods/cd1b1ad1-f380-4fb2-9e9b-eff4806121b4/volumes/kubernetes.io~csi/pvc-4c0caf5e-ac71-4dc2-a9d1-329913b244a6/mount: exit status 32

After investigation we have discovered that NFS export policy on SVM is missing this node’s IP (policy had 16 entries where cluster consists of 17 nodes).

trident-node / trident-controller did not produce any useful error messages regarding ‘publishing’ volume to the node.
SVM also did not complain about any problem.

Issue was manually resolved by storage team - missing node was manually added to export policy - after that POD was immediately able to mount PVC

Environment
kind: tridentbackendconfigs.trident.netapp.io for NFS share is using both parameters: autoExportPolicy: true
and
autoExportCIDRs: with subnet class /24 where k8s storage interfaces are

  • Trident version: v24.10
  • Kubernetes version: v1.30.6
  • Container runtime: docker://26.1.0
  • Kubernetes orchestrator: Rancher (custom cluster)
  • OS: Flatcar Container Linux by Kinvolk 4081.2.0
  • NetApp backend types: ONTAP AFF (ONTAP 9.12.1P12)

Expected behavior

Complete export list of all k8s worker nodes

Any advice ? what to do if the problem occurs again ? (any tshooting commands that can be used ?)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions