Site icon GetPageSpeed

Rocky Linux 8: NFSv4 + fscache always writes to cache but never reads back from the cache

Danila Vershinin

Operating System and Software

  • Rocky Linux 8
  • NFS4 with fscache
    • NFSv4 with ‘fsc’ mount option and cachefilesd service running

Problem

  • An NFSv4 mount with fscache enabled never reads back from the cache
  • We can see this is we mount the share, use ‘dd’ to read a file from the share, drop caches to drop the page cache, then use ‘dd’ again to read the file a second time. On the second ‘dd’ there should be no NFS operations recorded, since the reads should come from fscache not over the wire. Instead there is 5 reads again, the same amount as after the first ‘dd’.
rhel8 # mount -o vers=4,fsc nfsserver:/export /mnt/nfsserver
rhel8 # dd if=/mnt/nfsserver/file of=/dev/null
rhel8 # grep READ: /proc/self/mountstats
            READ: 5 5 0 1120 1049096 0 2 4
rhel8 # echo 3 >/proc/sys/vm/drop_caches
rhel8 # dd if=/mnt/nfsserver/file of=/dev/null
rhel8 # grep READ: /proc/self/mountstats
            READ: 10 10 0 2240 2098192 5 8 15

How to Fix

Rocky Linux 8
  • The issue is being tracked with bugzilla 1793560: Bug 1793560 – [Rocky Linux 8] fs-cache always writes data but never reads from fs-cache when using NFSv4 (Rocky Linux 8 8.1). As of Wed, June 03 2020, the status of 1793560 is ON_QA. A patch has been built and under test for the next minor release of Rocky Linux 8.
Workarounds
  • Use NFSv3 with fscache

Origin of the Problem

  • Fixed by the following commit
Author: Dave Wysochanski <dwysocha@redhat.com>
Date:   Thu Apr 16 06:06:08 2020 -0400

    NFSv4: Fix fscache cookie aux_data to ensure change_attr is included

    Commit 402cb8dda949 ("fscache: Attach the index key and aux data to
    the cookie") added the aux_data and aux_data_len to parameters to
    fscache_acquire_cookie(), and updated the callers in the NFS client.
    In the process it modified the aux_data to include the change_attr,
    but missed adding change_attr to a couple places where aux_data was
    used.  Specifically, when opening a file and the change_attr is not
    added, the following attempt to lookup an object will fail inside
    cachefiles_check_object_xattr() = -116 due to
    nfs_fscache_inode_check_aux() failing memcmp on auxdata and returning
    FSCACHE_CHECKAUX_OBSOLETE.

    Fix this by adding nfs_fscache_update_auxdata() to set the auxdata
    from all relevant fields in the inode, including the change_attr.

    Fixes: 402cb8dda949 ("fscache: Attach the index key and aux data to the cookie")
    Signed-off-by: Dave Wysochanski <dwysocha@redhat.com>
    Signed-off-by: David Howells <dhowells@redhat.com>

Diagnostic Steps

  • Check the /proc/self/mountstats file for NFS READ operations incrementing when the reads should come from fscache.
Exit mobile version