NFS Cache Cleaner

When the cache fills, some space needs to be freed to permit caching of more recently-accessed files to continue (instead of just turning caching off). Our implementation presupposes that filling the disk is reasonably rare, and that the policy for determining which cache files to remove should be left up to the local system administrator.

Thus, we have developed a simple interface for a user-level ``cleaner'' program that the kernel awakens when the cache fills. This process searches the cache directories, decides which files to remove, removes them, and tells the kernel how much space has been freed. The exact sequence of events is:

1.

The kernel realizes that it has run out of space. It does this by tracking of the number of cache pages being used per mounted filesystem. On each write to the cache, the client checks for either of the following cases:

(a)

the number of cached pages for this remote partition exceeds the disk_cache_max_size parameter specified when mounting; or

(b)

the number of free pages on the cache filesystem (i.e., the local disk partition) is less than the minimum free space permitted (the maximum of the disk_cache_min_free parameters for all of the mounted filesystems which are sharing this cache directory).

2.

The kernel writes a record of the violation to the file <cache_root>/.FULL. The violation is specified as a single line of information that the user-level cleaning process will need to do its job:

(a)

In the case of maximum space exceeded, the current time and the mounted filesystem (a server name and server device) are written to .FULL, and the group execute bit (g+x) is set on .FULL.

(b)

In the case of a minimum free violation, only the current time is written (a minimum free violation is viewed as being shared by all filesystems in the cache), and the owner execute bit (o+x) is set on .FULL. Setting the owner execute bit disables further caching, thus avoiding an out-of-disk-space error.

3.

A user-level cache cleaning process notices the change in the .FULL file (it should normally block in a loop checking the file size periodically--alternatively, the kernel could send the cleaner process a signal). It then reads the list of violations (there may be more than one by the time the cleaner has awakened) and handles the full partition(s). Our sample cache cleaning daemon, nccd, operates as follows:

(a)

candidates for removal are identified:

• when the maximum space for a remote partition is exceeded, only fully-cached files from that partition are considered for removal; or
• when the minimum free space for a local disk is violated, all fully cached files in the cache directory are considered equally.

(b)

This set of candidates is then sorted by access time and the oldest 10% (by size) are removed. The special kernel-generated pseudo-file /proc/net/nfs-mounts provides information about the space consumption of the various mounted partitions so that the cleaning daemon can determine how much space to free.

Note that only files that are completely cached are considered for removal. This constraint simplifies freeing space because fully cached files have no special kernel data structures associated with them. As a result, the user-level cleaning daemon can safely remove these files itself, rather than having to request that the kernel do this on its behalf. The limitation of not permitting removal of partially-cached files is not a problem in practice for two reasons: 1) the size of the cache is substantially greater than the average size of the files in it; and 2) the fill-in thread (see section , on page ) is working to keep most of the cached files fully cached.

4.

After the cleaning daemon removes the chosen files, it must inform the kernel of the results of its actions. For each mounted filesystem from which files were removed, the cleaner must write a single line record to the pseudo-device /dev/nfs-cache-space with the filesystem identifier (hostname and device number pair) and the number of pages freed. This step ensures that the kernel is able to maintain accurate information about how much disk space the cache consumes without requiring it to ever scan the entire directory for usage information.¹⁵

5.

Finally, the cleaning daemon should clear the execute bits on .FULL to re-enable caching.

By specifying a clean, simple interface between the kernel and a user-level process, we permit sophisticated custom cache cleaning policies to be used. At the same time, this design decision dramatically simplifies the in-kernel code at little performance cost.