Update Linux to v5.4.2
Change-Id: Idf6911045d9d382da2cfe01b1edff026404ac8fd
diff --git a/Documentation/filesystems/00-INDEX b/Documentation/filesystems/00-INDEX
deleted file mode 100644
index 0937bad..0000000
--- a/Documentation/filesystems/00-INDEX
+++ /dev/null
@@ -1,153 +0,0 @@
-00-INDEX
- - this file (info on some of the filesystems supported by linux).
-Locking
- - info on locking rules as they pertain to Linux VFS.
-9p.txt
- - 9p (v9fs) is an implementation of the Plan 9 remote fs protocol.
-adfs.txt
- - info and mount options for the Acorn Advanced Disc Filing System.
-afs.txt
- - info and examples for the distributed AFS (Andrew File System) fs.
-affs.txt
- - info and mount options for the Amiga Fast File System.
-autofs-mount-control.txt
- - info on device control operations for autofs module.
-automount-support.txt
- - information about filesystem automount support.
-befs.txt
- - information about the BeOS filesystem for Linux.
-bfs.txt
- - info for the SCO UnixWare Boot Filesystem (BFS).
-btrfs.txt
- - info for the BTRFS filesystem.
-caching/
- - directory containing filesystem cache documentation.
-ceph.txt
- - info for the Ceph Distributed File System.
-cifs/
- - directory containing CIFS filesystem documentation and example code.
-coda.txt
- - description of the CODA filesystem.
-configfs/
- - directory containing configfs documentation and example code.
-cramfs.txt
- - info on the cram filesystem for small storage (ROMs etc).
-dax.txt
- - info on avoiding the page cache for files stored on CPU-addressable
- storage devices.
-debugfs.txt
- - info on the debugfs filesystem.
-devpts.txt
- - info on the devpts filesystem.
-directory-locking
- - info about the locking scheme used for directory operations.
-dlmfs.txt
- - info on the userspace interface to the OCFS2 DLM.
-dnotify.txt
- - info about directory notification in Linux.
-dnotify_test.c
- - example program for dnotify.
-ecryptfs.txt
- - docs on eCryptfs: stacked cryptographic filesystem for Linux.
-efivarfs.txt
- - info for the efivarfs filesystem.
-exofs.txt
- - info, usage, mount options, design about EXOFS.
-ext2.txt
- - info, mount options and specifications for the Ext2 filesystem.
-ext3.txt
- - info, mount options and specifications for the Ext3 filesystem.
-ext4.txt
- - info, mount options and specifications for the Ext4 filesystem.
-f2fs.txt
- - info and mount options for the F2FS filesystem.
-fiemap.txt
- - info on fiemap ioctl.
-files.txt
- - info on file management in the Linux kernel.
-fuse.txt
- - info on the Filesystem in User SpacE including mount options.
-gfs2-glocks.txt
- - info on the Global File System 2 - Glock internal locking rules.
-gfs2-uevents.txt
- - info on the Global File System 2 - uevents.
-gfs2.txt
- - info on the Global File System 2.
-hfs.txt
- - info on the Macintosh HFS Filesystem for Linux.
-hfsplus.txt
- - info on the Macintosh HFSPlus Filesystem for Linux.
-hpfs.txt
- - info and mount options for the OS/2 HPFS.
-inotify.txt
- - info on the powerful yet simple file change notification system.
-isofs.txt
- - info and mount options for the ISO 9660 (CDROM) filesystem.
-jfs.txt
- - info and mount options for the JFS filesystem.
-locks.txt
- - info on file locking implementations, flock() vs. fcntl(), etc.
-mandatory-locking.txt
- - info on the Linux implementation of Sys V mandatory file locking.
-nfs/
- - nfs-related documentation.
-nilfs2.txt
- - info and mount options for the NILFS2 filesystem.
-ntfs.txt
- - info and mount options for the NTFS filesystem (Windows NT).
-ocfs2.txt
- - info and mount options for the OCFS2 clustered filesystem.
-omfs.txt
- - info on the Optimized MPEG FileSystem.
-path-lookup.txt
- - info on path walking and name lookup locking.
-pohmelfs/
- - directory containing pohmelfs filesystem documentation.
-porting
- - various information on filesystem porting.
-proc.txt
- - info on Linux's /proc filesystem.
-qnx6.txt
- - info on the QNX6 filesystem.
-quota.txt
- - info on Quota subsystem.
-ramfs-rootfs-initramfs.txt
- - info on the 'in memory' filesystems ramfs, rootfs and initramfs.
-relay.txt
- - info on relay, for efficient streaming from kernel to user space.
-romfs.txt
- - description of the ROMFS filesystem.
-seq_file.txt
- - how to use the seq_file API.
-sharedsubtree.txt
- - a description of shared subtrees for namespaces.
-spufs.txt
- - info and mount options for the SPU filesystem used on Cell.
-squashfs.txt
- - info on the squashfs filesystem.
-sysfs-pci.txt
- - info on accessing PCI device resources through sysfs.
-sysfs-tagging.txt
- - info on sysfs tagging to avoid duplicates.
-sysfs.txt
- - info on sysfs, a ram-based filesystem for exporting kernel objects.
-sysv-fs.txt
- - info on the SystemV/V7/Xenix/Coherent filesystem.
-tmpfs.txt
- - info on tmpfs, a filesystem that holds all files in virtual memory.
-ubifs.txt
- - info on the Unsorted Block Images FileSystem.
-udf.txt
- - info and mount options for the UDF filesystem.
-ufs.txt
- - info on the ufs filesystem.
-vfat.txt
- - info on using the VFAT filesystem used in Windows NT and Windows 95.
-vfs.txt
- - overview of the Virtual File System.
-xfs-delayed-logging-design.txt
- - info on the XFS Delayed Logging Design.
-xfs-self-describing-metadata.txt
- - info on XFS Self Describing Metadata.
-xfs.txt
- - info and mount options for the XFS filesystem.
diff --git a/Documentation/filesystems/api-summary.rst b/Documentation/filesystems/api-summary.rst
new file mode 100644
index 0000000..bbb0c1c
--- /dev/null
+++ b/Documentation/filesystems/api-summary.rst
@@ -0,0 +1,147 @@
+=============================
+Linux Filesystems API summary
+=============================
+
+This section contains API-level documentation, mostly taken from the source
+code itself.
+
+The Linux VFS
+=============
+
+The Filesystem types
+--------------------
+
+.. kernel-doc:: include/linux/fs.h
+ :internal:
+
+The Directory Cache
+-------------------
+
+.. kernel-doc:: fs/dcache.c
+ :export:
+
+.. kernel-doc:: include/linux/dcache.h
+ :internal:
+
+Inode Handling
+--------------
+
+.. kernel-doc:: fs/inode.c
+ :export:
+
+.. kernel-doc:: fs/bad_inode.c
+ :export:
+
+Registration and Superblocks
+----------------------------
+
+.. kernel-doc:: fs/super.c
+ :export:
+
+File Locks
+----------
+
+.. kernel-doc:: fs/locks.c
+ :export:
+
+.. kernel-doc:: fs/locks.c
+ :internal:
+
+Other Functions
+---------------
+
+.. kernel-doc:: fs/mpage.c
+ :export:
+
+.. kernel-doc:: fs/namei.c
+ :export:
+
+.. kernel-doc:: fs/buffer.c
+ :export:
+
+.. kernel-doc:: block/bio.c
+ :export:
+
+.. kernel-doc:: fs/seq_file.c
+ :export:
+
+.. kernel-doc:: fs/filesystems.c
+ :export:
+
+.. kernel-doc:: fs/fs-writeback.c
+ :export:
+
+.. kernel-doc:: fs/block_dev.c
+ :export:
+
+.. kernel-doc:: fs/anon_inodes.c
+ :export:
+
+.. kernel-doc:: fs/attr.c
+ :export:
+
+.. kernel-doc:: fs/d_path.c
+ :export:
+
+.. kernel-doc:: fs/dax.c
+ :export:
+
+.. kernel-doc:: fs/direct-io.c
+ :export:
+
+.. kernel-doc:: fs/libfs.c
+ :export:
+
+.. kernel-doc:: fs/posix_acl.c
+ :export:
+
+.. kernel-doc:: fs/stat.c
+ :export:
+
+.. kernel-doc:: fs/sync.c
+ :export:
+
+.. kernel-doc:: fs/xattr.c
+ :export:
+
+The proc filesystem
+===================
+
+sysctl interface
+----------------
+
+.. kernel-doc:: kernel/sysctl.c
+ :export:
+
+proc filesystem interface
+-------------------------
+
+.. kernel-doc:: fs/proc/base.c
+ :internal:
+
+Events based on file descriptors
+================================
+
+.. kernel-doc:: fs/eventfd.c
+ :export:
+
+The Filesystem for Exporting Kernel Objects
+===========================================
+
+.. kernel-doc:: fs/sysfs/file.c
+ :export:
+
+.. kernel-doc:: fs/sysfs/symlink.c
+ :export:
+
+The debugfs filesystem
+======================
+
+debugfs interface
+-----------------
+
+.. kernel-doc:: fs/debugfs/inode.c
+ :export:
+
+.. kernel-doc:: fs/debugfs/file.c
+ :export:
diff --git a/Documentation/filesystems/autofs-mount-control.txt b/Documentation/filesystems/autofs-mount-control.txt
index 45edad6..acc02fc 100644
--- a/Documentation/filesystems/autofs-mount-control.txt
+++ b/Documentation/filesystems/autofs-mount-control.txt
@@ -354,8 +354,10 @@
The call requires an initialized struct autofs_dev_ioctl with the
ioctlfd field set to the descriptor obtained from the open call. In
-addition an immediate expire, independent of the mount timeout, can be
-requested by setting the how field of struct args_expire to 1. If no
+addition an immediate expire that's independent of the mount timeout,
+and a forced expire that's independent of whether the mount is busy,
+can be requested by setting the how field of struct args_expire to
+AUTOFS_EXP_IMMEDIATE or AUTOFS_EXP_FORCED, respectively . If no
expire candidates can be found the ioctl returns -1 with errno set to
EAGAIN.
diff --git a/Documentation/filesystems/autofs.txt b/Documentation/filesystems/autofs.txt
index 373ad25..3af38c7 100644
--- a/Documentation/filesystems/autofs.txt
+++ b/Documentation/filesystems/autofs.txt
@@ -116,7 +116,7 @@
**DCACHE_MANAGE_TRANSIT**
If a dentry has DCACHE_MANAGE_TRANSIT set then two very different but
-related behaviors are invoked, both using the `d_op->d_manage()`
+related behaviours are invoked, both using the `d_op->d_manage()`
dentry operation.
Firstly, before checking to see if any filesystem is mounted on the
@@ -193,8 +193,8 @@
RCU-walk mode by returning `-ECHILD`.
So `d_manage()`, when called with `rcu_walk` set, should either return
--ECHILD if there is any reason to believe it is unsafe to end the
-mounted filesystem, and otherwise should return 0.
+-ECHILD if there is any reason to believe it is unsafe to enter the
+mounted filesystem, otherwise it should return 0.
autofs will return `-ECHILD` if an expiry of the filesystem has been
initiated or is being considered, otherwise it returns 0.
@@ -210,7 +210,7 @@
mounted. As autofs doesn't return such a filesystem but leaves the
mounting to the automount daemon, it must involve the automount daemon
in unmounting as well. This also means that autofs has more control
-of expiry.
+over expiry.
The VFS also supports "expiry" of mounts using the MNT_EXPIRE flag to
the `umount` system call. Unmounting with MNT_EXPIRE will fail unless
@@ -225,7 +225,7 @@
symbolic links or empty directories any time it likes. If the unmount
or removal is successful the filesystem will be returned to the state
it was before the mount or creation, so that any access of the name
-will trigger normal auto-mount processing. In particlar, `rmdir` and
+will trigger normal auto-mount processing. In particular, `rmdir` and
`unlink` do not leave negative entries in the dcache as a normal
filesystem would, so an attempt to access a recently-removed object is
passed to autofs for handling.
@@ -240,11 +240,18 @@
used for a while. For this purpose autofs maintains a "`last_used`"
time stamp on each directory or symlink. For symlinks it genuinely
does record the last time the symlink was "used" or followed to find
-out where it points to. For directories the field is a slight
-misnomer. It actually records the last time that autofs checked if
-the directory or one of its descendents was busy and found that it
-was. This is just as useful and doesn't require updating the field so
-often.
+out where it points to. For directories the field is used slightly
+differently. The field is updated at mount time and during expire
+checks if it is found to be in use (ie. open file descriptor or
+process working directory) and during path walks. The update done
+during path walks prevents frequent expire and immediate mount of
+frequently accessed automounts. But in the case where a GUI continually
+access or an application frequently scans an autofs directory tree
+there can be an accumulation of mounts that aren't actually being
+used. To cater for this case the "`strictexpire`" autofs mount option
+can be used to avoid the "`last_used`" update on path walk thereby
+preventing this apparent inability to expire mounts that aren't
+really in use.
The daemon is able to ask autofs if anything is due to be expired,
using an `ioctl` as discussed later. For a *direct* mount, autofs
@@ -255,8 +262,12 @@
There is an option with indirect mounts to consider each of the leaves
that has been mounted on instead of considering the top-level names.
-This is intended for compatability with version 4 of autofs and should
-be considered as deprecated.
+This was originally intended for compatibility with version 4 of autofs
+and should be considered as deprecated for Sun Format automount maps.
+However, it may be used again for amd format mount maps (which are
+generally indirect maps) because the amd automounter allows for the
+setting of an expire timeout for individual mounts. But there are
+some difficulties in making the needed changes for this.
When autofs considers a directory it checks the `last_used` time and
compares it with the "timeout" value set when the filesystem was
@@ -273,7 +284,7 @@
return the name of that thing. Once a name has been returned the
automount daemon needs to unmount any filesystems mounted below the
name normally. As described above, this is unsafe for non-toplevel
-mounts in a version-5 autofs. For this reason the current `automountd`
+mounts in a version-5 autofs. For this reason the current `automount(8)`
does not use this ioctl.
The second mechanism uses either the **AUTOFS_DEV_IOCTL_EXPIRE_CMD** or
@@ -345,7 +356,7 @@
particular request to be acknowledged. When a message is sent over
the pipe the affected dentry is marked as either "active" or
"expiring" and other accesses to it block until the message is
-acknowledged using one of the ioctls below and the relevant
+acknowledged using one of the ioctls below with the relevant
`wait_queue_token`.
Communicating with autofs: root directory ioctls
@@ -367,15 +378,14 @@
This mode is also entered if a write to the pipe fails.
- **AUTOFS_IOC_PROTOVER**: This returns the protocol version in use.
- **AUTOFS_IOC_PROTOSUBVER**: Returns the protocol sub-version which
- is really a version number for the implementation. It is
- currently 2.
+ is really a version number for the implementation.
- **AUTOFS_IOC_SETTIMEOUT**: This passes a pointer to an unsigned
long. The value is used to set the timeout for expiry, and
the current timeout value is stored back through the pointer.
- **AUTOFS_IOC_ASKUMOUNT**: Returns, in the pointed-to `int`, 1 if
the filesystem could be unmounted. This is only a hint as
the situation could change at any instant. This call can be
- use to avoid a more expensive full unmount attempt.
+ used to avoid a more expensive full unmount attempt.
- **AUTOFS_IOC_EXPIRE**: as described above, this asks if there is
anything suitable to expire. A pointer to a packet:
@@ -400,6 +410,11 @@
**AUTOFS_EXP_IMMEDIATE** causes `last_used` time to be ignored
and objects are expired if the are not in use.
+ **AUTOFS_EXP_FORCED** causes the in use status to be ignored
+ and objects are expired ieven if they are in use. This assumes
+ that the daemon has requested this because it is capable of
+ performing the umount.
+
**AUTOFS_EXP_LEAVES** will select a leaf rather than a top-level
name to expire. This is only safe when *maxproto* is 4.
@@ -415,7 +430,7 @@
It requires CAP_SYS_ADMIN for access.
The `ioctl`s that can be used on this device are described in a separate
-document `autofs-mount-control.txt`, and are summarized briefly here.
+document `autofs-mount-control.txt`, and are summarised briefly here.
Each ioctl is passed a pointer to an `autofs_dev_ioctl` structure:
struct autofs_dev_ioctl {
@@ -511,6 +526,21 @@
Catatonic mode can only be left via the
**AUTOFS_DEV_IOCTL_OPENMOUNT_CMD** ioctl on the `/dev/autofs`.
+The "ignore" mount option
+-------------------------
+
+The "ignore" mount option can be used to provide a generic indicator
+to applications that the mount entry should be ignored when displaying
+mount information.
+
+In other OSes that provide autofs and that provide a mount list to user
+space based on the kernel mount list a no-op mount option ("ignore" is
+the one use on the most common OSes) is allowed so that autofs file
+system users can optionally use it.
+
+This is intended to be used by user space programs to exclude autofs
+mounts from consideration when reading the mounts list.
+
autofs, name spaces, and shared mounts
--------------------------------------
diff --git a/Documentation/filesystems/caching/backend-api.txt b/Documentation/filesystems/caching/backend-api.txt
index c0bd567..c418280 100644
--- a/Documentation/filesystems/caching/backend-api.txt
+++ b/Documentation/filesystems/caching/backend-api.txt
@@ -704,7 +704,7 @@
void fscache_get_retrieval(struct fscache_retrieval *op);
void fscache_put_retrieval(struct fscache_retrieval *op);
- These two functions are used to retain a retrieval record whilst doing
+ These two functions are used to retain a retrieval record while doing
asynchronous data retrieval and block allocation.
diff --git a/Documentation/filesystems/caching/cachefiles.txt b/Documentation/filesystems/caching/cachefiles.txt
index 748a1ae..28aefcb 100644
--- a/Documentation/filesystems/caching/cachefiles.txt
+++ b/Documentation/filesystems/caching/cachefiles.txt
@@ -45,7 +45,7 @@
CacheFiles creates a misc character device - "/dev/cachefiles" - that is used
to communication with the daemon. Only one thing may have this open at once,
-and whilst it is open, a cache is at least partially in existence. The daemon
+and while it is open, a cache is at least partially in existence. The daemon
opens this and sends commands down it to control the cache.
CacheFiles is currently limited to a single cache.
@@ -163,7 +163,7 @@
kernel module contains its own very cut-down path walking facility that ignores
mountpoints, but the daemon can't avoid them.
-Do not create, rename or unlink files and directories in the cache whilst the
+Do not create, rename or unlink files and directories in the cache while the
cache is active, as this may cause the state to become uncertain.
Renaming files in the cache might make objects appear to be other objects (the
diff --git a/Documentation/filesystems/caching/netfs-api.txt b/Documentation/filesystems/caching/netfs-api.txt
index 2a6f739..ba968e8 100644
--- a/Documentation/filesystems/caching/netfs-api.txt
+++ b/Documentation/filesystems/caching/netfs-api.txt
@@ -382,7 +382,7 @@
An optional step is to request an object of miscellaneous type be created in
the cache. This is almost identical to index cookie acquisition. The only
difference is that the type in the object definition should be something other
-than index type. Whilst the parent object could be an index, it's more likely
+than index type. While the parent object could be an index, it's more likely
it would be some other type of object such as a data file.
xattr->cache =
diff --git a/Documentation/filesystems/caching/operations.txt b/Documentation/filesystems/caching/operations.txt
index a1c052c..d8976c4 100644
--- a/Documentation/filesystems/caching/operations.txt
+++ b/Documentation/filesystems/caching/operations.txt
@@ -171,7 +171,7 @@
(3) If the submitting thread wants to do the work itself, and has marked the
operation with FSCACHE_OP_MYTHREAD, then it should monitor
FSCACHE_OP_WAITING as described above and check the state of the object if
- necessary (the object might have died whilst the thread was waiting).
+ necessary (the object might have died while the thread was waiting).
When it has finished doing its processing, it should call
fscache_op_complete() and fscache_put_operation() on it.
diff --git a/Documentation/filesystems/ceph.txt b/Documentation/filesystems/ceph.txt
index 8bf6224..b19b6a0 100644
--- a/Documentation/filesystems/ceph.txt
+++ b/Documentation/filesystems/ceph.txt
@@ -22,9 +22,7 @@
on symmetric access by all clients to shared block devices, Ceph
separates data and metadata management into independent server
clusters, similar to Lustre. Unlike Lustre, however, metadata and
-storage nodes run entirely as user space daemons. Storage nodes
-utilize btrfs to store data objects, leveraging its advanced features
-(checksumming, metadata replication, etc.). File data is striped
+storage nodes run entirely as user space daemons. File data is striped
across storage nodes in large chunks to distribute workload and
facilitate high throughputs. When storage nodes fail, data is
re-replicated in a distributed fashion by the storage nodes themselves
@@ -118,6 +116,10 @@
of a non-responsive Ceph file system. The default is 30
seconds.
+ caps_max=X
+ Specify the maximum number of caps to hold. Unused caps are released
+ when number of caps exceeds the limit. The default is 0 (no limit)
+
rbytes
When stat() is called on a directory, set st_size to 'rbytes',
the summation of file sizes over all files nested beneath that
@@ -151,15 +153,34 @@
Report overall filesystem usage in statfs instead of using the root
directory quota.
+ nocopyfrom
+ Don't use the RADOS 'copy-from' operation to perform remote object
+ copies. Currently, it's only used in copy_file_range, which will revert
+ to the default VFS implementation if this option is used.
+
+ recover_session=<no|clean>
+ Set auto reconnect mode in the case where the client is blacklisted. The
+ available modes are "no" and "clean". The default is "no".
+
+ * no: never attempt to reconnect when client detects that it has been
+ blacklisted. Operations will generally fail after being blacklisted.
+
+ * clean: client reconnects to the ceph cluster automatically when it
+ detects that it has been blacklisted. During reconnect, client drops
+ dirty data/metadata, invalidates page caches and writable file handles.
+ After reconnect, file locks become stale because the MDS loses track
+ of them. If an inode contains any stale file locks, read/write on the
+ inode is not allowed until applications release all stale file locks.
+
More Information
================
For more information on Ceph, see the home page at
- http://ceph.newdream.net/
+ https://ceph.com/
The Linux kernel client source tree is available at
- git://ceph.newdream.net/git/ceph-client.git
+ https://github.com/ceph/ceph-client.git
git://git.kernel.org/pub/scm/linux/kernel/git/sage/ceph-client.git
and the source for the full system is at
- git://ceph.newdream.net/git/ceph.git
+ https://github.com/ceph/ceph.git
diff --git a/Documentation/filesystems/cifs/AUTHORS b/Documentation/filesystems/cifs/AUTHORS
deleted file mode 100644
index 75865da..0000000
--- a/Documentation/filesystems/cifs/AUTHORS
+++ /dev/null
@@ -1,63 +0,0 @@
-Original Author
-===============
-Steve French (sfrench@samba.org)
-
-The author wishes to express his appreciation and thanks to:
-Andrew Tridgell (Samba team) for his early suggestions about smb/cifs VFS
-improvements. Thanks to IBM for allowing me time and test resources to pursue
-this project, to Jim McDonough from IBM (and the Samba Team) for his help, to
-the IBM Linux JFS team for explaining many esoteric Linux filesystem features.
-Jeremy Allison of the Samba team has done invaluable work in adding the server
-side of the original CIFS Unix extensions and reviewing and implementing
-portions of the newer CIFS POSIX extensions into the Samba 3 file server. Thank
-Dave Boutcher of IBM Rochester (author of the OS/400 smb/cifs filesystem client)
-for proving years ago that very good smb/cifs clients could be done on Unix-like
-operating systems. Volker Lendecke, Andrew Tridgell, Urban Widmark, John
-Newbigin and others for their work on the Linux smbfs module. Thanks to
-the other members of the Storage Network Industry Association CIFS Technical
-Workgroup for their work specifying this highly complex protocol and finally
-thanks to the Samba team for their technical advice and encouragement.
-
-Patch Contributors
-------------------
-Zwane Mwaikambo
-Andi Kleen
-Amrut Joshi
-Shobhit Dayal
-Sergey Vlasov
-Richard Hughes
-Yury Umanets
-Mark Hamzy (for some of the early cifs IPv6 work)
-Domen Puncer
-Jesper Juhl (in particular for lots of whitespace/formatting cleanup)
-Vince Negri and Dave Stahl (for finding an important caching bug)
-Adrian Bunk (kcalloc cleanups)
-Miklos Szeredi
-Kazeon team for various fixes especially for 2.4 version.
-Asser Ferno (Change Notify support)
-Shaggy (Dave Kleikamp) for innumerable small fs suggestions and some good cleanup
-Gunter Kukkukk (testing and suggestions for support of old servers)
-Igor Mammedov (DFS support)
-Jeff Layton (many, many fixes, as well as great work on the cifs Kerberos code)
-Scott Lovenberg
-Pavel Shilovsky (for great work adding SMB2 support, and various SMB3 features)
-Aurelien Aptel (for DFS SMB3 work and some key bug fixes)
-Ronnie Sahlberg (for SMB3 xattr work, bug fixes, and lots of great work on compounding)
-Shirish Pargaonkar (for many ACL patches over the years)
-Sachin Prabhu (many bug fixes, including for reconnect, copy offload and security)
-Paulo Alcantara
-Long Li (some great work on RDMA, SMB Direct)
-
-
-Test case and Bug Report contributors
--------------------------------------
-Thanks to those in the community who have submitted detailed bug reports
-and debug of problems they have found: Jochen Dolze, David Blaine,
-Rene Scharfe, Martin Josefsson, Alexander Wild, Anthony Liguori,
-Lars Muller, Urban Widmark, Massimiliano Ferrero, Howard Owen,
-Olaf Kirch, Kieron Briggs, Nick Millington and others. Also special
-mention to the Stanford Checker (SWAT) which pointed out many minor
-bugs in error paths. Valuable suggestions also have come from Al Viro
-and Dave Miller.
-
-And thanks to the IBM LTC and Power test teams and SuSE and Citrix and RedHat testers for finding multiple bugs during excellent stress test runs.
diff --git a/Documentation/filesystems/cifs/CHANGES b/Documentation/filesystems/cifs/CHANGES
deleted file mode 100644
index 1df7f49..0000000
--- a/Documentation/filesystems/cifs/CHANGES
+++ /dev/null
@@ -1,4 +0,0 @@
-See https://wiki.samba.org/index.php/LinuxCIFSKernel for summary
-information (that may be easier to read than parsing the output of
-"git log fs/cifs") about fixes/improvements to CIFS/SMB2/SMB3 support (changes
-to cifs.ko module) by kernel version (and cifs internal module version).
diff --git a/Documentation/filesystems/cifs/README b/Documentation/filesystems/cifs/README
deleted file mode 100644
index 4a80461..0000000
--- a/Documentation/filesystems/cifs/README
+++ /dev/null
@@ -1,743 +0,0 @@
-This module supports the SMB3 family of advanced network protocols (as well
-as older dialects, originally called "CIFS" or SMB1).
-
-The CIFS VFS module for Linux supports many advanced network filesystem
-features such as hierarchical DFS like namespace, hardlinks, locking and more.
-It was designed to comply with the SNIA CIFS Technical Reference (which
-supersedes the 1992 X/Open SMB Standard) as well as to perform best practice
-practical interoperability with Windows 2000, Windows XP, Samba and equivalent
-servers. This code was developed in participation with the Protocol Freedom
-Information Foundation. CIFS and now SMB3 has now become a defacto
-standard for interoperating between Macs and Windows and major NAS appliances.
-
-Please see
- MS-SMB2 (for detailed SMB2/SMB3/SMB3.1.1 protocol specification)
- http://protocolfreedom.org/ and
- http://samba.org/samba/PFIF/
-for more details.
-
-
-For questions or bug reports please contact:
- smfrench@gmail.com
-
-See the project page at: https://wiki.samba.org/index.php/LinuxCIFS_utils
-
-Build instructions:
-==================
-For Linux:
-1) Download the kernel (e.g. from http://www.kernel.org)
-and change directory into the top of the kernel directory tree
-(e.g. /usr/src/linux-2.5.73)
-2) make menuconfig (or make xconfig)
-3) select cifs from within the network filesystem choices
-4) save and exit
-5) make
-
-
-Installation instructions:
-=========================
-If you have built the CIFS vfs as module (successfully) simply
-type "make modules_install" (or if you prefer, manually copy the file to
-the modules directory e.g. /lib/modules/2.4.10-4GB/kernel/fs/cifs/cifs.ko).
-
-If you have built the CIFS vfs into the kernel itself, follow the instructions
-for your distribution on how to install a new kernel (usually you
-would simply type "make install").
-
-If you do not have the utility mount.cifs (in the Samba 4.x source tree and on
-the CIFS VFS web site) copy it to the same directory in which mount helpers
-reside (usually /sbin). Although the helper software is not
-required, mount.cifs is recommended. Most distros include a "cifs-utils"
-package that includes this utility so it is recommended to install this.
-
-Note that running the Winbind pam/nss module (logon service) on all of your
-Linux clients is useful in mapping Uids and Gids consistently across the
-domain to the proper network user. The mount.cifs mount helper can be
-found at cifs-utils.git on git.samba.org
-
-If cifs is built as a module, then the size and number of network buffers
-and maximum number of simultaneous requests to one server can be configured.
-Changing these from their defaults is not recommended. By executing modinfo
- modinfo kernel/fs/cifs/cifs.ko
-on kernel/fs/cifs/cifs.ko the list of configuration changes that can be made
-at module initialization time (by running insmod cifs.ko) can be seen.
-
-Recommendations
-===============
-To improve security the SMB2.1 dialect or later (usually will get SMB3) is now
-the new default. To use old dialects (e.g. to mount Windows XP) use "vers=1.0"
-on mount (or vers=2.0 for Windows Vista). Note that the CIFS (vers=1.0) is
-much older and less secure than the default dialect SMB3 which includes
-many advanced security features such as downgrade attack detection
-and encrypted shares and stronger signing and authentication algorithms.
-There are additional mount options that may be helpful for SMB3 to get
-improved POSIX behavior (NB: can use vers=3.0 to force only SMB3, never 2.1):
- "mfsymlinks" and "cifsacl" and "idsfromsid"
-
-Allowing User Mounts
-====================
-To permit users to mount and unmount over directories they own is possible
-with the cifs vfs. A way to enable such mounting is to mark the mount.cifs
-utility as suid (e.g. "chmod +s /sbin/mount.cifs). To enable users to
-umount shares they mount requires
-1) mount.cifs version 1.4 or later
-2) an entry for the share in /etc/fstab indicating that a user may
-unmount it e.g.
-//server/usersharename /mnt/username cifs user 0 0
-
-Note that when the mount.cifs utility is run suid (allowing user mounts),
-in order to reduce risks, the "nosuid" mount flag is passed in on mount to
-disallow execution of an suid program mounted on the remote target.
-When mount is executed as root, nosuid is not passed in by default,
-and execution of suid programs on the remote target would be enabled
-by default. This can be changed, as with nfs and other filesystems,
-by simply specifying "nosuid" among the mount options. For user mounts
-though to be able to pass the suid flag to mount requires rebuilding
-mount.cifs with the following flag: CIFS_ALLOW_USR_SUID
-
-There is a corresponding manual page for cifs mounting in the Samba 3.0 and
-later source tree in docs/manpages/mount.cifs.8
-
-Allowing User Unmounts
-======================
-To permit users to ummount directories that they have user mounted (see above),
-the utility umount.cifs may be used. It may be invoked directly, or if
-umount.cifs is placed in /sbin, umount can invoke the cifs umount helper
-(at least for most versions of the umount utility) for umount of cifs
-mounts, unless umount is invoked with -i (which will avoid invoking a umount
-helper). As with mount.cifs, to enable user unmounts umount.cifs must be marked
-as suid (e.g. "chmod +s /sbin/umount.cifs") or equivalent (some distributions
-allow adding entries to a file to the /etc/permissions file to achieve the
-equivalent suid effect). For this utility to succeed the target path
-must be a cifs mount, and the uid of the current user must match the uid
-of the user who mounted the resource.
-
-Also note that the customary way of allowing user mounts and unmounts is
-(instead of using mount.cifs and unmount.cifs as suid) to add a line
-to the file /etc/fstab for each //server/share you wish to mount, but
-this can become unwieldy when potential mount targets include many
-or unpredictable UNC names.
-
-Samba Considerations
-====================
-Most current servers support SMB2.1 and SMB3 which are more secure,
-but there are useful protocol extensions for the older less secure CIFS
-dialect, so to get the maximum benefit if mounting using the older dialect
-(CIFS/SMB1), we recommend using a server that supports the SNIA CIFS
-Unix Extensions standard (e.g. almost any version of Samba ie version
-2.2.5 or later) but the CIFS vfs works fine with a wide variety of CIFS servers.
-Note that uid, gid and file permissions will display default values if you do
-not have a server that supports the Unix extensions for CIFS (such as Samba
-2.2.5 or later). To enable the Unix CIFS Extensions in the Samba server, add
-the line:
-
- unix extensions = yes
-
-to your smb.conf file on the server. Note that the following smb.conf settings
-are also useful (on the Samba server) when the majority of clients are Unix or
-Linux:
-
- case sensitive = yes
- delete readonly = yes
- ea support = yes
-
-Note that server ea support is required for supporting xattrs from the Linux
-cifs client, and that EA support is present in later versions of Samba (e.g.
-3.0.6 and later (also EA support works in all versions of Windows, at least to
-shares on NTFS filesystems). Extended Attribute (xattr) support is an optional
-feature of most Linux filesystems which may require enabling via
-make menuconfig. Client support for extended attributes (user xattr) can be
-disabled on a per-mount basis by specifying "nouser_xattr" on mount.
-
-The CIFS client can get and set POSIX ACLs (getfacl, setfacl) to Samba servers
-version 3.10 and later. Setting POSIX ACLs requires enabling both XATTR and
-then POSIX support in the CIFS configuration options when building the cifs
-module. POSIX ACL support can be disabled on a per mount basic by specifying
-"noacl" on mount.
-
-Some administrators may want to change Samba's smb.conf "map archive" and
-"create mask" parameters from the default. Unless the create mask is changed
-newly created files can end up with an unnecessarily restrictive default mode,
-which may not be what you want, although if the CIFS Unix extensions are
-enabled on the server and client, subsequent setattr calls (e.g. chmod) can
-fix the mode. Note that creating special devices (mknod) remotely
-may require specifying a mkdev function to Samba if you are not using
-Samba 3.0.6 or later. For more information on these see the manual pages
-("man smb.conf") on the Samba server system. Note that the cifs vfs,
-unlike the smbfs vfs, does not read the smb.conf on the client system
-(the few optional settings are passed in on mount via -o parameters instead).
-Note that Samba 2.2.7 or later includes a fix that allows the CIFS VFS to delete
-open files (required for strict POSIX compliance). Windows Servers already
-supported this feature. Samba server does not allow symlinks that refer to files
-outside of the share, so in Samba versions prior to 3.0.6, most symlinks to
-files with absolute paths (ie beginning with slash) such as:
- ln -s /mnt/foo bar
-would be forbidden. Samba 3.0.6 server or later includes the ability to create
-such symlinks safely by converting unsafe symlinks (ie symlinks to server
-files that are outside of the share) to a samba specific format on the server
-that is ignored by local server applications and non-cifs clients and that will
-not be traversed by the Samba server). This is opaque to the Linux client
-application using the cifs vfs. Absolute symlinks will work to Samba 3.0.5 or
-later, but only for remote clients using the CIFS Unix extensions, and will
-be invisbile to Windows clients and typically will not affect local
-applications running on the same server as Samba.
-
-Use instructions:
-================
-Once the CIFS VFS support is built into the kernel or installed as a module
-(cifs.ko), you can use mount syntax like the following to access Samba or
-Mac or Windows servers:
-
- mount -t cifs //9.53.216.11/e$ /mnt -o username=myname,password=mypassword
-
-Before -o the option -v may be specified to make the mount.cifs
-mount helper display the mount steps more verbosely.
-After -o the following commonly used cifs vfs specific options
-are supported:
-
- username=<username>
- password=<password>
- domain=<domain name>
-
-Other cifs mount options are described below. Use of TCP names (in addition to
-ip addresses) is available if the mount helper (mount.cifs) is installed. If
-you do not trust the server to which are mounted, or if you do not have
-cifs signing enabled (and the physical network is insecure), consider use
-of the standard mount options "noexec" and "nosuid" to reduce the risk of
-running an altered binary on your local system (downloaded from a hostile server
-or altered by a hostile router).
-
-Although mounting using format corresponding to the CIFS URL specification is
-not possible in mount.cifs yet, it is possible to use an alternate format
-for the server and sharename (which is somewhat similar to NFS style mount
-syntax) instead of the more widely used UNC format (i.e. \\server\share):
- mount -t cifs tcp_name_of_server:share_name /mnt -o user=myname,pass=mypasswd
-
-When using the mount helper mount.cifs, passwords may be specified via alternate
-mechanisms, instead of specifying it after -o using the normal "pass=" syntax
-on the command line:
-1) By including it in a credential file. Specify credentials=filename as one
-of the mount options. Credential files contain two lines
- username=someuser
- password=your_password
-2) By specifying the password in the PASSWD environment variable (similarly
-the user name can be taken from the USER environment variable).
-3) By specifying the password in a file by name via PASSWD_FILE
-4) By specifying the password in a file by file descriptor via PASSWD_FD
-
-If no password is provided, mount.cifs will prompt for password entry
-
-Restrictions
-============
-Servers must support either "pure-TCP" (port 445 TCP/IP CIFS connections) or RFC
-1001/1002 support for "Netbios-Over-TCP/IP." This is not likely to be a
-problem as most servers support this.
-
-Valid filenames differ between Windows and Linux. Windows typically restricts
-filenames which contain certain reserved characters (e.g.the character :
-which is used to delimit the beginning of a stream name by Windows), while
-Linux allows a slightly wider set of valid characters in filenames. Windows
-servers can remap such characters when an explicit mapping is specified in
-the Server's registry. Samba starting with version 3.10 will allow such
-filenames (ie those which contain valid Linux characters, which normally
-would be forbidden for Windows/CIFS semantics) as long as the server is
-configured for Unix Extensions (and the client has not disabled
-/proc/fs/cifs/LinuxExtensionsEnabled). In addition the mount option
-"mapposix" can be used on CIFS (vers=1.0) to force the mapping of
-illegal Windows/NTFS/SMB characters to a remap range (this mount parm
-is the default for SMB3). This remap ("mapposix") range is also
-compatible with Mac (and "Services for Mac" on some older Windows).
-
-CIFS VFS Mount Options
-======================
-A partial list of the supported mount options follows:
- username The user name to use when trying to establish
- the CIFS session.
- password The user password. If the mount helper is
- installed, the user will be prompted for password
- if not supplied.
- ip The ip address of the target server
- unc The target server Universal Network Name (export) to
- mount.
- domain Set the SMB/CIFS workgroup name prepended to the
- username during CIFS session establishment
- forceuid Set the default uid for inodes to the uid
- passed in on mount. For mounts to servers
- which do support the CIFS Unix extensions, such as a
- properly configured Samba server, the server provides
- the uid, gid and mode so this parameter should not be
- specified unless the server and clients uid and gid
- numbering differ. If the server and client are in the
- same domain (e.g. running winbind or nss_ldap) and
- the server supports the Unix Extensions then the uid
- and gid can be retrieved from the server (and uid
- and gid would not have to be specified on the mount.
- For servers which do not support the CIFS Unix
- extensions, the default uid (and gid) returned on lookup
- of existing files will be the uid (gid) of the person
- who executed the mount (root, except when mount.cifs
- is configured setuid for user mounts) unless the "uid="
- (gid) mount option is specified. Also note that permission
- checks (authorization checks) on accesses to a file occur
- at the server, but there are cases in which an administrator
- may want to restrict at the client as well. For those
- servers which do not report a uid/gid owner
- (such as Windows), permissions can also be checked at the
- client, and a crude form of client side permission checking
- can be enabled by specifying file_mode and dir_mode on
- the client. (default)
- forcegid (similar to above but for the groupid instead of uid) (default)
- noforceuid Fill in file owner information (uid) by requesting it from
- the server if possible. With this option, the value given in
- the uid= option (on mount) will only be used if the server
- can not support returning uids on inodes.
- noforcegid (similar to above but for the group owner, gid, instead of uid)
- uid Set the default uid for inodes, and indicate to the
- cifs kernel driver which local user mounted. If the server
- supports the unix extensions the default uid is
- not used to fill in the owner fields of inodes (files)
- unless the "forceuid" parameter is specified.
- gid Set the default gid for inodes (similar to above).
- file_mode If CIFS Unix extensions are not supported by the server
- this overrides the default mode for file inodes.
- fsc Enable local disk caching using FS-Cache (off by default). This
- option could be useful to improve performance on a slow link,
- heavily loaded server and/or network where reading from the
- disk is faster than reading from the server (over the network).
- This could also impact scalability positively as the
- number of calls to the server are reduced. However, local
- caching is not suitable for all workloads for e.g. read-once
- type workloads. So, you need to consider carefully your
- workload/scenario before using this option. Currently, local
- disk caching is functional for CIFS files opened as read-only.
- dir_mode If CIFS Unix extensions are not supported by the server
- this overrides the default mode for directory inodes.
- port attempt to contact the server on this tcp port, before
- trying the usual ports (port 445, then 139).
- iocharset Codepage used to convert local path names to and from
- Unicode. Unicode is used by default for network path
- names if the server supports it. If iocharset is
- not specified then the nls_default specified
- during the local client kernel build will be used.
- If server does not support Unicode, this parameter is
- unused.
- rsize default read size (usually 16K). The client currently
- can not use rsize larger than CIFSMaxBufSize. CIFSMaxBufSize
- defaults to 16K and may be changed (from 8K to the maximum
- kmalloc size allowed by your kernel) at module install time
- for cifs.ko. Setting CIFSMaxBufSize to a very large value
- will cause cifs to use more memory and may reduce performance
- in some cases. To use rsize greater than 127K (the original
- cifs protocol maximum) also requires that the server support
- a new Unix Capability flag (for very large read) which some
- newer servers (e.g. Samba 3.0.26 or later) do. rsize can be
- set from a minimum of 2048 to a maximum of 130048 (127K or
- CIFSMaxBufSize, whichever is smaller)
- wsize default write size (default 57344)
- maximum wsize currently allowed by CIFS is 57344 (fourteen
- 4096 byte pages)
- actimeo=n attribute cache timeout in seconds (default 1 second).
- After this timeout, the cifs client requests fresh attribute
- information from the server. This option allows to tune the
- attribute cache timeout to suit the workload needs. Shorter
- timeouts mean better the cache coherency, but increased number
- of calls to the server. Longer timeouts mean reduced number
- of calls to the server at the expense of less stricter cache
- coherency checks (i.e. incorrect attribute cache for a short
- period of time).
- rw mount the network share read-write (note that the
- server may still consider the share read-only)
- ro mount network share read-only
- version used to distinguish different versions of the
- mount helper utility (not typically needed)
- sep if first mount option (after the -o), overrides
- the comma as the separator between the mount
- parms. e.g.
- -o user=myname,password=mypassword,domain=mydom
- could be passed instead with period as the separator by
- -o sep=.user=myname.password=mypassword.domain=mydom
- this might be useful when comma is contained within username
- or password or domain. This option is less important
- when the cifs mount helper cifs.mount (version 1.1 or later)
- is used.
- nosuid Do not allow remote executables with the suid bit
- program to be executed. This is only meaningful for mounts
- to servers such as Samba which support the CIFS Unix Extensions.
- If you do not trust the servers in your network (your mount
- targets) it is recommended that you specify this option for
- greater security.
- exec Permit execution of binaries on the mount.
- noexec Do not permit execution of binaries on the mount.
- dev Recognize block devices on the remote mount.
- nodev Do not recognize devices on the remote mount.
- suid Allow remote files on this mountpoint with suid enabled to
- be executed (default for mounts when executed as root,
- nosuid is default for user mounts).
- credentials Although ignored by the cifs kernel component, it is used by
- the mount helper, mount.cifs. When mount.cifs is installed it
- opens and reads the credential file specified in order
- to obtain the userid and password arguments which are passed to
- the cifs vfs.
- guest Although ignored by the kernel component, the mount.cifs
- mount helper will not prompt the user for a password
- if guest is specified on the mount options. If no
- password is specified a null password will be used.
- perm Client does permission checks (vfs_permission check of uid
- and gid of the file against the mode and desired operation),
- Note that this is in addition to the normal ACL check on the
- target machine done by the server software.
- Client permission checking is enabled by default.
- noperm Client does not do permission checks. This can expose
- files on this mount to access by other users on the local
- client system. It is typically only needed when the server
- supports the CIFS Unix Extensions but the UIDs/GIDs on the
- client and server system do not match closely enough to allow
- access by the user doing the mount, but it may be useful with
- non CIFS Unix Extension mounts for cases in which the default
- mode is specified on the mount but is not to be enforced on the
- client (e.g. perhaps when MultiUserMount is enabled)
- Note that this does not affect the normal ACL check on the
- target machine done by the server software (of the server
- ACL against the user name provided at mount time).
- serverino Use server's inode numbers instead of generating automatically
- incrementing inode numbers on the client. Although this will
- make it easier to spot hardlinked files (as they will have
- the same inode numbers) and inode numbers may be persistent,
- note that the server does not guarantee that the inode numbers
- are unique if multiple server side mounts are exported under a
- single share (since inode numbers on the servers might not
- be unique if multiple filesystems are mounted under the same
- shared higher level directory). Note that some older
- (e.g. pre-Windows 2000) do not support returning UniqueIDs
- or the CIFS Unix Extensions equivalent and for those
- this mount option will have no effect. Exporting cifs mounts
- under nfsd requires this mount option on the cifs mount.
- This is now the default if server supports the
- required network operation.
- noserverino Client generates inode numbers (rather than using the actual one
- from the server). These inode numbers will vary after
- unmount or reboot which can confuse some applications,
- but not all server filesystems support unique inode
- numbers.
- setuids If the CIFS Unix extensions are negotiated with the server
- the client will attempt to set the effective uid and gid of
- the local process on newly created files, directories, and
- devices (create, mkdir, mknod). If the CIFS Unix Extensions
- are not negotiated, for newly created files and directories
- instead of using the default uid and gid specified on
- the mount, cache the new file's uid and gid locally which means
- that the uid for the file can change when the inode is
- reloaded (or the user remounts the share).
- nosetuids The client will not attempt to set the uid and gid on
- on newly created files, directories, and devices (create,
- mkdir, mknod) which will result in the server setting the
- uid and gid to the default (usually the server uid of the
- user who mounted the share). Letting the server (rather than
- the client) set the uid and gid is the default. If the CIFS
- Unix Extensions are not negotiated then the uid and gid for
- new files will appear to be the uid (gid) of the mounter or the
- uid (gid) parameter specified on the mount.
- netbiosname When mounting to servers via port 139, specifies the RFC1001
- source name to use to represent the client netbios machine
- name when doing the RFC1001 netbios session initialize.
- direct Do not do inode data caching on files opened on this mount.
- This precludes mmapping files on this mount. In some cases
- with fast networks and little or no caching benefits on the
- client (e.g. when the application is doing large sequential
- reads bigger than page size without rereading the same data)
- this can provide better performance than the default
- behavior which caches reads (readahead) and writes
- (writebehind) through the local Linux client pagecache
- if oplock (caching token) is granted and held. Note that
- direct allows write operations larger than page size
- to be sent to the server.
- strictcache Use for switching on strict cache mode. In this mode the
- client read from the cache all the time it has Oplock Level II,
- otherwise - read from the server. All written data are stored
- in the cache, but if the client doesn't have Exclusive Oplock,
- it writes the data to the server.
- rwpidforward Forward pid of a process who opened a file to any read or write
- operation on that file. This prevent applications like WINE
- from failing on read and write if we use mandatory brlock style.
- acl Allow setfacl and getfacl to manage posix ACLs if server
- supports them. (default)
- noacl Do not allow setfacl and getfacl calls on this mount
- user_xattr Allow getting and setting user xattrs (those attributes whose
- name begins with "user." or "os2.") as OS/2 EAs (extended
- attributes) to the server. This allows support of the
- setfattr and getfattr utilities. (default)
- nouser_xattr Do not allow getfattr/setfattr to get/set/list xattrs
- mapchars Translate six of the seven reserved characters (not backslash)
- *?<>|:
- to the remap range (above 0xF000), which also
- allows the CIFS client to recognize files created with
- such characters by Windows's POSIX emulation. This can
- also be useful when mounting to most versions of Samba
- (which also forbids creating and opening files
- whose names contain any of these seven characters).
- This has no effect if the server does not support
- Unicode on the wire.
- nomapchars Do not translate any of these seven characters (default).
- nocase Request case insensitive path name matching (case
- sensitive is the default if the server supports it).
- (mount option "ignorecase" is identical to "nocase")
- posixpaths If CIFS Unix extensions are supported, attempt to
- negotiate posix path name support which allows certain
- characters forbidden in typical CIFS filenames, without
- requiring remapping. (default)
- noposixpaths If CIFS Unix extensions are supported, do not request
- posix path name support (this may cause servers to
- reject creatingfile with certain reserved characters).
- nounix Disable the CIFS Unix Extensions for this mount (tree
- connection). This is rarely needed, but it may be useful
- in order to turn off multiple settings all at once (ie
- posix acls, posix locks, posix paths, symlink support
- and retrieving uids/gids/mode from the server) or to
- work around a bug in server which implement the Unix
- Extensions.
- nobrl Do not send byte range lock requests to the server.
- This is necessary for certain applications that break
- with cifs style mandatory byte range locks (and most
- cifs servers do not yet support requesting advisory
- byte range locks).
- forcemandatorylock Even if the server supports posix (advisory) byte range
- locking, send only mandatory lock requests. For some
- (presumably rare) applications, originally coded for
- DOS/Windows, which require Windows style mandatory byte range
- locking, they may be able to take advantage of this option,
- forcing the cifs client to only send mandatory locks
- even if the cifs server would support posix advisory locks.
- "forcemand" is accepted as a shorter form of this mount
- option.
- nostrictsync If this mount option is set, when an application does an
- fsync call then the cifs client does not send an SMB Flush
- to the server (to force the server to write all dirty data
- for this file immediately to disk), although cifs still sends
- all dirty (cached) file data to the server and waits for the
- server to respond to the write. Since SMB Flush can be
- very slow, and some servers may be reliable enough (to risk
- delaying slightly flushing the data to disk on the server),
- turning on this option may be useful to improve performance for
- applications that fsync too much, at a small risk of server
- crash. If this mount option is not set, by default cifs will
- send an SMB flush request (and wait for a response) on every
- fsync call.
- nodfs Disable DFS (global name space support) even if the
- server claims to support it. This can help work around
- a problem with parsing of DFS paths with Samba server
- versions 3.0.24 and 3.0.25.
- remount remount the share (often used to change from ro to rw mounts
- or vice versa)
- cifsacl Report mode bits (e.g. on stat) based on the Windows ACL for
- the file. (EXPERIMENTAL)
- servern Specify the server 's netbios name (RFC1001 name) to use
- when attempting to setup a session to the server.
- This is needed for mounting to some older servers (such
- as OS/2 or Windows 98 and Windows ME) since they do not
- support a default server name. A server name can be up
- to 15 characters long and is usually uppercased.
- sfu When the CIFS Unix Extensions are not negotiated, attempt to
- create device files and fifos in a format compatible with
- Services for Unix (SFU). In addition retrieve bits 10-12
- of the mode via the SETFILEBITS extended attribute (as
- SFU does). In the future the bottom 9 bits of the
- mode also will be emulated using queries of the security
- descriptor (ACL).
- mfsymlinks Enable support for Minshall+French symlinks
- (see http://wiki.samba.org/index.php/UNIX_Extensions#Minshall.2BFrench_symlinks)
- This option is ignored when specified together with the
- 'sfu' option. Minshall+French symlinks are used even if
- the server supports the CIFS Unix Extensions.
- sign Must use packet signing (helps avoid unwanted data modification
- by intermediate systems in the route). Note that signing
- does not work with lanman or plaintext authentication.
- seal Must seal (encrypt) all data on this mounted share before
- sending on the network. Requires support for Unix Extensions.
- Note that this differs from the sign mount option in that it
- causes encryption of data sent over this mounted share but other
- shares mounted to the same server are unaffected.
- locallease This option is rarely needed. Fcntl F_SETLEASE is
- used by some applications such as Samba and NFSv4 server to
- check to see whether a file is cacheable. CIFS has no way
- to explicitly request a lease, but can check whether a file
- is cacheable (oplocked). Unfortunately, even if a file
- is not oplocked, it could still be cacheable (ie cifs client
- could grant fcntl leases if no other local processes are using
- the file) for cases for example such as when the server does not
- support oplocks and the user is sure that the only updates to
- the file will be from this client. Specifying this mount option
- will allow the cifs client to check for leases (only) locally
- for files which are not oplocked instead of denying leases
- in that case. (EXPERIMENTAL)
- sec Security mode. Allowed values are:
- none attempt to connection as a null user (no name)
- krb5 Use Kerberos version 5 authentication
- krb5i Use Kerberos authentication and packet signing
- ntlm Use NTLM password hashing (default)
- ntlmi Use NTLM password hashing with signing (if
- /proc/fs/cifs/PacketSigningEnabled on or if
- server requires signing also can be the default)
- ntlmv2 Use NTLMv2 password hashing
- ntlmv2i Use NTLMv2 password hashing with packet signing
- lanman (if configured in kernel config) use older
- lanman hash
-hard Retry file operations if server is not responding
-soft Limit retries to unresponsive servers (usually only
- one retry) before returning an error. (default)
-
-The mount.cifs mount helper also accepts a few mount options before -o
-including:
-
- -S take password from stdin (equivalent to setting the environment
- variable "PASSWD_FD=0"
- -V print mount.cifs version
- -? display simple usage information
-
-With most 2.6 kernel versions of modutils, the version of the cifs kernel
-module can be displayed via modinfo.
-
-Misc /proc/fs/cifs Flags and Debug Info
-=======================================
-Informational pseudo-files:
-DebugData Displays information about active CIFS sessions and
- shares, features enabled as well as the cifs.ko
- version.
-Stats Lists summary resource usage information as well as per
- share statistics.
-
-Configuration pseudo-files:
-SecurityFlags Flags which control security negotiation and
- also packet signing. Authentication (may/must)
- flags (e.g. for NTLM and/or NTLMv2) may be combined with
- the signing flags. Specifying two different password
- hashing mechanisms (as "must use") on the other hand
- does not make much sense. Default flags are
- 0x07007
- (NTLM, NTLMv2 and packet signing allowed). The maximum
- allowable flags if you want to allow mounts to servers
- using weaker password hashes is 0x37037 (lanman,
- plaintext, ntlm, ntlmv2, signing allowed). Some
- SecurityFlags require the corresponding menuconfig
- options to be enabled (lanman and plaintext require
- CONFIG_CIFS_WEAK_PW_HASH for example). Enabling
- plaintext authentication currently requires also
- enabling lanman authentication in the security flags
- because the cifs module only supports sending
- laintext passwords using the older lanman dialect
- form of the session setup SMB. (e.g. for authentication
- using plain text passwords, set the SecurityFlags
- to 0x30030):
-
- may use packet signing 0x00001
- must use packet signing 0x01001
- may use NTLM (most common password hash) 0x00002
- must use NTLM 0x02002
- may use NTLMv2 0x00004
- must use NTLMv2 0x04004
- may use Kerberos security 0x00008
- must use Kerberos 0x08008
- may use lanman (weak) password hash 0x00010
- must use lanman password hash 0x10010
- may use plaintext passwords 0x00020
- must use plaintext passwords 0x20020
- (reserved for future packet encryption) 0x00040
-
-cifsFYI If set to non-zero value, additional debug information
- will be logged to the system error log. This field
- contains three flags controlling different classes of
- debugging entries. The maximum value it can be set
- to is 7 which enables all debugging points (default 0).
- Some debugging statements are not compiled into the
- cifs kernel unless CONFIG_CIFS_DEBUG2 is enabled in the
- kernel configuration. cifsFYI may be set to one or
- nore of the following flags (7 sets them all):
-
- log cifs informational messages 0x01
- log return codes from cifs entry points 0x02
- log slow responses (ie which take longer than 1 second)
- CONFIG_CIFS_STATS2 must be enabled in .config 0x04
-
-
-traceSMB If set to one, debug information is logged to the
- system error log with the start of smb requests
- and responses (default 0)
-LookupCacheEnable If set to one, inode information is kept cached
- for one second improving performance of lookups
- (default 1)
-LinuxExtensionsEnabled If set to one then the client will attempt to
- use the CIFS "UNIX" extensions which are optional
- protocol enhancements that allow CIFS servers
- to return accurate UID/GID information as well
- as support symbolic links. If you use servers
- such as Samba that support the CIFS Unix
- extensions but do not want to use symbolic link
- support and want to map the uid and gid fields
- to values supplied at mount (rather than the
- actual values, then set this to zero. (default 1)
-
-These experimental features and tracing can be enabled by changing flags in
-/proc/fs/cifs (after the cifs module has been installed or built into the
-kernel, e.g. insmod cifs). To enable a feature set it to 1 e.g. to enable
-tracing to the kernel message log type:
-
- echo 7 > /proc/fs/cifs/cifsFYI
-
-cifsFYI functions as a bit mask. Setting it to 1 enables additional kernel
-logging of various informational messages. 2 enables logging of non-zero
-SMB return codes while 4 enables logging of requests that take longer
-than one second to complete (except for byte range lock requests).
-Setting it to 4 requires CONFIG_CIFS_STATS2 to be set in kernel configuration
-(.config). Setting it to seven enables all three. Finally, tracing
-the start of smb requests and responses can be enabled via:
-
- echo 1 > /proc/fs/cifs/traceSMB
-
-Per share (per client mount) statistics are available in /proc/fs/cifs/Stats.
-Additional information is available if CONFIG_CIFS_STATS2 is enabled in the
-kernel configuration (.config). The statistics returned include counters which
-represent the number of attempted and failed (ie non-zero return code from the
-server) SMB3 (or cifs) requests grouped by request type (read, write, close etc.).
-Also recorded is the total bytes read and bytes written to the server for
-that share. Note that due to client caching effects this can be less than the
-number of bytes read and written by the application running on the client.
-Statistics can be reset to zero by "echo 0 > /proc/fs/cifs/Stats" which may be
-useful if comparing performance of two different scenarios.
-
-Also note that "cat /proc/fs/cifs/DebugData" will display information about
-the active sessions and the shares that are mounted.
-
-Enabling Kerberos (extended security) works but requires version 1.2 or later
-of the helper program cifs.upcall to be present and to be configured in the
-/etc/request-key.conf file. The cifs.upcall helper program is from the Samba
-project(http://www.samba.org). NTLM and NTLMv2 and LANMAN support do not
-require this helper. Note that NTLMv2 security (which does not require the
-cifs.upcall helper program), instead of using Kerberos, is sufficient for
-some use cases.
-
-DFS support allows transparent redirection to shares in an MS-DFS name space.
-In addition, DFS support for target shares which are specified as UNC
-names which begin with host names (rather than IP addresses) requires
-a user space helper (such as cifs.upcall) to be present in order to
-translate host names to ip address, and the user space helper must also
-be configured in the file /etc/request-key.conf. Samba, Windows servers and
-many NAS appliances support DFS as a way of constructing a global name
-space to ease network configuration and improve reliability.
-
-To use cifs Kerberos and DFS support, the Linux keyutils package should be
-installed and something like the following lines should be added to the
-/etc/request-key.conf file:
-
-create cifs.spnego * * /usr/local/sbin/cifs.upcall %k
-create dns_resolver * * /usr/local/sbin/cifs.upcall %k
-
-CIFS kernel module parameters
-=============================
-These module parameters can be specified or modified either during the time of
-module loading or during the runtime by using the interface
- /proc/module/cifs/parameters/<param>
-
-i.e. echo "value" > /sys/module/cifs/parameters/<param>
-
-1. enable_oplocks - Enable or disable oplocks. Oplocks are enabled by default.
- [Y/y/1]. To disable use any of [N/n/0].
-
diff --git a/Documentation/filesystems/cifs/TODO b/Documentation/filesystems/cifs/TODO
deleted file mode 100644
index 852499a..0000000
--- a/Documentation/filesystems/cifs/TODO
+++ /dev/null
@@ -1,104 +0,0 @@
-Version 2.11 September 13, 2017
-
-A Partial List of Missing Features
-==================================
-
-Contributions are welcome. There are plenty of opportunities
-for visible, important contributions to this module. Here
-is a partial list of the known problems and missing features:
-
-a) SMB3 (and SMB3.02) missing optional features:
- - multichannel (started), integration with RDMA
- - directory leases (improved metadata caching), started (root dir only)
- - T10 copy offload ie "ODX" (copy chunk, and "Duplicate Extents" ioctl
- currently the only two server side copy mechanisms supported)
-
-b) improved sparse file support
-
-c) Directory entry caching relies on a 1 second timer, rather than
-using Directory Leases, currently only the root file handle is cached longer
-
-d) quota support (needs minor kernel change since quota calls
-to make it to network filesystems or deviceless filesystems)
-
-e) Compounding (in progress) to reduce number of roundtrips, and also
-better optimize open to reduce redundant opens (using reference counts more).
-
-f) Finish inotify support so kde and gnome file list windows
-will autorefresh (partially complete by Asser). Needs minor kernel
-vfs change to support removing D_NOTIFY on a file.
-
-g) Add GUI tool to configure /proc/fs/cifs settings and for display of
-the CIFS statistics (started)
-
-h) implement support for security and trusted categories of xattrs
-(requires minor protocol extension) to enable better support for SELINUX
-
-i) Add support for tree connect contexts (see MS-SMB2) a new SMB3.1.1 protocol
- feature (may be especially useful for virtualization).
-
-j) Create UID mapping facility so server UIDs can be mapped on a per
-mount or a per server basis to client UIDs or nobody if no mapping
-exists. Also better integration with winbind for resolving SID owners
-
-k) Add tools to take advantage of more smb3 specific ioctls and features
-(passthrough ioctl/fsctl for sending various SMB3 fsctls to the server
-is in progress)
-
-l) encrypted file support
-
-m) improved stats gathering, tools (perhaps integration with nfsometer?)
-
-n) allow setting more NTFS/SMB3 file attributes remotely (currently limited to compressed
-file attribute via chflags) and improve user space tools for managing and
-viewing them.
-
-o) mount helper GUI (to simplify the various configuration options on mount)
-
-p) Add support for witness protocol (perhaps ioctl to cifs.ko from user space
- tool listening on witness protocol RPC) to allow for notification of share
- move, server failover, and server adapter changes. And also improve other
- failover scenarios, e.g. when client knows multiple DFS entries point to
- different servers, and the server we are connected to has gone down.
-
-q) Allow mount.cifs to be more verbose in reporting errors with dialect
-or unsupported feature errors.
-
-r) updating cifs documentation, and user guide.
-
-s) Addressing bugs found by running a broader set of xfstests in standard
-file system xfstest suite.
-
-t) split cifs and smb3 support into separate modules so legacy (and less
-secure) CIFS dialect can be disabled in environments that don't need it
-and simplify the code.
-
-v) POSIX Extensions for SMB3.1.1 (started, create and mkdir support added
-so far).
-
-KNOWN BUGS
-====================================
-See http://bugzilla.samba.org - search on product "CifsVFS" for
-current bug list. Also check http://bugzilla.kernel.org (Product = File System, Component = CIFS)
-
-1) existing symbolic links (Windows reparse points) are recognized but
-can not be created remotely. They are implemented for Samba and those that
-support the CIFS Unix extensions, although earlier versions of Samba
-overly restrict the pathnames.
-2) follow_link and readdir code does not follow dfs junctions
-but recognizes them
-
-Misc testing to do
-==================
-1) check out max path names and max path name components against various server
-types. Try nested symlinks (8 deep). Return max path name in stat -f information
-
-2) Improve xfstest's cifs/smb3 enablement and adapt xfstests where needed to test
-cifs/smb3 better
-
-3) Additional performance testing and optimization using iozone and similar -
-there are some easy changes that can be done to parallelize sequential writes,
-and when signing is disabled to request larger read sizes (larger than
-negotiated size) and send larger write sizes to modern servers.
-
-4) More exhaustively test against less common servers
diff --git a/Documentation/filesystems/cifs/cifs.txt b/Documentation/filesystems/cifs/cifs.txt
deleted file mode 100644
index 6775660..0000000
--- a/Documentation/filesystems/cifs/cifs.txt
+++ /dev/null
@@ -1,35 +0,0 @@
- This is the client VFS module for the SMB3 NAS protocol as well
- older dialects such as the Common Internet File System (CIFS)
- protocol which was the successor to the Server Message Block
- (SMB) protocol, the native file sharing mechanism for most early
- PC operating systems. New and improved versions of CIFS are now
- called SMB2 and SMB3. These dialects are also supported by the
- CIFS VFS module. CIFS is fully supported by network
- file servers such as Windows 2000, 2003, 2008, 2012 and 2016
- as well by Samba (which provides excellent CIFS
- server support for Linux and many other operating systems), Apple
- systems, as well as most Network Attached Storage vendors, so
- this network filesystem client can mount to a wide variety of
- servers.
-
- The intent of this module is to provide the most advanced network
- file system function for SMB3 compliant servers, including advanced
- security features, excellent parallelized high performance i/o, better
- POSIX compliance, secure per-user session establishment, encryption,
- high performance safe distributed caching (leases/oplocks), optional packet
- signing, large files, Unicode support and other internationalization
- improvements. Since both Samba server and this filesystem client support
- the CIFS Unix extensions (and in the future SMB3 POSIX extensions),
- the combination can provide a reasonable alternative to other network and
- cluster file systems for fileserving in some Linux to Linux environments,
- not just in Linux to Windows (or Linux to Mac) environments.
-
- This filesystem has an mount utility (mount.cifs) that can be obtained from
-
- https://ftp.samba.org/pub/linux-cifs/cifs-utils/
-
- It must be installed in the directory with the other mount helpers.
-
- For more information on the module see the project wiki page at
-
- https://wiki.samba.org/index.php/LinuxCIFS_utils
diff --git a/Documentation/filesystems/cifs/cifsroot.txt b/Documentation/filesystems/cifs/cifsroot.txt
new file mode 100644
index 0000000..0fa1a2c
--- /dev/null
+++ b/Documentation/filesystems/cifs/cifsroot.txt
@@ -0,0 +1,97 @@
+Mounting root file system via SMB (cifs.ko)
+===========================================
+
+Written 2019 by Paulo Alcantara <palcantara@suse.de>
+Written 2019 by Aurelien Aptel <aaptel@suse.com>
+
+The CONFIG_CIFS_ROOT option enables experimental root file system
+support over the SMB protocol via cifs.ko.
+
+It introduces a new kernel command-line option called 'cifsroot='
+which will tell the kernel to mount the root file system over the
+network by utilizing SMB or CIFS protocol.
+
+In order to mount, the network stack will also need to be set up by
+using 'ip=' config option. For more details, see
+Documentation/filesystems/nfs/nfsroot.txt.
+
+A CIFS root mount currently requires the use of SMB1+UNIX Extensions
+which is only supported by the Samba server. SMB1 is the older
+deprecated version of the protocol but it has been extended to support
+POSIX features (See [1]). The equivalent extensions for the newer
+recommended version of the protocol (SMB3) have not been fully
+implemented yet which means SMB3 doesn't support some required POSIX
+file system objects (e.g. block devices, pipes, sockets).
+
+As a result, a CIFS root will default to SMB1 for now but the version
+to use can nonetheless be changed via the 'vers=' mount option. This
+default will change once the SMB3 POSIX extensions are fully
+implemented.
+
+Server configuration
+====================
+
+To enable SMB1+UNIX extensions you will need to set these global
+settings in Samba smb.conf:
+
+ [global]
+ server min protocol = NT1
+ unix extension = yes # default
+
+Kernel command line
+===================
+
+root=/dev/cifs
+
+This is just a virtual device that basically tells the kernel to mount
+the root file system via SMB protocol.
+
+cifsroot=//<server-ip>/<share>[,options]
+
+Enables the kernel to mount the root file system via SMB that are
+located in the <server-ip> and <share> specified in this option.
+
+The default mount options are set in fs/cifs/cifsroot.c.
+
+server-ip
+ IPv4 address of the server.
+
+share
+ Path to SMB share (rootfs).
+
+options
+ Optional mount options. For more information, see mount.cifs(8).
+
+Examples
+========
+
+Export root file system as a Samba share in smb.conf file.
+
+...
+[linux]
+ path = /path/to/rootfs
+ read only = no
+ guest ok = yes
+ force user = root
+ force group = root
+ browseable = yes
+ writeable = yes
+ admin users = root
+ public = yes
+ create mask = 0777
+ directory mask = 0777
+...
+
+Restart smb service.
+
+# systemctl restart smb
+
+Test it under QEMU on a kernel built with CONFIG_CIFS_ROOT and
+CONFIG_IP_PNP options enabled.
+
+# qemu-system-x86_64 -enable-kvm -cpu host -m 1024 \
+ -kernel /path/to/linux/arch/x86/boot/bzImage -nographic \
+ -append "root=/dev/cifs rw ip=dhcp cifsroot=//10.0.2.2/linux,username=foo,password=bar console=ttyS0 3"
+
+
+1: https://wiki.samba.org/index.php/UNIX_Extensions
diff --git a/Documentation/filesystems/cifs/winucase_convert.pl b/Documentation/filesystems/cifs/winucase_convert.pl
deleted file mode 100755
index 322a9c8..0000000
--- a/Documentation/filesystems/cifs/winucase_convert.pl
+++ /dev/null
@@ -1,62 +0,0 @@
-#!/usr/bin/perl -w
-#
-# winucase_convert.pl -- convert "Windows 8 Upper Case Mapping Table.txt" to
-# a two-level set of C arrays.
-#
-# Copyright 2013: Jeff Layton <jlayton@redhat.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-#
-
-while(<>) {
- next if (!/^0x(..)(..)\t0x(....)\t/);
- $firstchar = hex($1);
- $secondchar = hex($2);
- $uppercase = hex($3);
-
- $top[$firstchar][$secondchar] = $uppercase;
-}
-
-for ($i = 0; $i < 256; $i++) {
- next if (!$top[$i]);
-
- printf("static const wchar_t t2_%2.2x[256] = {", $i);
- for ($j = 0; $j < 256; $j++) {
- if (($j % 8) == 0) {
- print "\n\t";
- } else {
- print " ";
- }
- printf("0x%4.4x,", $top[$i][$j] ? $top[$i][$j] : 0);
- }
- print "\n};\n\n";
-}
-
-printf("static const wchar_t *const toplevel[256] = {", $i);
-for ($i = 0; $i < 256; $i++) {
- if (($i % 8) == 0) {
- print "\n\t";
- } elsif ($top[$i]) {
- print " ";
- } else {
- print " ";
- }
-
- if ($top[$i]) {
- printf("t2_%2.2x,", $i);
- } else {
- print "NULL,";
- }
-}
-print "\n};\n\n";
diff --git a/Documentation/filesystems/coda.txt b/Documentation/filesystems/coda.txt
index 6131135..1711ad4 100644
--- a/Documentation/filesystems/coda.txt
+++ b/Documentation/filesystems/coda.txt
@@ -421,14 +421,14 @@
The CodaCred structure defines a variety of user and group ids as
- they are set for the calling process. The vuid_t and guid_t are 32 bit
+ they are set for the calling process. The vuid_t and vgid_t are 32 bit
unsigned integers. It also defines group membership in an array. On
Unix the CodaCred has proven sufficient to implement good security
semantics for Coda but the structure may have to undergo modification
for the Windows environment when these mature.
struct CodaCred {
- vuid_t cr_uid, cr_euid, cr_suid, cr_fsuid; /* Real, effective, set, fs uid*/
+ vuid_t cr_uid, cr_euid, cr_suid, cr_fsuid; /* Real, effective, set, fs uid */
vgid_t cr_gid, cr_egid, cr_sgid, cr_fsgid; /* same for groups */
vgid_t cr_groups[NGROUPS]; /* Group membership for caller */
};
@@ -481,7 +481,10 @@
-
+ struct coda_timespec {
+ int64_t tv_sec; /* seconds */
+ long tv_nsec; /* nanoseconds */
+ };
struct coda_vattr {
enum coda_vtype va_type; /* vnode type (for create) */
@@ -493,9 +496,9 @@
long va_fileid; /* file id */
u_quad_t va_size; /* file size in bytes */
long va_blocksize; /* blocksize preferred for i/o */
- struct timespec va_atime; /* time of last access */
- struct timespec va_mtime; /* time of last modification */
- struct timespec va_ctime; /* time file changed */
+ struct coda_timespec va_atime; /* time of last access */
+ struct coda_timespec va_mtime; /* time of last modification */
+ struct coda_timespec va_ctime; /* time file changed */
u_long va_gen; /* generation number of file */
u_long va_flags; /* flags defined for file */
dev_t va_rdev; /* device special file represents */
diff --git a/Documentation/filesystems/conf.py b/Documentation/filesystems/conf.py
deleted file mode 100644
index ea44172..0000000
--- a/Documentation/filesystems/conf.py
+++ /dev/null
@@ -1,10 +0,0 @@
-# -*- coding: utf-8; mode: python -*-
-
-project = "Linux Filesystems API"
-
-tags.add("subproject")
-
-latex_documents = [
- ('index', 'filesystems.tex', project,
- 'The kernel development community', 'manual'),
-]
diff --git a/Documentation/filesystems/configfs/configfs.txt b/Documentation/filesystems/configfs/configfs.txt
index 3828e85..16e606c 100644
--- a/Documentation/filesystems/configfs/configfs.txt
+++ b/Documentation/filesystems/configfs/configfs.txt
@@ -216,7 +216,7 @@
[struct configfs_bin_attribute]
- struct configfs_attribute {
+ struct configfs_bin_attribute {
struct configfs_attribute cb_attr;
void *cb_private;
size_t cb_max_size;
diff --git a/Documentation/filesystems/dax.txt b/Documentation/filesystems/dax.txt
index 70cb68b..6797294 100644
--- a/Documentation/filesystems/dax.txt
+++ b/Documentation/filesystems/dax.txt
@@ -75,8 +75,8 @@
These filesystems may be used for inspiration:
- ext2: see Documentation/filesystems/ext2.txt
-- ext4: see Documentation/filesystems/ext4.txt
-- xfs: see Documentation/filesystems/xfs.txt
+- ext4: see Documentation/filesystems/ext4/
+- xfs: see Documentation/admin-guide/xfs.rst
Handling Media Errors
diff --git a/Documentation/filesystems/debugfs.txt b/Documentation/filesystems/debugfs.txt
index 4f45f71..9e27c84 100644
--- a/Documentation/filesystems/debugfs.txt
+++ b/Documentation/filesystems/debugfs.txt
@@ -31,10 +31,10 @@
indicated parent directory. If parent is NULL, the directory will be
created in the debugfs root. On success, the return value is a struct
dentry pointer which can be used to create files in the directory (and to
-clean it up at the end). A NULL return value indicates that something went
-wrong. If ERR_PTR(-ENODEV) is returned, that is an indication that the
-kernel has been built without debugfs support and none of the functions
-described below will work.
+clean it up at the end). An ERR_PTR(-ERROR) return value indicates that
+something went wrong. If ERR_PTR(-ENODEV) is returned, that is an
+indication that the kernel has been built without debugfs support and none
+of the functions described below will work.
The most general way to create a file within a debugfs directory is with:
@@ -48,8 +48,9 @@
resulting inode structure, and fops is a set of file operations which
implement the file's behavior. At a minimum, the read() and/or write()
operations should be provided; others can be included as needed. Again,
-the return value will be a dentry pointer to the created file, NULL for
-error, or ERR_PTR(-ENODEV) if debugfs support is missing.
+the return value will be a dentry pointer to the created file,
+ERR_PTR(-ERROR) on error, or ERR_PTR(-ENODEV) if debugfs support is
+missing.
Create a file with an initial size, the following function can be used
instead:
@@ -168,7 +169,7 @@
If you want to dump an u32 array in debugfs, you can create file with:
- struct dentry *debugfs_create_u32_array(const char *name, umode_t mode,
+ void debugfs_create_u32_array(const char *name, umode_t mode,
struct dentry *parent,
u32 *array, u32 elements);
@@ -214,7 +215,8 @@
void debugfs_remove(struct dentry *dentry);
-The dentry value can be NULL, in which case nothing will be removed.
+The dentry value can be NULL or an error value, in which case nothing will
+be removed.
Once upon a time, debugfs users were required to remember the dentry
pointer for every debugfs file they created so that all files could be
diff --git a/Documentation/filesystems/directory-locking b/Documentation/filesystems/directory-locking.rst
similarity index 85%
rename from Documentation/filesystems/directory-locking
rename to Documentation/filesystems/directory-locking.rst
index 4e32cb9..de12016 100644
--- a/Documentation/filesystems/directory-locking
+++ b/Documentation/filesystems/directory-locking.rst
@@ -1,12 +1,17 @@
- Locking scheme used for directory operations is based on two
+=================
+Directory Locking
+=================
+
+
+Locking scheme used for directory operations is based on two
kinds of locks - per-inode (->i_rwsem) and per-filesystem
(->s_vfs_rename_mutex).
- When taking the i_rwsem on multiple non-directory objects, we
+When taking the i_rwsem on multiple non-directory objects, we
always acquire the locks in order by increasing address. We'll call
that "inode pointer" order in the following.
- For our purposes all operations fall in 5 classes:
+For our purposes all operations fall in 5 classes:
1) read access. Locking rules: caller locks directory we are accessing.
The lock is taken shared.
@@ -27,25 +32,29 @@
case) shared.
5) link creation. Locking rules:
+
* lock parent
* check that source is not a directory
* lock source
* call the method.
+
All locks are exclusive.
6) cross-directory rename. The trickiest in the whole bunch. Locking
rules:
+
* lock the filesystem
* lock parents in "ancestors first" order.
* find source and target.
* if old parent is equal to or is a descendent of target
- fail with -ENOTEMPTY
+ fail with -ENOTEMPTY
* if new parent is equal to or is a descendent of source
- fail with -ELOOP
+ fail with -ELOOP
* If it's an exchange, lock both the source and the target.
* If the target exists, lock it. If the source is a non-directory,
lock it. If we need to lock both, do so in inode pointer order.
* call the method.
+
All ->i_rwsem are taken exclusive. Again, we might get away with locking
the the source (and target in exchange case) shared.
@@ -54,10 +63,11 @@
If no directory is its own ancestor, the scheme above is deadlock-free.
+
Proof:
First of all, at any moment we have a partial ordering of the
-objects - A < B iff A is an ancestor of B.
+ objects - A < B iff A is an ancestor of B.
That ordering can change. However, the following is true:
@@ -77,32 +87,32 @@
non-directory object, except renames, which take locks on source and
target in inode pointer order in the case they are not directories.)
- Now consider the minimal deadlock. Each process is blocked on
+Now consider the minimal deadlock. Each process is blocked on
attempt to acquire some lock and already holds at least one lock. Let's
consider the set of contended locks. First of all, filesystem lock is
not contended, since any process blocked on it is not holding any locks.
Thus all processes are blocked on ->i_rwsem.
- By (3), any process holding a non-directory lock can only be
+By (3), any process holding a non-directory lock can only be
waiting on another non-directory lock with a larger address. Therefore
the process holding the "largest" such lock can always make progress, and
non-directory objects are not included in the set of contended locks.
- Thus link creation can't be a part of deadlock - it can't be
+Thus link creation can't be a part of deadlock - it can't be
blocked on source and it means that it doesn't hold any locks.
- Any contended object is either held by cross-directory rename or
+Any contended object is either held by cross-directory rename or
has a child that is also contended. Indeed, suppose that it is held by
operation other than cross-directory rename. Then the lock this operation
is blocked on belongs to child of that object due to (1).
- It means that one of the operations is cross-directory rename.
+It means that one of the operations is cross-directory rename.
Otherwise the set of contended objects would be infinite - each of them
would have a contended child and we had assumed that no object is its
own descendent. Moreover, there is exactly one cross-directory rename
(see above).
- Consider the object blocking the cross-directory rename. One
+Consider the object blocking the cross-directory rename. One
of its descendents is locked by cross-directory rename (otherwise we
would again have an infinite set of contended objects). But that
means that cross-directory rename is taking locks out of order. Due
@@ -112,7 +122,7 @@
Contradiction. I.e. deadlock is impossible. Q.E.D.
- These operations are guaranteed to avoid loop creation. Indeed,
+These operations are guaranteed to avoid loop creation. Indeed,
the only operation that could introduce loops is cross-directory rename.
Since the only new (parent, child) pair added by rename() is (new parent,
source), such loop would have to contain these objects and the rest of it
@@ -123,13 +133,13 @@
we had acquired filesystem lock and rename() would fail with -ELOOP in that
case.
- While this locking scheme works for arbitrary DAGs, it relies on
+While this locking scheme works for arbitrary DAGs, it relies on
ability to check that directory is a descendent of another object. Current
implementation assumes that directory graph is a tree. This assumption is
also preserved by all operations (cross-directory rename on a tree that would
not introduce a cycle will leave it a tree and link() fails for directories).
- Notice that "directory" in the above == "anything that might have
+Notice that "directory" in the above == "anything that might have
children", so if we are going to introduce hybrid objects we will need
either to make sure that link(2) doesn't work for them or to make changes
in is_subdir() that would make it work even in presence of such beasts.
diff --git a/Documentation/filesystems/erofs.txt b/Documentation/filesystems/erofs.txt
new file mode 100644
index 0000000..b0c0853
--- /dev/null
+++ b/Documentation/filesystems/erofs.txt
@@ -0,0 +1,210 @@
+Overview
+========
+
+EROFS file-system stands for Enhanced Read-Only File System. Different
+from other read-only file systems, it aims to be designed for flexibility,
+scalability, but be kept simple and high performance.
+
+It is designed as a better filesystem solution for the following scenarios:
+ - read-only storage media or
+
+ - part of a fully trusted read-only solution, which means it needs to be
+ immutable and bit-for-bit identical to the official golden image for
+ their releases due to security and other considerations and
+
+ - hope to save some extra storage space with guaranteed end-to-end performance
+ by using reduced metadata and transparent file compression, especially
+ for those embedded devices with limited memory (ex, smartphone);
+
+Here is the main features of EROFS:
+ - Little endian on-disk design;
+
+ - Currently 4KB block size (nobh) and therefore maximum 16TB address space;
+
+ - Metadata & data could be mixed by design;
+
+ - 2 inode versions for different requirements:
+ v1 v2
+ Inode metadata size: 32 bytes 64 bytes
+ Max file size: 4 GB 16 EB (also limited by max. vol size)
+ Max uids/gids: 65536 4294967296
+ File creation time: no yes (64 + 32-bit timestamp)
+ Max hardlinks: 65536 4294967296
+ Metadata reserved: 4 bytes 14 bytes
+
+ - Support extended attributes (xattrs) as an option;
+
+ - Support xattr inline and tail-end data inline for all files;
+
+ - Support POSIX.1e ACLs by using xattrs;
+
+ - Support transparent file compression as an option:
+ LZ4 algorithm with 4 KB fixed-output compression for high performance;
+
+The following git tree provides the file system user-space tools under
+development (ex, formatting tool mkfs.erofs):
+>> git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git
+
+Bugs and patches are welcome, please kindly help us and send to the following
+linux-erofs mailing list:
+>> linux-erofs mailing list <linux-erofs@lists.ozlabs.org>
+
+Mount options
+=============
+
+(no)user_xattr Setup Extended User Attributes. Note: xattr is enabled
+ by default if CONFIG_EROFS_FS_XATTR is selected.
+(no)acl Setup POSIX Access Control List. Note: acl is enabled
+ by default if CONFIG_EROFS_FS_POSIX_ACL is selected.
+cache_strategy=%s Select a strategy for cached decompression from now on:
+ disabled: In-place I/O decompression only;
+ readahead: Cache the last incomplete compressed physical
+ cluster for further reading. It still does
+ in-place I/O decompression for the rest
+ compressed physical clusters;
+ readaround: Cache the both ends of incomplete compressed
+ physical clusters for further reading.
+ It still does in-place I/O decompression
+ for the rest compressed physical clusters.
+
+On-disk details
+===============
+
+Summary
+-------
+Different from other read-only file systems, an EROFS volume is designed
+to be as simple as possible:
+
+ |-> aligned with the block size
+ ____________________________________________________________
+ | |SB| | ... | Metadata | ... | Data | Metadata | ... | Data |
+ |_|__|_|_____|__________|_____|______|__________|_____|______|
+ 0 +1K
+
+All data areas should be aligned with the block size, but metadata areas
+may not. All metadatas can be now observed in two different spaces (views):
+ 1. Inode metadata space
+ Each valid inode should be aligned with an inode slot, which is a fixed
+ value (32 bytes) and designed to be kept in line with v1 inode size.
+
+ Each inode can be directly found with the following formula:
+ inode offset = meta_blkaddr * block_size + 32 * nid
+
+ |-> aligned with 8B
+ |-> followed closely
+ + meta_blkaddr blocks |-> another slot
+ _____________________________________________________________________
+ | ... | inode | xattrs | extents | data inline | ... | inode ...
+ |________|_______|(optional)|(optional)|__(optional)_|_____|__________
+ |-> aligned with the inode slot size
+ . .
+ . .
+ . .
+ . .
+ . .
+ . .
+ .____________________________________________________|-> aligned with 4B
+ | xattr_ibody_header | shared xattrs | inline xattrs |
+ |____________________|_______________|_______________|
+ |-> 12 bytes <-|->x * 4 bytes<-| .
+ . . .
+ . . .
+ . . .
+ ._______________________________.______________________.
+ | id | id | id | id | ... | id | ent | ... | ent| ... |
+ |____|____|____|____|______|____|_____|_____|____|_____|
+ |-> aligned with 4B
+ |-> aligned with 4B
+
+ Inode could be 32 or 64 bytes, which can be distinguished from a common
+ field which all inode versions have -- i_advise:
+
+ __________________ __________________
+ | i_advise | | i_advise |
+ |__________________| |__________________|
+ | ... | | ... |
+ | | | |
+ |__________________| 32 bytes | |
+ | |
+ |__________________| 64 bytes
+
+ Xattrs, extents, data inline are followed by the corresponding inode with
+ proper alignes, and they could be optional for different data mappings,
+ _currently_ there are totally 3 valid data mappings supported:
+
+ 1) flat file data without data inline (no extent);
+ 2) fixed-output size data compression (must have extents);
+ 3) flat file data with tail-end data inline (no extent);
+
+ The size of the optional xattrs is indicated by i_xattr_count in inode
+ header. Large xattrs or xattrs shared by many different files can be
+ stored in shared xattrs metadata rather than inlined right after inode.
+
+ 2. Shared xattrs metadata space
+ Shared xattrs space is similar to the above inode space, started with
+ a specific block indicated by xattr_blkaddr, organized one by one with
+ proper align.
+
+ Each share xattr can also be directly found by the following formula:
+ xattr offset = xattr_blkaddr * block_size + 4 * xattr_id
+
+ |-> aligned by 4 bytes
+ + xattr_blkaddr blocks |-> aligned with 4 bytes
+ _________________________________________________________________________
+ | ... | xattr_entry | xattr data | ... | xattr_entry | xattr data ...
+ |________|_____________|_____________|_____|______________|_______________
+
+Directories
+-----------
+All directories are now organized in a compact on-disk format. Note that
+each directory block is divided into index and name areas in order to support
+random file lookup, and all directory entries are _strictly_ recorded in
+alphabetical order in order to support improved prefix binary search
+algorithm (could refer to the related source code).
+
+ ___________________________
+ / |
+ / ______________|________________
+ / / | nameoff1 | nameoffN-1
+ ____________.______________._______________v________________v__________
+| dirent | dirent | ... | dirent | filename | filename | ... | filename |
+|___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
+ \ ^
+ \ | * could have
+ \ | trailing '\0'
+ \________________________| nameoff0
+
+ Directory block
+
+Note that apart from the offset of the first filename, nameoff0 also indicates
+the total number of directory entries in this block since it is no need to
+introduce another on-disk field at all.
+
+Compression
+-----------
+Currently, EROFS supports 4KB fixed-output clustersize transparent file
+compression, as illustrated below:
+
+ |---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
+ clusterofs clusterofs clusterofs
+ | | | logical data
+_________v_______________________________v_____________________v_______________
+... | . | | . | | . | ...
+____|____.________|_____________|________.____|_____________|__.__________|____
+ |-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|
+ size size size size size
+ . . . .
+ . . . .
+ . . . .
+ _______._____________._____________._____________._____________________
+ ... | | | | ... physical data
+ _______|_____________|_____________|_____________|_____________________
+ |-> cluster <-|-> cluster <-|-> cluster <-|
+ size size size
+
+Currently each on-disk physical cluster can contain 4KB (un)compressed data
+at most. For each logical cluster, there is a corresponding on-disk index to
+describe its cluster type, physical cluster address, etc.
+
+See "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.
+
diff --git a/Documentation/filesystems/exofs.txt b/Documentation/filesystems/exofs.txt
deleted file mode 100644
index 23583a1..0000000
--- a/Documentation/filesystems/exofs.txt
+++ /dev/null
@@ -1,185 +0,0 @@
-===============================================================================
-WHAT IS EXOFS?
-===============================================================================
-
-exofs is a file system that uses an OSD and exports the API of a normal Linux
-file system. Users access exofs like any other local file system, and exofs
-will in turn issue commands to the local OSD initiator.
-
-OSD is a new T10 command set that views storage devices not as a large/flat
-array of sectors but as a container of objects, each having a length, quota,
-time attributes and more. Each object is addressed by a 64bit ID, and is
-contained in a 64bit ID partition. Each object has associated attributes
-attached to it, which are integral part of the object and provide metadata about
-the object. The standard defines some common obligatory attributes, but user
-attributes can be added as needed.
-
-===============================================================================
-ENVIRONMENT
-===============================================================================
-
-To use this file system, you need to have an object store to run it on. You
-may download a target from:
-http://open-osd.org
-
-See Documentation/scsi/osd.txt for how to setup a working osd environment.
-
-===============================================================================
-USAGE
-===============================================================================
-
-1. Download and compile exofs and open-osd initiator:
- You need an external Kernel source tree or kernel headers from your
- distribution. (anything based on 2.6.26 or later).
-
- a. download open-osd including exofs source using:
- [parent-directory]$ git clone git://git.open-osd.org/open-osd.git
-
- b. Build the library module like this:
- [parent-directory]$ make -C KSRC=$(KER_DIR) open-osd
-
- This will build both the open-osd initiator as well as the exofs kernel
- module. Use whatever parameters you compiled your Kernel with and
- $(KER_DIR) above pointing to the Kernel you compile against. See the file
- open-osd/top-level-Makefile for an example.
-
-2. Get the OSD initiator and target set up properly, and login to the target.
- See Documentation/scsi/osd.txt for farther instructions. Also see ./do-osd
- for example script that does all these steps.
-
-3. Insmod the exofs.ko module:
- [exofs]$ insmod exofs.ko
-
-4. Make sure the directory where you want to mount exists. If not, create it.
- (For example, mkdir /mnt/exofs)
-
-5. At first run you will need to invoke the mkfs.exofs application
-
- As an example, this will create the file system on:
- /dev/osd0 partition ID 65536
-
- mkfs.exofs --pid=65536 --format /dev/osd0
-
- The --format is optional. If not specified, no OSD_FORMAT will be
- performed and a clean file system will be created in the specified pid,
- in the available space of the target. (Use --format=size_in_meg to limit
- the total LUN space available)
-
- If pid already exists, it will be deleted and a new one will be created in
- its place. Be careful.
-
- An exofs lives inside a single OSD partition. You can create multiple exofs
- filesystems on the same device using multiple pids.
-
- (run mkfs.exofs without any parameters for usage help message)
-
-6. Mount the file system.
-
- For example, to mount /dev/osd0, partition ID 0x10000 on /mnt/exofs:
-
- mount -t exofs -o pid=65536 /dev/osd0 /mnt/exofs/
-
-7. For reference (See do-exofs example script):
- do-exofs start - an example of how to perform the above steps.
- do-exofs stop - an example of how to unmount the file system.
- do-exofs format - an example of how to format and mkfs a new exofs.
-
-8. Extra compilation flags (uncomment in fs/exofs/Kbuild):
- CONFIG_EXOFS_DEBUG - for debug messages and extra checks.
-
-===============================================================================
-exofs mount options
-===============================================================================
-Similar to any mount command:
- mount -t exofs -o exofs_options /dev/osdX mount_exofs_directory
-
-Where:
- -t exofs: specifies the exofs file system
-
- /dev/osdX: X is a decimal number. /dev/osdX was created after a successful
- login into an OSD target.
-
- mount_exofs_directory: The directory to mount the file system on
-
- exofs specific options: Options are separated by commas (,)
- pid=<integer> - The partition number to mount/create as
- container of the filesystem.
- This option is mandatory. integer can be
- Hex by pre-pending an 0x to the number.
- osdname=<id> - Mount by a device's osdname.
- osdname is usually a 36 character uuid of the
- form "d2683732-c906-4ee1-9dbd-c10c27bb40df".
- It is one of the device's uuid specified in the
- mkfs.exofs format command.
- If this option is specified then the /dev/osdX
- above can be empty and is ignored.
- to=<integer> - Timeout in ticks for a single command.
- default is (60 * HZ) [for debugging only]
-
-===============================================================================
-DESIGN
-===============================================================================
-
-* The file system control block (AKA on-disk superblock) resides in an object
- with a special ID (defined in common.h).
- Information included in the file system control block is used to fill the
- in-memory superblock structure at mount time. This object is created before
- the file system is used by mkexofs.c. It contains information such as:
- - The file system's magic number
- - The next inode number to be allocated
-
-* Each file resides in its own object and contains the data (and it will be
- possible to extend the file over multiple objects, though this has not been
- implemented yet).
-
-* A directory is treated as a file, and essentially contains a list of <file
- name, inode #> pairs for files that are found in that directory. The object
- IDs correspond to the files' inode numbers and will be allocated according to
- a bitmap (stored in a separate object). Now they are allocated using a
- counter.
-
-* Each file's control block (AKA on-disk inode) is stored in its object's
- attributes. This applies to both regular files and other types (directories,
- device files, symlinks, etc.).
-
-* Credentials are generated per object (inode and superblock) when they are
- created in memory (read from disk or created). The credential works for all
- operations and is used as long as the object remains in memory.
-
-* Async OSD operations are used whenever possible, but the target may execute
- them out of order. The operations that concern us are create, delete,
- readpage, writepage, update_inode, and truncate. The following pairs of
- operations should execute in the order written, and we need to prevent them
- from executing in reverse order:
- - The following are handled with the OBJ_CREATED and OBJ_2BCREATED
- flags. OBJ_CREATED is set when we know the object exists on the OSD -
- in create's callback function, and when we successfully do a
- read_inode.
- OBJ_2BCREATED is set in the beginning of the create function, so we
- know that we should wait.
- - create/delete: delete should wait until the object is created
- on the OSD.
- - create/readpage: readpage should be able to return a page
- full of zeroes in this case. If there was a write already
- en-route (i.e. create, writepage, readpage) then the page
- would be locked, and so it would really be the same as
- create/writepage.
- - create/writepage: if writepage is called for a sync write, it
- should wait until the object is created on the OSD.
- Otherwise, it should just return.
- - create/truncate: truncate should wait until the object is
- created on the OSD.
- - create/update_inode: update_inode should wait until the
- object is created on the OSD.
- - Handled by VFS locks:
- - readpage/delete: shouldn't happen because of page lock.
- - writepage/delete: shouldn't happen because of page lock.
- - readpage/writepage: shouldn't happen because of page lock.
-
-===============================================================================
-LICENSE/COPYRIGHT
-===============================================================================
-The exofs file system is based on ext2 v0.5b (distributed with the Linux kernel
-version 2.6.10). All files include the original copyrights, and the license
-is GPL version 2 (only version 2, as is true for the Linux kernel). The
-Linux kernel can be downloaded from www.kernel.org.
diff --git a/Documentation/filesystems/ext2.txt b/Documentation/filesystems/ext2.txt
index 81c0bec..94c2cf0 100644
--- a/Documentation/filesystems/ext2.txt
+++ b/Documentation/filesystems/ext2.txt
@@ -57,7 +57,13 @@
nobh Do not attach buffer_heads to file pagecache.
-grpquota,noquota,quota,usrquota Quota options are silently ignored by ext2.
+quota, usrquota Enable user disk quota support
+ (requires CONFIG_QUOTA).
+
+grpquota Enable group disk quota support
+ (requires CONFIG_QUOTA).
+
+noquota option ls silently ignored by ext2.
Specification
@@ -358,7 +364,7 @@
the time of the crash, then there is no guarantee of consistency for
the blocks in that transaction so they are discarded (which means any
filesystem changes they represent are also lost).
-Check Documentation/filesystems/ext4.txt if you want to read more about
+Check Documentation/filesystems/ext4/ if you want to read more about
ext4 and journaling.
References
diff --git a/Documentation/filesystems/ext4/ondisk/about.rst b/Documentation/filesystems/ext4/about.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/about.rst
rename to Documentation/filesystems/ext4/about.rst
diff --git a/Documentation/filesystems/ext4/ondisk/allocators.rst b/Documentation/filesystems/ext4/allocators.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/allocators.rst
rename to Documentation/filesystems/ext4/allocators.rst
diff --git a/Documentation/filesystems/ext4/ondisk/attributes.rst b/Documentation/filesystems/ext4/attributes.rst
similarity index 98%
rename from Documentation/filesystems/ext4/ondisk/attributes.rst
rename to Documentation/filesystems/ext4/attributes.rst
index 0b01b67..54386a0 100644
--- a/Documentation/filesystems/ext4/ondisk/attributes.rst
+++ b/Documentation/filesystems/ext4/attributes.rst
@@ -30,7 +30,7 @@
``ext4_xattr_ibody_header`` that is 4 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -47,7 +47,7 @@
``struct ext4_xattr_header``, which is 32 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -92,7 +92,7 @@
Attributes stored inside an inode do not need be stored in sorted order.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -157,7 +157,7 @@
the key name. Here is a map of name index values to key prefixes:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Name Index
diff --git a/Documentation/filesystems/ext4/bigalloc.rst b/Documentation/filesystems/ext4/bigalloc.rst
new file mode 100644
index 0000000..72075aa
--- /dev/null
+++ b/Documentation/filesystems/ext4/bigalloc.rst
@@ -0,0 +1,34 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Bigalloc
+--------
+
+At the moment, the default size of a block is 4KiB, which is a commonly
+supported page size on most MMU-capable hardware. This is fortunate, as
+ext4 code is not prepared to handle the case where the block size
+exceeds the page size. However, for a filesystem of mostly huge files,
+it is desirable to be able to allocate disk blocks in units of multiple
+blocks to reduce both fragmentation and metadata overhead. The
+bigalloc feature provides exactly this ability.
+
+The bigalloc feature (EXT4_FEATURE_RO_COMPAT_BIGALLOC) changes ext4 to
+use clustered allocation, so that each bit in the ext4 block allocation
+bitmap addresses a power of two number of blocks. For example, if the
+file system is mainly going to be storing large files in the 4-32
+megabyte range, it might make sense to set a cluster size of 1 megabyte.
+This means that each bit in the block allocation bitmap now addresses
+256 4k blocks. This shrinks the total size of the block allocation
+bitmaps for a 2T file system from 64 megabytes to 256 kilobytes. It also
+means that a block group addresses 32 gigabytes instead of 128 megabytes,
+also shrinking the amount of file system overhead for metadata.
+
+The administrator can set a block cluster size at mkfs time (which is
+stored in the s\_log\_cluster\_size field in the superblock); from then
+on, the block bitmaps track clusters, not individual blocks. This means
+that block groups can be several gigabytes in size (instead of just
+128MiB); however, the minimum allocation unit becomes a cluster, not a
+block, even for directories. TaoBao had a patchset to extend the “use
+units of clusters instead of blocks” to the extent tree, though it is
+not clear where those patches went-- they eventually morphed into
+“extent tree v2” but that code has not landed as of May 2015.
+
diff --git a/Documentation/filesystems/ext4/ondisk/bitmaps.rst b/Documentation/filesystems/ext4/bitmaps.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/bitmaps.rst
rename to Documentation/filesystems/ext4/bitmaps.rst
diff --git a/Documentation/filesystems/ext4/ondisk/blockgroup.rst b/Documentation/filesystems/ext4/blockgroup.rst
similarity index 94%
rename from Documentation/filesystems/ext4/ondisk/blockgroup.rst
rename to Documentation/filesystems/ext4/blockgroup.rst
index baf888e..3da1566 100644
--- a/Documentation/filesystems/ext4/ondisk/blockgroup.rst
+++ b/Documentation/filesystems/ext4/blockgroup.rst
@@ -71,11 +71,11 @@
superblock, group descriptors, data block bitmaps for groups 0-3, inode
bitmaps for groups 0-3, inode tables for groups 0-3, and the remaining
space in group 0 is for file data. The effect of this is to group the
-block metadata close together for faster loading, and to enable large
-files to be continuous on disk. Backup copies of the superblock and
-group descriptors are always at the beginning of block groups, even if
-flex\_bg is enabled. The number of block groups that make up a flex\_bg
-is given by 2 ^ ``sb.s_log_groups_per_flex``.
+block group metadata close together for faster loading, and to enable
+large files to be continuous on disk. Backup copies of the superblock
+and group descriptors are always at the beginning of block groups, even
+if flex\_bg is enabled. The number of block groups that make up a
+flex\_bg is given by 2 ^ ``sb.s_log_groups_per_flex``.
Meta Block Groups
-----------------
diff --git a/Documentation/filesystems/ext4/ondisk/blockmap.rst b/Documentation/filesystems/ext4/blockmap.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/blockmap.rst
rename to Documentation/filesystems/ext4/blockmap.rst
diff --git a/Documentation/filesystems/ext4/ondisk/blocks.rst b/Documentation/filesystems/ext4/blocks.rst
similarity index 93%
rename from Documentation/filesystems/ext4/ondisk/blocks.rst
rename to Documentation/filesystems/ext4/blocks.rst
index 73d4dc0..bd722ec 100644
--- a/Documentation/filesystems/ext4/ondisk/blocks.rst
+++ b/Documentation/filesystems/ext4/blocks.rst
@@ -10,7 +10,9 @@
4KiB. You may experience mounting problems if block size is greater than
page size (i.e. 64KiB blocks on a i386 which only has 4KiB memory
pages). By default a filesystem can contain 2^32 blocks; if the '64bit'
-feature is enabled, then a filesystem can have 2^64 blocks.
+feature is enabled, then a filesystem can have 2^64 blocks. The location
+of structures is stored in terms of the block number the structure lives
+in and not the absolute offset on disk.
For 32-bit filesystems, limits are as follows:
diff --git a/Documentation/filesystems/ext4/ondisk/checksums.rst b/Documentation/filesystems/ext4/checksums.rst
similarity index 99%
rename from Documentation/filesystems/ext4/ondisk/checksums.rst
rename to Documentation/filesystems/ext4/checksums.rst
index 9d6a793..5519e25 100644
--- a/Documentation/filesystems/ext4/ondisk/checksums.rst
+++ b/Documentation/filesystems/ext4/checksums.rst
@@ -28,7 +28,7 @@
(crc32c as of October 2013) unless noted otherwise.
.. list-table::
- :widths: 1 1 4
+ :widths: 20 8 50
:header-rows: 1
* - Metadata
diff --git a/Documentation/filesystems/ext4/ondisk/directory.rst b/Documentation/filesystems/ext4/directory.rst
similarity index 97%
rename from Documentation/filesystems/ext4/ondisk/directory.rst
rename to Documentation/filesystems/ext4/directory.rst
index 8fcba68..073940c 100644
--- a/Documentation/filesystems/ext4/ondisk/directory.rst
+++ b/Documentation/filesystems/ext4/directory.rst
@@ -34,7 +34,7 @@
``dirent.rec_len`` to know for sure.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -59,14 +59,14 @@
- File name.
Since file names cannot be longer than 255 bytes, the new directory
-entry format shortens the rec\_len field and uses the space for a file
+entry format shortens the name\_len field and uses the space for a file
type flag, probably to avoid having to load every inode during directory
tree traversal. This format is ``ext4_dir_entry_2``, which is at most
263 bytes long, though on disk you'll need to reference
``dirent.rec_len`` to know for sure.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -99,7 +99,7 @@
The directory file type is one of the following values:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -130,7 +130,7 @@
``struct ext4_dir_entry_tail``:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -212,7 +212,7 @@
of a data block:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -305,7 +305,7 @@
The directory hash is one of the following values:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -327,7 +327,7 @@
also the full length of a data block:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -375,7 +375,7 @@
long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -405,7 +405,7 @@
The dx\_tail structure is 8 bytes long and looks like this:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
diff --git a/Documentation/filesystems/ext4/ondisk/dynamic.rst b/Documentation/filesystems/ext4/dynamic.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/dynamic.rst
rename to Documentation/filesystems/ext4/dynamic.rst
diff --git a/Documentation/filesystems/ext4/ondisk/eainode.rst b/Documentation/filesystems/ext4/eainode.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/eainode.rst
rename to Documentation/filesystems/ext4/eainode.rst
diff --git a/Documentation/filesystems/ext4/ext4.rst b/Documentation/filesystems/ext4/ext4.rst
deleted file mode 100644
index 9d4368d..0000000
--- a/Documentation/filesystems/ext4/ext4.rst
+++ /dev/null
@@ -1,613 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0
-
-========================
-General Information
-========================
-
-Ext4 is an advanced level of the ext3 filesystem which incorporates
-scalability and reliability enhancements for supporting large filesystems
-(64 bit) in keeping with increasing disk capacities and state-of-the-art
-feature requirements.
-
-Mailing list: linux-ext4@vger.kernel.org
-Web site: http://ext4.wiki.kernel.org
-
-
-Quick usage instructions
-========================
-
-Note: More extensive information for getting started with ext4 can be
-found at the ext4 wiki site at the URL:
-http://ext4.wiki.kernel.org/index.php/Ext4_Howto
-
- - The latest version of e2fsprogs can be found at:
-
- https://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/
-
- or
-
- http://sourceforge.net/project/showfiles.php?group_id=2406
-
- or grab the latest git repository from:
-
- https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
-
- - Create a new filesystem using the ext4 filesystem type:
-
- # mke2fs -t ext4 /dev/hda1
-
- Or to configure an existing ext3 filesystem to support extents:
-
- # tune2fs -O extents /dev/hda1
-
- If the filesystem was created with 128 byte inodes, it can be
- converted to use 256 byte for greater efficiency via:
-
- # tune2fs -I 256 /dev/hda1
-
- - Mounting:
-
- # mount -t ext4 /dev/hda1 /wherever
-
- - When comparing performance with other filesystems, it's always
- important to try multiple workloads; very often a subtle change in a
- workload parameter can completely change the ranking of which
- filesystems do well compared to others. When comparing versus ext3,
- note that ext4 enables write barriers by default, while ext3 does
- not enable write barriers by default. So it is useful to use
- explicitly specify whether barriers are enabled or not when via the
- '-o barriers=[0|1]' mount option for both ext3 and ext4 filesystems
- for a fair comparison. When tuning ext3 for best benchmark numbers,
- it is often worthwhile to try changing the data journaling mode; '-o
- data=writeback' can be faster for some workloads. (Note however that
- running mounted with data=writeback can potentially leave stale data
- exposed in recently written files in case of an unclean shutdown,
- which could be a security exposure in some situations.) Configuring
- the filesystem with a large journal can also be helpful for
- metadata-intensive workloads.
-
-Features
-========
-
-Currently Available
--------------------
-
-* ability to use filesystems > 16TB (e2fsprogs support not available yet)
-* extent format reduces metadata overhead (RAM, IO for access, transactions)
-* extent format more robust in face of on-disk corruption due to magics,
-* internal redundancy in tree
-* improved file allocation (multi-block alloc)
-* lift 32000 subdirectory limit imposed by i_links_count[1]
-* nsec timestamps for mtime, atime, ctime, create time
-* inode version field on disk (NFSv4, Lustre)
-* reduced e2fsck time via uninit_bg feature
-* journal checksumming for robustness, performance
-* persistent file preallocation (e.g for streaming media, databases)
-* ability to pack bitmaps and inode tables into larger virtual groups via the
- flex_bg feature
-* large file support
-* inode allocation using large virtual block groups via flex_bg
-* delayed allocation
-* large block (up to pagesize) support
-* efficient new ordered mode in JBD2 and ext4 (avoid using buffer head to force
- the ordering)
-
-[1] Filesystems with a block size of 1k may see a limit imposed by the
-directory hash tree having a maximum depth of two.
-
-Options
-=======
-
-When mounting an ext4 filesystem, the following option are accepted:
-(*) == default
-
-======================= =======================================================
-Mount Option Description
-======================= =======================================================
-ro Mount filesystem read only. Note that ext4 will
- replay the journal (and thus write to the
- partition) even when mounted "read only". The
- mount options "ro,noload" can be used to prevent
- writes to the filesystem.
-
-journal_checksum Enable checksumming of the journal transactions.
- This will allow the recovery code in e2fsck and the
- kernel to detect corruption in the kernel. It is a
- compatible change and will be ignored by older kernels.
-
-journal_async_commit Commit block can be written to disk without waiting
- for descriptor blocks. If enabled older kernels cannot
- mount the device. This will enable 'journal_checksum'
- internally.
-
-journal_path=path
-journal_dev=devnum When the external journal device's major/minor numbers
- have changed, these options allow the user to specify
- the new journal location. The journal device is
- identified through either its new major/minor numbers
- encoded in devnum, or via a path to the device.
-
-norecovery Don't load the journal on mounting. Note that
-noload if the filesystem was not unmounted cleanly,
- skipping the journal replay will lead to the
- filesystem containing inconsistencies that can
- lead to any number of problems.
-
-data=journal All data are committed into the journal prior to being
- written into the main file system. Enabling
- this mode will disable delayed allocation and
- O_DIRECT support.
-
-data=ordered (*) All data are forced directly out to the main file
- system prior to its metadata being committed to the
- journal.
-
-data=writeback Data ordering is not preserved, data may be written
- into the main file system after its metadata has been
- committed to the journal.
-
-commit=nrsec (*) Ext4 can be told to sync all its data and metadata
- every 'nrsec' seconds. The default value is 5 seconds.
- This means that if you lose your power, you will lose
- as much as the latest 5 seconds of work (your
- filesystem will not be damaged though, thanks to the
- journaling). This default value (or any low value)
- will hurt performance, but it's good for data-safety.
- Setting it to 0 will have the same effect as leaving
- it at the default (5 seconds).
- Setting it to very large values will improve
- performance.
-
-barrier=<0|1(*)> This enables/disables the use of write barriers in
-barrier(*) the jbd code. barrier=0 disables, barrier=1 enables.
-nobarrier This also requires an IO stack which can support
- barriers, and if jbd gets an error on a barrier
- write, it will disable again with a warning.
- Write barriers enforce proper on-disk ordering
- of journal commits, making volatile disk write caches
- safe to use, at some performance penalty. If
- your disks are battery-backed in one way or another,
- disabling barriers may safely improve performance.
- The mount options "barrier" and "nobarrier" can
- also be used to enable or disable barriers, for
- consistency with other ext4 mount options.
-
-inode_readahead_blks=n This tuning parameter controls the maximum
- number of inode table blocks that ext4's inode
- table readahead algorithm will pre-read into
- the buffer cache. The default value is 32 blocks.
-
-nouser_xattr Disables Extended User Attributes. See the
- attr(5) manual page for more information about
- extended attributes.
-
-noacl This option disables POSIX Access Control List
- support. If ACL support is enabled in the kernel
- configuration (CONFIG_EXT4_FS_POSIX_ACL), ACL is
- enabled by default on mount. See the acl(5) manual
- page for more information about acl.
-
-bsddf (*) Make 'df' act like BSD.
-minixdf Make 'df' act like Minix.
-
-debug Extra debugging information is sent to syslog.
-
-abort Simulate the effects of calling ext4_abort() for
- debugging purposes. This is normally used while
- remounting a filesystem which is already mounted.
-
-errors=remount-ro Remount the filesystem read-only on an error.
-errors=continue Keep going on a filesystem error.
-errors=panic Panic and halt the machine if an error occurs.
- (These mount options override the errors behavior
- specified in the superblock, which can be configured
- using tune2fs)
-
-data_err=ignore(*) Just print an error message if an error occurs
- in a file data buffer in ordered mode.
-data_err=abort Abort the journal if an error occurs in a file
- data buffer in ordered mode.
-
-grpid New objects have the group ID of their parent.
-bsdgroups
-
-nogrpid (*) New objects have the group ID of their creator.
-sysvgroups
-
-resgid=n The group ID which may use the reserved blocks.
-
-resuid=n The user ID which may use the reserved blocks.
-
-sb=n Use alternate superblock at this location.
-
-quota These options are ignored by the filesystem. They
-noquota are used only by quota tools to recognize volumes
-grpquota where quota should be turned on. See documentation
-usrquota in the quota-tools package for more details
- (http://sourceforge.net/projects/linuxquota).
-
-jqfmt=<quota type> These options tell filesystem details about quota
-usrjquota=<file> so that quota information can be properly updated
-grpjquota=<file> during journal replay. They replace the above
- quota options. See documentation in the quota-tools
- package for more details
- (http://sourceforge.net/projects/linuxquota).
-
-stripe=n Number of filesystem blocks that mballoc will try
- to use for allocation size and alignment. For RAID5/6
- systems this should be the number of data
- disks * RAID chunk size in file system blocks.
-
-delalloc (*) Defer block allocation until just before ext4
- writes out the block(s) in question. This
- allows ext4 to better allocation decisions
- more efficiently.
-nodelalloc Disable delayed allocation. Blocks are allocated
- when the data is copied from userspace to the
- page cache, either via the write(2) system call
- or when an mmap'ed page which was previously
- unallocated is written for the first time.
-
-max_batch_time=usec Maximum amount of time ext4 should wait for
- additional filesystem operations to be batch
- together with a synchronous write operation.
- Since a synchronous write operation is going to
- force a commit and then a wait for the I/O
- complete, it doesn't cost much, and can be a
- huge throughput win, we wait for a small amount
- of time to see if any other transactions can
- piggyback on the synchronous write. The
- algorithm used is designed to automatically tune
- for the speed of the disk, by measuring the
- amount of time (on average) that it takes to
- finish committing a transaction. Call this time
- the "commit time". If the time that the
- transaction has been running is less than the
- commit time, ext4 will try sleeping for the
- commit time to see if other operations will join
- the transaction. The commit time is capped by
- the max_batch_time, which defaults to 15000us
- (15ms). This optimization can be turned off
- entirely by setting max_batch_time to 0.
-
-min_batch_time=usec This parameter sets the commit time (as
- described above) to be at least min_batch_time.
- It defaults to zero microseconds. Increasing
- this parameter may improve the throughput of
- multi-threaded, synchronous workloads on very
- fast disks, at the cost of increasing latency.
-
-journal_ioprio=prio The I/O priority (from 0 to 7, where 0 is the
- highest priority) which should be used for I/O
- operations submitted by kjournald2 during a
- commit operation. This defaults to 3, which is
- a slightly higher priority than the default I/O
- priority.
-
-auto_da_alloc(*) Many broken applications don't use fsync() when
-noauto_da_alloc replacing existing files via patterns such as
- fd = open("foo.new")/write(fd,..)/close(fd)/
- rename("foo.new", "foo"), or worse yet,
- fd = open("foo", O_TRUNC)/write(fd,..)/close(fd).
- If auto_da_alloc is enabled, ext4 will detect
- the replace-via-rename and replace-via-truncate
- patterns and force that any delayed allocation
- blocks are allocated such that at the next
- journal commit, in the default data=ordered
- mode, the data blocks of the new file are forced
- to disk before the rename() operation is
- committed. This provides roughly the same level
- of guarantees as ext3, and avoids the
- "zero-length" problem that can happen when a
- system crashes before the delayed allocation
- blocks are forced to disk.
-
-noinit_itable Do not initialize any uninitialized inode table
- blocks in the background. This feature may be
- used by installation CD's so that the install
- process can complete as quickly as possible; the
- inode table initialization process would then be
- deferred until the next time the file system
- is unmounted.
-
-init_itable=n The lazy itable init code will wait n times the
- number of milliseconds it took to zero out the
- previous block group's inode table. This
- minimizes the impact on the system performance
- while file system's inode table is being initialized.
-
-discard Controls whether ext4 should issue discard/TRIM
-nodiscard(*) commands to the underlying block device when
- blocks are freed. This is useful for SSD devices
- and sparse/thinly-provisioned LUNs, but it is off
- by default until sufficient testing has been done.
-
-nouid32 Disables 32-bit UIDs and GIDs. This is for
- interoperability with older kernels which only
- store and expect 16-bit values.
-
-block_validity(*) These options enable or disable the in-kernel
-noblock_validity facility for tracking filesystem metadata blocks
- within internal data structures. This allows multi-
- block allocator and other routines to notice
- bugs or corrupted allocation bitmaps which cause
- blocks to be allocated which overlap with
- filesystem metadata blocks.
-
-dioread_lock Controls whether or not ext4 should use the DIO read
-dioread_nolock locking. If the dioread_nolock option is specified
- ext4 will allocate uninitialized extent before buffer
- write and convert the extent to initialized after IO
- completes. This approach allows ext4 code to avoid
- using inode mutex, which improves scalability on high
- speed storages. However this does not work with
- data journaling and dioread_nolock option will be
- ignored with kernel warning. Note that dioread_nolock
- code path is only used for extent-based files.
- Because of the restrictions this options comprises
- it is off by default (e.g. dioread_lock).
-
-max_dir_size_kb=n This limits the size of directories so that any
- attempt to expand them beyond the specified
- limit in kilobytes will cause an ENOSPC error.
- This is useful in memory constrained
- environments, where a very large directory can
- cause severe performance problems or even
- provoke the Out Of Memory killer. (For example,
- if there is only 512mb memory available, a 176mb
- directory may seriously cramp the system's style.)
-
-i_version Enable 64-bit inode version support. This option is
- off by default.
-
-dax Use direct access (no page cache). See
- Documentation/filesystems/dax.txt. Note that
- this option is incompatible with data=journal.
-======================= =======================================================
-
-Data Mode
-=========
-There are 3 different data modes:
-
-* writeback mode
-
- In data=writeback mode, ext4 does not journal data at all. This mode provides
- a similar level of journaling as that of XFS, JFS, and ReiserFS in its default
- mode - metadata journaling. A crash+recovery can cause incorrect data to
- appear in files which were written shortly before the crash. This mode will
- typically provide the best ext4 performance.
-
-* ordered mode
-
- In data=ordered mode, ext4 only officially journals metadata, but it logically
- groups metadata information related to data changes with the data blocks into
- a single unit called a transaction. When it's time to write the new metadata
- out to disk, the associated data blocks are written first. In general, this
- mode performs slightly slower than writeback but significantly faster than
- journal mode.
-
-* journal mode
-
- data=journal mode provides full data and metadata journaling. All new data is
- written to the journal first, and then to its final location. In the event of
- a crash, the journal can be replayed, bringing both data and metadata into a
- consistent state. This mode is the slowest except when data needs to be read
- from and written to disk at the same time where it outperforms all others
- modes. Enabling this mode will disable delayed allocation and O_DIRECT
- support.
-
-/proc entries
-=============
-
-Information about mounted ext4 file systems can be found in
-/proc/fs/ext4. Each mounted filesystem will have a directory in
-/proc/fs/ext4 based on its device name (i.e., /proc/fs/ext4/hdc or
-/proc/fs/ext4/dm-0). The files in each per-device directory are shown
-in table below.
-
-Files in /proc/fs/ext4/<devname>
-
-================ =======
- File Content
-================ =======
- mb_groups details of multiblock allocator buddy cache of free blocks
-================ =======
-
-/sys entries
-============
-
-Information about mounted ext4 file systems can be found in
-/sys/fs/ext4. Each mounted filesystem will have a directory in
-/sys/fs/ext4 based on its device name (i.e., /sys/fs/ext4/hdc or
-/sys/fs/ext4/dm-0). The files in each per-device directory are shown
-in table below.
-
-Files in /sys/fs/ext4/<devname>:
-
-(see also Documentation/ABI/testing/sysfs-fs-ext4)
-
-============================= =================================================
-File Content
-============================= =================================================
- delayed_allocation_blocks This file is read-only and shows the number of
- blocks that are dirty in the page cache, but
- which do not have their location in the
- filesystem allocated yet.
-
-inode_goal Tuning parameter which (if non-zero) controls
- the goal inode used by the inode allocator in
- preference to all other allocation heuristics.
- This is intended for debugging use only, and
- should be 0 on production systems.
-
-inode_readahead_blks Tuning parameter which controls the maximum
- number of inode table blocks that ext4's inode
- table readahead algorithm will pre-read into
- the buffer cache
-
-lifetime_write_kbytes This file is read-only and shows the number of
- kilobytes of data that have been written to this
- filesystem since it was created.
-
- max_writeback_mb_bump The maximum number of megabytes the writeback
- code will try to write out before move on to
- another inode.
-
- mb_group_prealloc The multiblock allocator will round up allocation
- requests to a multiple of this tuning parameter if
- the stripe size is not set in the ext4 superblock
-
- mb_max_to_scan The maximum number of extents the multiblock
- allocator will search to find the best extent
-
- mb_min_to_scan The minimum number of extents the multiblock
- allocator will search to find the best extent
-
- mb_order2_req Tuning parameter which controls the minimum size
- for requests (as a power of 2) where the buddy
- cache is used
-
- mb_stats Controls whether the multiblock allocator should
- collect statistics, which are shown during the
- unmount. 1 means to collect statistics, 0 means
- not to collect statistics
-
- mb_stream_req Files which have fewer blocks than this tunable
- parameter will have their blocks allocated out
- of a block group specific preallocation pool, so
- that small files are packed closely together.
- Each large file will have its blocks allocated
- out of its own unique preallocation pool.
-
- session_write_kbytes This file is read-only and shows the number of
- kilobytes of data that have been written to this
- filesystem since it was mounted.
-
- reserved_clusters This is RW file and contains number of reserved
- clusters in the file system which will be used
- in the specific situations to avoid costly
- zeroout, unexpected ENOSPC, or possible data
- loss. The default is 2% or 4096 clusters,
- whichever is smaller and this can be changed
- however it can never exceed number of clusters
- in the file system. If there is not enough space
- for the reserved space when mounting the file
- mount will _not_ fail.
-============================= =================================================
-
-Ioctls
-======
-
-There is some Ext4 specific functionality which can be accessed by applications
-through the system call interfaces. The list of all Ext4 specific ioctls are
-shown in the table below.
-
-Table of Ext4 specific ioctls
-
-============================= =================================================
-Ioctl Description
-============================= =================================================
- EXT4_IOC_GETFLAGS Get additional attributes associated with inode.
- The ioctl argument is an integer bitfield, with
- bit values described in ext4.h. This ioctl is an
- alias for FS_IOC_GETFLAGS.
-
- EXT4_IOC_SETFLAGS Set additional attributes associated with inode.
- The ioctl argument is an integer bitfield, with
- bit values described in ext4.h. This ioctl is an
- alias for FS_IOC_SETFLAGS.
-
- EXT4_IOC_GETVERSION
- EXT4_IOC_GETVERSION_OLD
- Get the inode i_generation number stored for
- each inode. The i_generation number is normally
- changed only when new inode is created and it is
- particularly useful for network filesystems. The
- '_OLD' version of this ioctl is an alias for
- FS_IOC_GETVERSION.
-
- EXT4_IOC_SETVERSION
- EXT4_IOC_SETVERSION_OLD
- Set the inode i_generation number stored for
- each inode. The '_OLD' version of this ioctl
- is an alias for FS_IOC_SETVERSION.
-
- EXT4_IOC_GROUP_EXTEND This ioctl has the same purpose as the resize
- mount option. It allows to resize filesystem
- to the end of the last existing block group,
- further resize has to be done with resize2fs,
- either online, or offline. The argument points
- to the unsigned logn number representing the
- filesystem new block count.
-
- EXT4_IOC_MOVE_EXT Move the block extents from orig_fd (the one
- this ioctl is pointing to) to the donor_fd (the
- one specified in move_extent structure passed
- as an argument to this ioctl). Then, exchange
- inode metadata between orig_fd and donor_fd.
- This is especially useful for online
- defragmentation, because the allocator has the
- opportunity to allocate moved blocks better,
- ideally into one contiguous extent.
-
- EXT4_IOC_GROUP_ADD Add a new group descriptor to an existing or
- new group descriptor block. The new group
- descriptor is described by ext4_new_group_input
- structure, which is passed as an argument to
- this ioctl. This is especially useful in
- conjunction with EXT4_IOC_GROUP_EXTEND,
- which allows online resize of the filesystem
- to the end of the last existing block group.
- Those two ioctls combined is used in userspace
- online resize tool (e.g. resize2fs).
-
- EXT4_IOC_MIGRATE This ioctl operates on the filesystem itself.
- It converts (migrates) ext3 indirect block mapped
- inode to ext4 extent mapped inode by walking
- through indirect block mapping of the original
- inode and converting contiguous block ranges
- into ext4 extents of the temporary inode. Then,
- inodes are swapped. This ioctl might help, when
- migrating from ext3 to ext4 filesystem, however
- suggestion is to create fresh ext4 filesystem
- and copy data from the backup. Note, that
- filesystem has to support extents for this ioctl
- to work.
-
- EXT4_IOC_ALLOC_DA_BLKS Force all of the delay allocated blocks to be
- allocated to preserve application-expected ext3
- behaviour. Note that this will also start
- triggering a write of the data blocks, but this
- behaviour may change in the future as it is
- not necessary and has been done this way only
- for sake of simplicity.
-
- EXT4_IOC_RESIZE_FS Resize the filesystem to a new size. The number
- of blocks of resized filesystem is passed in via
- 64 bit integer argument. The kernel allocates
- bitmaps and inode table, the userspace tool thus
- just passes the new number of blocks.
-
- EXT4_IOC_SWAP_BOOT Swap i_blocks and associated attributes
- (like i_blocks, i_size, i_flags, ...) from
- the specified inode with inode
- EXT4_BOOT_LOADER_INO (#5). This is typically
- used to store a boot loader in a secure part of
- the filesystem, where it can't be changed by a
- normal user by accident.
- The data blocks of the previous boot loader
- will be associated with the given inode.
-============================= =================================================
-
-References
-==========
-
-kernel source: <file:fs/ext4/>
- <file:fs/jbd2/>
-
-programs: http://e2fsprogs.sourceforge.net/
-
-useful links: http://fedoraproject.org/wiki/ext3-devel
- http://www.bullopensource.org/ext4/
- http://ext4.wiki.kernel.org/index.php/Main_Page
- http://fedoraproject.org/wiki/Features/Ext4
diff --git a/Documentation/filesystems/ext4/ondisk/globals.rst b/Documentation/filesystems/ext4/globals.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/globals.rst
rename to Documentation/filesystems/ext4/globals.rst
diff --git a/Documentation/filesystems/ext4/ondisk/group_descr.rst b/Documentation/filesystems/ext4/group_descr.rst
similarity index 92%
rename from Documentation/filesystems/ext4/ondisk/group_descr.rst
rename to Documentation/filesystems/ext4/group_descr.rst
index 759827e..7ba6114 100644
--- a/Documentation/filesystems/ext4/ondisk/group_descr.rst
+++ b/Documentation/filesystems/ext4/group_descr.rst
@@ -43,7 +43,7 @@
The block group descriptor is laid out in ``struct ext4_group_desc``.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -99,9 +99,12 @@
* - 0x1E
- \_\_le16
- bg\_checksum
- - Group descriptor checksum; crc16(sb\_uuid+group+desc) if the
- RO\_COMPAT\_GDT\_CSUM feature is set, or crc32c(sb\_uuid+group\_desc) &
- 0xFFFF if the RO\_COMPAT\_METADATA\_CSUM feature is set.
+ - Group descriptor checksum; crc16(sb\_uuid+group\_num+bg\_desc) if the
+ RO\_COMPAT\_GDT\_CSUM feature is set, or
+ crc32c(sb\_uuid+group\_num+bg\_desc) & 0xFFFF if the
+ RO\_COMPAT\_METADATA\_CSUM feature is set. The bg\_checksum
+ field in bg\_desc is skipped when calculating crc16 checksum,
+ and set to zero if crc32c checksum is used.
* -
-
-
@@ -157,7 +160,7 @@
Block group flags can be any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
diff --git a/Documentation/filesystems/ext4/ondisk/ifork.rst b/Documentation/filesystems/ext4/ifork.rst
similarity index 98%
rename from Documentation/filesystems/ext4/ondisk/ifork.rst
rename to Documentation/filesystems/ext4/ifork.rst
index 5dbe3b2..b9816d5 100644
--- a/Documentation/filesystems/ext4/ondisk/ifork.rst
+++ b/Documentation/filesystems/ext4/ifork.rst
@@ -68,7 +68,7 @@
which is 12 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -104,7 +104,7 @@
recorded as ``struct ext4_extent_idx``, and are 12 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -134,7 +134,7 @@
and are also 12 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -174,7 +174,7 @@
``struct ext4_extent_tail`` is 4 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
diff --git a/Documentation/filesystems/ext4/index.rst b/Documentation/filesystems/ext4/index.rst
index 7112160..705d813 100644
--- a/Documentation/filesystems/ext4/index.rst
+++ b/Documentation/filesystems/ext4/index.rst
@@ -1,17 +1,14 @@
.. SPDX-License-Identifier: GPL-2.0
-===============
-ext4 Filesystem
-===============
-
-General usage and on-disk artifacts writen by ext4. More documentation may
-be ported from the wiki as time permits. This should be considered the
-canonical source of information as the details here have been reviewed by
-the ext4 community.
+===================================
+ext4 Data Structures and Algorithms
+===================================
.. toctree::
- :maxdepth: 5
+ :maxdepth: 6
:numbered:
- ext4
- ondisk/index
+ about
+ overview
+ globals
+ dynamic
diff --git a/Documentation/filesystems/ext4/ondisk/inlinedata.rst b/Documentation/filesystems/ext4/inlinedata.rst
similarity index 100%
rename from Documentation/filesystems/ext4/ondisk/inlinedata.rst
rename to Documentation/filesystems/ext4/inlinedata.rst
diff --git a/Documentation/filesystems/ext4/ondisk/inodes.rst b/Documentation/filesystems/ext4/inodes.rst
similarity index 97%
rename from Documentation/filesystems/ext4/ondisk/inodes.rst
rename to Documentation/filesystems/ext4/inodes.rst
index 655ce89..a65baff 100644
--- a/Documentation/filesystems/ext4/ondisk/inodes.rst
+++ b/Documentation/filesystems/ext4/inodes.rst
@@ -29,8 +29,9 @@
The inode table entry is laid out in ``struct ext4_inode``.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
+ :class: longtable
* - Offset
- Size
@@ -176,7 +177,7 @@
The ``i_mode`` value is a combination of the following flags:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -227,7 +228,7 @@
The ``i_flags`` field is a combination of these values:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -276,6 +277,8 @@
- This is a huge file (EXT4\_HUGE\_FILE\_FL).
* - 0x80000
- Inode uses extents (EXT4\_EXTENTS\_FL).
+ * - 0x100000
+ - Verity protected file (EXT4\_VERITY\_FL).
* - 0x200000
- Inode stores a large extended attribute value in its data blocks
(EXT4\_EA\_INODE\_FL).
@@ -298,9 +301,9 @@
- Reserved for ext4 library (EXT4\_RESERVED\_FL).
* -
- Aggregate flags:
- * - 0x4BDFFF
+ * - 0x705BDFFF
- User-visible flags.
- * - 0x4B80FF
+ * - 0x604BC0FF
- User-modifiable flags. Note that while EXT4\_JOURNAL\_DATA\_FL and
EXT4\_EXTENTS\_FL can be set with setattr, they are not in the kernel's
EXT4\_FL\_USER\_MODIFIABLE mask, since it needs to handle the setting of
@@ -314,7 +317,7 @@
Linux:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -331,7 +334,7 @@
Hurd:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -346,7 +349,7 @@
Masix:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -365,7 +368,7 @@
Linux:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -402,7 +405,7 @@
Hurd:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -433,7 +436,7 @@
Masix:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -469,8 +472,8 @@
having to upgrade all of the on-disk inodes. Access to fields beyond
EXT2\_GOOD\_OLD\_INODE\_SIZE should be verified to be within
``i_extra_isize``. By default, ext4 inode records are 256 bytes, and (as
-of October 2013) the inode structure is 156 bytes
-(``i_extra_isize = 28``). The extra space between the end of the inode
+of August 2019) the inode structure is 160 bytes
+(``i_extra_isize = 32``). The extra space between the end of the inode
structure and the end of the inode record can be used to store extended
attributes. Each inode record can be as large as the filesystem block
size, though this is not terribly efficient.
diff --git a/Documentation/filesystems/ext4/ondisk/journal.rst b/Documentation/filesystems/ext4/journal.rst
similarity index 97%
rename from Documentation/filesystems/ext4/ondisk/journal.rst
rename to Documentation/filesystems/ext4/journal.rst
index e7031af..ea613ee 100644
--- a/Documentation/filesystems/ext4/ondisk/journal.rst
+++ b/Documentation/filesystems/ext4/journal.rst
@@ -48,7 +48,7 @@
Generally speaking, the journal has this format:
.. list-table::
- :widths: 1 1 78
+ :widths: 16 48 16
:header-rows: 1
* - Superblock
@@ -76,7 +76,7 @@
superblock.
.. list-table::
- :widths: 1 1 1 1 76
+ :widths: 12 12 12 32 12
:header-rows: 1
* - 1024 bytes of padding
@@ -98,7 +98,7 @@
``struct journal_header_s``:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -124,7 +124,7 @@
The journal block type can be any one of:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -154,7 +154,7 @@
which is 1024 bytes long:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -264,7 +264,7 @@
The journal compat features are any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -278,7 +278,7 @@
The journal incompat features are any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -306,7 +306,7 @@
most likely choices.
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -330,7 +330,7 @@
Descriptor blocks consume at least 36 bytes, but use a full block:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -355,7 +355,7 @@
following. The size is 16 or 32 bytes.
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -400,7 +400,7 @@
The journal tag flags are any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -421,7 +421,7 @@
following. The size is 8, 12, 24, or 28 bytes:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -471,7 +471,7 @@
``struct jbd2_journal_block_tail``, which looks like this:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -513,7 +513,7 @@
length, but use a full block:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -543,7 +543,7 @@
block is a ``struct jbd2_journal_revoke_tail``, which has this format:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -567,7 +567,7 @@
bytes long (but uses a full block):
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
diff --git a/Documentation/filesystems/ext4/ondisk/mmp.rst b/Documentation/filesystems/ext4/mmp.rst
similarity index 98%
rename from Documentation/filesystems/ext4/ondisk/mmp.rst
rename to Documentation/filesystems/ext4/mmp.rst
index b7d7a31..2566098 100644
--- a/Documentation/filesystems/ext4/ondisk/mmp.rst
+++ b/Documentation/filesystems/ext4/mmp.rst
@@ -32,7 +32,7 @@
The MMP structure (``struct mmp_struct``) is as follows:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 12 20 40
:header-rows: 1
* - Offset
diff --git a/Documentation/filesystems/ext4/ondisk/bigalloc.rst b/Documentation/filesystems/ext4/ondisk/bigalloc.rst
deleted file mode 100644
index c6d8855..0000000
--- a/Documentation/filesystems/ext4/ondisk/bigalloc.rst
+++ /dev/null
@@ -1,22 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0
-
-Bigalloc
---------
-
-At the moment, the default size of a block is 4KiB, which is a commonly
-supported page size on most MMU-capable hardware. This is fortunate, as
-ext4 code is not prepared to handle the case where the block size
-exceeds the page size. However, for a filesystem of mostly huge files,
-it is desirable to be able to allocate disk blocks in units of multiple
-blocks to reduce both fragmentation and metadata overhead. The
-`bigalloc <Bigalloc>`__ feature provides exactly this ability. The
-administrator can set a block cluster size at mkfs time (which is stored
-in the s\_log\_cluster\_size field in the superblock); from then on, the
-block bitmaps track clusters, not individual blocks. This means that
-block groups can be several gigabytes in size (instead of just 128MiB);
-however, the minimum allocation unit becomes a cluster, not a block,
-even for directories. TaoBao had a patchset to extend the “use units of
-clusters instead of blocks” to the extent tree, though it is not clear
-where those patches went-- they eventually morphed into “extent tree v2”
-but that code has not landed as of May 2015.
-
diff --git a/Documentation/filesystems/ext4/ondisk/index.rst b/Documentation/filesystems/ext4/ondisk/index.rst
deleted file mode 100644
index f7d082c..0000000
--- a/Documentation/filesystems/ext4/ondisk/index.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0
-
-==============================
-Data Structures and Algorithms
-==============================
-.. include:: about.rst
-.. include:: overview.rst
-.. include:: globals.rst
-.. include:: dynamic.rst
diff --git a/Documentation/filesystems/ext4/ondisk/overview.rst b/Documentation/filesystems/ext4/overview.rst
similarity index 97%
rename from Documentation/filesystems/ext4/ondisk/overview.rst
rename to Documentation/filesystems/ext4/overview.rst
index cbab18b..123ebfd 100644
--- a/Documentation/filesystems/ext4/ondisk/overview.rst
+++ b/Documentation/filesystems/ext4/overview.rst
@@ -24,3 +24,4 @@
.. include:: bigalloc.rst
.. include:: inlinedata.rst
.. include:: eainode.rst
+.. include:: verity.rst
diff --git a/Documentation/filesystems/ext4/ondisk/special_inodes.rst b/Documentation/filesystems/ext4/special_inodes.rst
similarity index 97%
rename from Documentation/filesystems/ext4/ondisk/special_inodes.rst
rename to Documentation/filesystems/ext4/special_inodes.rst
index a82f70c..9061aab 100644
--- a/Documentation/filesystems/ext4/ondisk/special_inodes.rst
+++ b/Documentation/filesystems/ext4/special_inodes.rst
@@ -6,7 +6,7 @@
ext4 reserves some inode for special features, as follows:
.. list-table::
- :widths: 1 79
+ :widths: 6 70
:header-rows: 1
* - inode Number
diff --git a/Documentation/filesystems/ext4/ondisk/super.rst b/Documentation/filesystems/ext4/super.rst
similarity index 96%
rename from Documentation/filesystems/ext4/ondisk/super.rst
rename to Documentation/filesystems/ext4/super.rst
index 5f81dd8..93e55d7 100644
--- a/Documentation/filesystems/ext4/ondisk/super.rst
+++ b/Documentation/filesystems/ext4/super.rst
@@ -19,7 +19,7 @@
``struct ext4_super_block``:
.. list-table::
- :widths: 1 1 1 77
+ :widths: 8 8 24 40
:header-rows: 1
* - Offset
@@ -58,7 +58,7 @@
* - 0x1C
- \_\_le32
- s\_log\_cluster\_size
- - Cluster size is (2 ^ s\_log\_cluster\_size) blocks if bigalloc is
+ - Cluster size is 2 ^ (10 + s\_log\_cluster\_size) blocks if bigalloc is
enabled. Otherwise s\_log\_cluster\_size must equal s\_log\_block\_size.
* - 0x20
- \_\_le32
@@ -447,7 +447,7 @@
- Upper 8 bits of the s_wtime field.
* - 0x275
- \_\_u8
- - s\_wtime_hi
+ - s\_mtime_hi
- Upper 8 bits of the s_mtime field.
* - 0x276
- \_\_u8
@@ -466,12 +466,20 @@
- s\_last_error_time_hi
- Upper 8 bits of the s_last_error_time_hi field.
* - 0x27A
- - \_\_u8[2]
- - s\_pad
+ - \_\_u8
+ - s\_pad[2]
- Zero padding.
* - 0x27C
+ - \_\_le16
+ - s\_encoding
+ - Filename charset encoding.
+ * - 0x27E
+ - \_\_le16
+ - s\_encoding_flags
+ - Filename charset encoding flags.
+ * - 0x280
- \_\_le32
- - s\_reserved[96]
+ - s\_reserved[95]
- Padding to the end of the block.
* - 0x3FC
- \_\_le32
@@ -483,7 +491,7 @@
The superblock state is some combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -500,7 +508,7 @@
The superblock error policy is one of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -517,7 +525,7 @@
The filesystem creator is one of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -538,7 +546,7 @@
The superblock revision is one of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -556,7 +564,7 @@
following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -595,7 +603,7 @@
following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -617,7 +625,7 @@
* - 0x80
- Enable a filesystem size of 2^64 blocks (INCOMPAT\_64BIT).
* - 0x100
- - Multiple mount protection. Not implemented (INCOMPAT\_MMP).
+ - Multiple mount protection (INCOMPAT\_MMP).
* - 0x200
- Flexible block groups. See the earlier discussion of this feature
(INCOMPAT\_FLEX\_BG).
@@ -647,7 +655,7 @@
the following:
.. list-table::
- :widths: 1 79
+ :widths: 16 64
:header-rows: 1
* - Value
@@ -696,13 +704,15 @@
(RO\_COMPAT\_READONLY)
* - 0x2000
- Filesystem tracks project quotas. (RO\_COMPAT\_PROJECT)
+ * - 0x8000
+ - Verity inodes may be present on the filesystem. (RO\_COMPAT\_VERITY)
.. _super_def_hash:
The ``s_def_hash_version`` field is one of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -725,7 +735,7 @@
The ``s_default_mount_opts`` field is any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -767,7 +777,7 @@
The ``s_flags`` field is any combination of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
@@ -784,7 +794,7 @@
The ``s_encrypt_algos`` list can contain any of the following:
.. list-table::
- :widths: 1 79
+ :widths: 8 72
:header-rows: 1
* - Value
diff --git a/Documentation/filesystems/ext4/verity.rst b/Documentation/filesystems/ext4/verity.rst
new file mode 100644
index 0000000..3e4c0ee
--- /dev/null
+++ b/Documentation/filesystems/ext4/verity.rst
@@ -0,0 +1,41 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Verity files
+------------
+
+ext4 supports fs-verity, which is a filesystem feature that provides
+Merkle tree based hashing for individual readonly files. Most of
+fs-verity is common to all filesystems that support it; see
+:ref:`Documentation/filesystems/fsverity.rst <fsverity>` for the
+fs-verity documentation. However, the on-disk layout of the verity
+metadata is filesystem-specific. On ext4, the verity metadata is
+stored after the end of the file data itself, in the following format:
+
+- Zero-padding to the next 65536-byte boundary. This padding need not
+ actually be allocated on-disk, i.e. it may be a hole.
+
+- The Merkle tree, as documented in
+ :ref:`Documentation/filesystems/fsverity.rst
+ <fsverity_merkle_tree>`, with the tree levels stored in order from
+ root to leaf, and the tree blocks within each level stored in their
+ natural order.
+
+- Zero-padding to the next filesystem block boundary.
+
+- The verity descriptor, as documented in
+ :ref:`Documentation/filesystems/fsverity.rst <fsverity_descriptor>`,
+ with optionally appended signature blob.
+
+- Zero-padding to the next offset that is 4 bytes before a filesystem
+ block boundary.
+
+- The size of the verity descriptor in bytes, as a 4-byte little
+ endian integer.
+
+Verity inodes have EXT4_VERITY_FL set, and they must use extents, i.e.
+EXT4_EXTENTS_FL must be set and EXT4_INLINE_DATA_FL must be clear.
+They can have EXT4_ENCRYPT_FL set, in which case the verity metadata
+is encrypted as well as the data itself.
+
+Verity files cannot have blocks allocated past the end of the verity
+metadata.
diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt
index e5edd29..7e19913 100644
--- a/Documentation/filesystems/f2fs.txt
+++ b/Documentation/filesystems/f2fs.txt
@@ -126,6 +126,8 @@
does not aware of cold files such as media files.
inline_xattr Enable the inline xattrs feature.
noinline_xattr Disable the inline xattrs feature.
+inline_xattr_size=%u Support configuring inline xattr size, it depends on
+ flexible inline xattr feature.
inline_data Enable the inline data feature: New created small(<~3.4k)
files can be written into inode block.
inline_dentry Enable the inline dir feature: data in new created
@@ -155,6 +157,11 @@
enabled by default.
data_flush Enable data flushing before checkpoint in order to
persist data of regular and symlink.
+reserve_root=%d Support configuring reserved space which is used for
+ allocation from a privileged user with specified uid or
+ gid, unit: 4KB, the default limit is 0.2% of user blocks.
+resuid=%d The user ID which may use the reserved blocks.
+resgid=%d The group ID which may use the reserved blocks.
fault_injection=%d Enable fault injection in all supported types with
specified injection rate.
fault_type=%d Support configuring fault injection type, should be
@@ -172,9 +179,10 @@
FAULT_DIR_DEPTH 0x000000100
FAULT_EVICT_INODE 0x000000200
FAULT_TRUNCATE 0x000000400
- FAULT_IO 0x000000800
+ FAULT_READ_IO 0x000000800
FAULT_CHECKPOINT 0x000001000
FAULT_DISCARD 0x000002000
+ FAULT_WRITE_IO 0x000004000
mode=%s Control block allocation mode which supports "adaptive"
and "lfs". In "lfs" mode, there should be no random
writes towards main area.
@@ -211,6 +219,22 @@
non-atomic files likewise "nobarrier" mount option.
test_dummy_encryption Enable dummy encryption, which provides a fake fscrypt
context. The fake fscrypt context is used by xfstests.
+checkpoint=%s[:%u[%]] Set to "disable" to turn off checkpointing. Set to "enable"
+ to reenable checkpointing. Is enabled by default. While
+ disabled, any unmounting or unexpected shutdowns will cause
+ the filesystem contents to appear as they did when the
+ filesystem was mounted with that option.
+ While mounting with checkpoint=disabled, the filesystem must
+ run garbage collection to ensure that all available space can
+ be used. If this takes too much time, the mount may return
+ EAGAIN. You may optionally add a value to indicate how much
+ of the disk you would be willing to temporarily give up to
+ avoid additional garbage collection. This can be given as a
+ number of blocks, or as a percent. For instance, mounting
+ with checkpoint=disable:100% would always succeed, but it may
+ hide up to all remaining free space. The actual space that
+ would be unusable can be viewed at /sys/fs/f2fs/<disk>/unusable
+ This space is reclaimed once checkpoint=enable.
================================================================================
DEBUGFS ENTRIES
@@ -238,11 +262,14 @@
..............................................................................
File Content
- gc_max_sleep_time This tuning parameter controls the maximum sleep
+ gc_urgent_sleep_time This parameter controls sleep time for gc_urgent.
+ 500 ms is set by default. See above gc_urgent.
+
+ gc_min_sleep_time This tuning parameter controls the minimum sleep
time for the garbage collection thread. Time is
in milliseconds.
- gc_min_sleep_time This tuning parameter controls the minimum sleep
+ gc_max_sleep_time This tuning parameter controls the maximum sleep
time for the garbage collection thread. Time is
in milliseconds.
@@ -262,9 +289,6 @@
to 1, background thread starts to do GC by given
gc_urgent_sleep_time interval.
- gc_urgent_sleep_time This parameter controls sleep time for gc_urgent.
- 500 ms is set by default. See above gc_urgent.
-
reclaim_segments This parameter controls the number of prefree
segments to be reclaimed. If the number of prefree
segments is larger than the number of segments
@@ -279,7 +303,16 @@
checkpoint is triggered, and issued during the
checkpoint. By default, it is disabled with 0.
- trim_sections This parameter controls the number of sections
+ discard_granularity This parameter controls the granularity of discard
+ command size. It will issue discard commands iif
+ the size is larger than given granularity. Its
+ unit size is 4KB, and 4 (=16KB) is set by default.
+ The maximum value is 128 (=512KB).
+
+ reserved_blocks This parameter indicates the number of blocks that
+ f2fs reserves internally for root.
+
+ batched_trim_sections This parameter controls the number of sections
to be trimmed out in batch mode when FITRIM
conducts. 32 sections is set by default.
@@ -301,11 +334,35 @@
the number is less than this value, it triggers
in-place-updates.
+ min_seq_blocks This parameter controls the threshold to serialize
+ write IOs issued by multiple threads in parallel.
+
+ min_hot_blocks This parameter controls the threshold to allocate
+ a hot data log for pending data blocks to write.
+
+ min_ssr_sections This parameter adds the threshold when deciding
+ SSR block allocation. If this is large, SSR mode
+ will be enabled early.
+
+ ram_thresh This parameter controls the memory footprint used
+ by free nids and cached nat entries. By default,
+ 10 is set, which indicates 10 MB / 1 GB RAM.
+
+ ra_nid_pages When building free nids, F2FS reads NAT blocks
+ ahead for speed up. Default is 0.
+
+ dirty_nats_ratio Given dirty ratio of cached nat entries, F2FS
+ determines flushing them in background.
+
max_victim_search This parameter controls the number of trials to
find a victim segment when conducting SSR and
cleaning operations. The default value is 4096
which covers 8GB block address range.
+ migration_granularity For large-sized sections, F2FS can stop GC given
+ this granularity instead of reclaiming entire
+ section.
+
dir_level This parameter controls the directory level to
support large directory. If a directory has a
number of files, it can reduce the file lookup
@@ -313,9 +370,56 @@
Otherwise, it needs to decrease this value to
reduce the space overhead. The default value is 0.
- ram_thresh This parameter controls the memory footprint used
- by free nids and cached nat entries. By default,
- 10 is set, which indicates 10 MB / 1 GB RAM.
+ cp_interval F2FS tries to do checkpoint periodically, 60 secs
+ by default.
+
+ idle_interval F2FS detects system is idle, if there's no F2FS
+ operations during given interval, 5 secs by
+ default.
+
+ discard_idle_interval F2FS detects the discard thread is idle, given
+ time interval. Default is 5 secs.
+
+ gc_idle_interval F2FS detects the GC thread is idle, given time
+ interval. Default is 5 secs.
+
+ umount_discard_timeout When unmounting the disk, F2FS waits for finishing
+ queued discard commands which can take huge time.
+ This gives time out for it, 5 secs by default.
+
+ iostat_enable This controls to enable/disable iostat in F2FS.
+
+ readdir_ra This enables/disabled readahead of inode blocks
+ in readdir, and default is enabled.
+
+ gc_pin_file_thresh This indicates how many GC can be failed for the
+ pinned file. If it exceeds this, F2FS doesn't
+ guarantee its pinning state. 2048 trials is set
+ by default.
+
+ extension_list This enables to change extension_list for hot/cold
+ files in runtime.
+
+ inject_rate This controls injection rate of arbitrary faults.
+
+ inject_type This controls injection type of arbitrary faults.
+
+ dirty_segments This shows # of dirty segments.
+
+ lifetime_write_kbytes This shows # of data written to the disk.
+
+ features This shows current features enabled on F2FS.
+
+ current_reserved_blocks This shows # of blocks currently reserved.
+
+ unusable If checkpoint=disable, this shows the number of
+ blocks that are unusable.
+ If checkpoint=enable it shows the number of blocks
+ that would be unusable if checkpoint=disable were
+ to be set.
+
+encoding This shows the encoding used for casefolding.
+ If casefolding is not enabled, returns (none)
================================================================================
USAGE
@@ -708,3 +812,28 @@
WRITE_LIFE_NONE " WRITE_LIFE_NONE
WRITE_LIFE_MEDIUM " WRITE_LIFE_MEDIUM
WRITE_LIFE_LONG " WRITE_LIFE_LONG
+
+Fallocate(2) Policy
+-------------------
+
+The default policy follows the below posix rule.
+
+Allocating disk space
+ The default operation (i.e., mode is zero) of fallocate() allocates
+ the disk space within the range specified by offset and len. The
+ file size (as reported by stat(2)) will be changed if offset+len is
+ greater than the file size. Any subregion within the range specified
+ by offset and len that did not contain data before the call will be
+ initialized to zero. This default behavior closely resembles the
+ behavior of the posix_fallocate(3) library function, and is intended
+ as a method of optimally implementing that function.
+
+However, once F2FS receives ioctl(fd, F2FS_IOC_SET_PIN_FILE) in prior to
+fallocate(fd, DEFAULT_MODE), it allocates on-disk blocks addressess having
+zero or random data, which is useful to the below scenario where:
+ 1. create(fd)
+ 2. ioctl(fd, F2FS_IOC_SET_PIN_FILE)
+ 3. fallocate(fd, 0, 0, size)
+ 4. address = fibmap(fd, offset)
+ 5. open(blkdev)
+ 6. write(blkdev, address)
diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst
index cfbc18f..8a0700a 100644
--- a/Documentation/filesystems/fscrypt.rst
+++ b/Documentation/filesystems/fscrypt.rst
@@ -72,6 +72,9 @@
fscrypt (and storage encryption in general) can only provide limited
protection, if any at all, against online attacks. In detail:
+Side-channel attacks
+~~~~~~~~~~~~~~~~~~~~
+
fscrypt is only resistant to side-channel attacks, such as timing or
electromagnetic attacks, to the extent that the underlying Linux
Cryptographic API algorithms are. If a vulnerable algorithm is used,
@@ -80,29 +83,90 @@
Side channel attacks may also be mounted against applications
consuming decrypted data.
-After an encryption key has been provided, fscrypt is not designed to
-hide the plaintext file contents or filenames from other users on the
-same system, regardless of the visibility of the keyring key.
-Instead, existing access control mechanisms such as file mode bits,
-POSIX ACLs, LSMs, or mount namespaces should be used for this purpose.
-Also note that as long as the encryption keys are *anywhere* in
-memory, an online attacker can necessarily compromise them by mounting
-a physical attack or by exploiting any kernel security vulnerability
-which provides an arbitrary memory read primitive.
+Unauthorized file access
+~~~~~~~~~~~~~~~~~~~~~~~~
-While it is ostensibly possible to "evict" keys from the system,
-recently accessed encrypted files will remain accessible at least
-until the filesystem is unmounted or the VFS caches are dropped, e.g.
-using ``echo 2 > /proc/sys/vm/drop_caches``. Even after that, if the
-RAM is compromised before being powered off, it will likely still be
-possible to recover portions of the plaintext file contents, if not
-some of the encryption keys as well. (Since Linux v4.12, all
-in-kernel keys related to fscrypt are sanitized before being freed.
-However, userspace would need to do its part as well.)
+After an encryption key has been added, fscrypt does not hide the
+plaintext file contents or filenames from other users on the same
+system. Instead, existing access control mechanisms such as file mode
+bits, POSIX ACLs, LSMs, or namespaces should be used for this purpose.
-Currently, fscrypt does not prevent a user from maliciously providing
-an incorrect key for another user's existing encrypted files. A
-protection against this is planned.
+(For the reasoning behind this, understand that while the key is
+added, the confidentiality of the data, from the perspective of the
+system itself, is *not* protected by the mathematical properties of
+encryption but rather only by the correctness of the kernel.
+Therefore, any encryption-specific access control checks would merely
+be enforced by kernel *code* and therefore would be largely redundant
+with the wide variety of access control mechanisms already available.)
+
+Kernel memory compromise
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker who compromises the system enough to read from arbitrary
+memory, e.g. by mounting a physical attack or by exploiting a kernel
+security vulnerability, can compromise all encryption keys that are
+currently in use.
+
+However, fscrypt allows encryption keys to be removed from the kernel,
+which may protect them from later compromise.
+
+In more detail, the FS_IOC_REMOVE_ENCRYPTION_KEY ioctl (or the
+FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS ioctl) can wipe a master
+encryption key from kernel memory. If it does so, it will also try to
+evict all cached inodes which had been "unlocked" using the key,
+thereby wiping their per-file keys and making them once again appear
+"locked", i.e. in ciphertext or encrypted form.
+
+However, these ioctls have some limitations:
+
+- Per-file keys for in-use files will *not* be removed or wiped.
+ Therefore, for maximum effect, userspace should close the relevant
+ encrypted files and directories before removing a master key, as
+ well as kill any processes whose working directory is in an affected
+ encrypted directory.
+
+- The kernel cannot magically wipe copies of the master key(s) that
+ userspace might have as well. Therefore, userspace must wipe all
+ copies of the master key(s) it makes as well; normally this should
+ be done immediately after FS_IOC_ADD_ENCRYPTION_KEY, without waiting
+ for FS_IOC_REMOVE_ENCRYPTION_KEY. Naturally, the same also applies
+ to all higher levels in the key hierarchy. Userspace should also
+ follow other security precautions such as mlock()ing memory
+ containing keys to prevent it from being swapped out.
+
+- In general, decrypted contents and filenames in the kernel VFS
+ caches are freed but not wiped. Therefore, portions thereof may be
+ recoverable from freed memory, even after the corresponding key(s)
+ were wiped. To partially solve this, you can set
+ CONFIG_PAGE_POISONING=y in your kernel config and add page_poison=1
+ to your kernel command line. However, this has a performance cost.
+
+- Secret keys might still exist in CPU registers, in crypto
+ accelerator hardware (if used by the crypto API to implement any of
+ the algorithms), or in other places not explicitly considered here.
+
+Limitations of v1 policies
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+v1 encryption policies have some weaknesses with respect to online
+attacks:
+
+- There is no verification that the provided master key is correct.
+ Therefore, a malicious user can temporarily associate the wrong key
+ with another user's encrypted files to which they have read-only
+ access. Because of filesystem caching, the wrong key will then be
+ used by the other user's accesses to those files, even if the other
+ user has the correct key in their own keyring. This violates the
+ meaning of "read-only access".
+
+- A compromise of a per-file key also compromises the master key from
+ which it was derived.
+
+- Non-root users cannot securely remove encryption keys.
+
+All the above problems are fixed with v2 encryption policies. For
+this reason among others, it is recommended to use v2 encryption
+policies on all new encrypted directories.
Key hierarchy
=============
@@ -123,63 +187,110 @@
of which protects any number of directory trees on any number of
filesystems.
-Userspace should generate master keys either using a cryptographically
-secure random number generator, or by using a KDF (Key Derivation
-Function). Note that whenever a KDF is used to "stretch" a
-lower-entropy secret such as a passphrase, it is critical that a KDF
-designed for this purpose be used, such as scrypt, PBKDF2, or Argon2.
+Master keys must be real cryptographic keys, i.e. indistinguishable
+from random bytestrings of the same length. This implies that users
+**must not** directly use a password as a master key, zero-pad a
+shorter key, or repeat a shorter key. Security cannot be guaranteed
+if userspace makes any such error, as the cryptographic proofs and
+analysis would no longer apply.
+
+Instead, users should generate master keys either using a
+cryptographically secure random number generator, or by using a KDF
+(Key Derivation Function). The kernel does not do any key stretching;
+therefore, if userspace derives the key from a low-entropy secret such
+as a passphrase, it is critical that a KDF designed for this purpose
+be used, such as scrypt, PBKDF2, or Argon2.
+
+Key derivation function
+-----------------------
+
+With one exception, fscrypt never uses the master key(s) for
+encryption directly. Instead, they are only used as input to a KDF
+(Key Derivation Function) to derive the actual keys.
+
+The KDF used for a particular master key differs depending on whether
+the key is used for v1 encryption policies or for v2 encryption
+policies. Users **must not** use the same key for both v1 and v2
+encryption policies. (No real-world attack is currently known on this
+specific case of key reuse, but its security cannot be guaranteed
+since the cryptographic proofs and analysis would no longer apply.)
+
+For v1 encryption policies, the KDF only supports deriving per-file
+encryption keys. It works by encrypting the master key with
+AES-128-ECB, using the file's 16-byte nonce as the AES key. The
+resulting ciphertext is used as the derived key. If the ciphertext is
+longer than needed, then it is truncated to the needed length.
+
+For v2 encryption policies, the KDF is HKDF-SHA512. The master key is
+passed as the "input keying material", no salt is used, and a distinct
+"application-specific information string" is used for each distinct
+key to be derived. For example, when a per-file encryption key is
+derived, the application-specific information string is the file's
+nonce prefixed with "fscrypt\\0" and a context byte. Different
+context bytes are used for other types of derived keys.
+
+HKDF-SHA512 is preferred to the original AES-128-ECB based KDF because
+HKDF is more flexible, is nonreversible, and evenly distributes
+entropy from the master key. HKDF is also standardized and widely
+used by other software, whereas the AES-128-ECB based KDF is ad-hoc.
Per-file keys
-------------
-Master keys are not used to encrypt file contents or names directly.
-Instead, a unique key is derived for each encrypted file, including
-each regular file, directory, and symbolic link. This has several
-advantages:
+Since each master key can protect many files, it is necessary to
+"tweak" the encryption of each file so that the same plaintext in two
+files doesn't map to the same ciphertext, or vice versa. In most
+cases, fscrypt does this by deriving per-file keys. When a new
+encrypted inode (regular file, directory, or symlink) is created,
+fscrypt randomly generates a 16-byte nonce and stores it in the
+inode's encryption xattr. Then, it uses a KDF (as described in `Key
+derivation function`_) to derive the file's key from the master key
+and nonce.
-- In cryptosystems, the same key material should never be used for
- different purposes. Using the master key as both an XTS key for
- contents encryption and as a CTS-CBC key for filenames encryption
- would violate this rule.
-- Per-file keys simplify the choice of IVs (Initialization Vectors)
- for contents encryption. Without per-file keys, to ensure IV
- uniqueness both the inode and logical block number would need to be
- encoded in the IVs. This would make it impossible to renumber
- inodes, which e.g. ``resize2fs`` can do when resizing an ext4
- filesystem. With per-file keys, it is sufficient to encode just the
- logical block number in the IVs.
-- Per-file keys strengthen the encryption of filenames, where IVs are
- reused out of necessity. With a unique key per directory, IV reuse
- is limited to within a single directory.
-- Per-file keys allow individual files to be securely erased simply by
- securely erasing their keys. (Not yet implemented.)
+Key derivation was chosen over key wrapping because wrapped keys would
+require larger xattrs which would be less likely to fit in-line in the
+filesystem's inode table, and there didn't appear to be any
+significant advantages to key wrapping. In particular, currently
+there is no requirement to support unlocking a file with multiple
+alternative master keys or to support rotating master keys. Instead,
+the master keys may be wrapped in userspace, e.g. as is done by the
+`fscrypt <https://github.com/google/fscrypt>`_ tool.
-A KDF (Key Derivation Function) is used to derive per-file keys from
-the master key. This is done instead of wrapping a randomly-generated
-key for each file because it reduces the size of the encryption xattr,
-which for some filesystems makes the xattr more likely to fit in-line
-in the filesystem's inode table. With a KDF, only a 16-byte nonce is
-required --- long enough to make key reuse extremely unlikely. A
-wrapped key, on the other hand, would need to be up to 64 bytes ---
-the length of an AES-256-XTS key. Furthermore, currently there is no
-requirement to support unlocking a file with multiple alternative
-master keys or to support rotating master keys. Instead, the master
-keys may be wrapped in userspace, e.g. as done by the `fscrypt
-<https://github.com/google/fscrypt>`_ tool.
+Including the inode number in the IVs was considered. However, it was
+rejected as it would have prevented ext4 filesystems from being
+resized, and by itself still wouldn't have been sufficient to prevent
+the same key from being directly reused for both XTS and CTS-CBC.
-The current KDF encrypts the master key using the 16-byte nonce as an
-AES-128-ECB key. The output is used as the derived key. If the
-output is longer than needed, then it is truncated to the needed
-length. Truncation is the norm for directories and symlinks, since
-those use the CTS-CBC encryption mode which requires a key half as
-long as that required by the XTS encryption mode.
+DIRECT_KEY and per-mode keys
+----------------------------
-Note: this KDF meets the primary security requirement, which is to
-produce unique derived keys that preserve the entropy of the master
-key, assuming that the master key is already a good pseudorandom key.
-However, it is nonstandard and has some problems such as being
-reversible, so it is generally considered to be a mistake! It may be
-replaced with HKDF or another more standard KDF in the future.
+The Adiantum encryption mode (see `Encryption modes and usage`_) is
+suitable for both contents and filenames encryption, and it accepts
+long IVs --- long enough to hold both an 8-byte logical block number
+and a 16-byte per-file nonce. Also, the overhead of each Adiantum key
+is greater than that of an AES-256-XTS key.
+
+Therefore, to improve performance and save memory, for Adiantum a
+"direct key" configuration is supported. When the user has enabled
+this by setting FSCRYPT_POLICY_FLAG_DIRECT_KEY in the fscrypt policy,
+per-file keys are not used. Instead, whenever any data (contents or
+filenames) is encrypted, the file's 16-byte nonce is included in the
+IV. Moreover:
+
+- For v1 encryption policies, the encryption is done directly with the
+ master key. Because of this, users **must not** use the same master
+ key for any other purpose, even for other v1 policies.
+
+- For v2 encryption policies, the encryption is done with a per-mode
+ key derived using the KDF. Users may use the same master key for
+ other v2 encryption policies.
+
+Key identifiers
+---------------
+
+For master keys used for v2 encryption policies, a unique 16-byte "key
+identifier" is also derived using the KDF. This value is stored in
+the clear, since it is needed to reliably identify the key itself.
Encryption modes and usage
==========================
@@ -191,54 +302,83 @@
- AES-256-XTS for contents and AES-256-CTS-CBC for filenames
- AES-128-CBC for contents and AES-128-CTS-CBC for filenames
+- Adiantum for both contents and filenames
-It is strongly recommended to use AES-256-XTS for contents encryption.
+If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
+
AES-128-CBC was added only for low-powered embedded devices with
-crypto accelerators such as CAAM or CESA that do not support XTS.
+crypto accelerators such as CAAM or CESA that do not support XTS. To
+use AES-128-CBC, CONFIG_CRYPTO_SHA256 (or another SHA-256
+implementation) must be enabled so that ESSIV can be used.
+
+Adiantum is a (primarily) stream cipher-based mode that is fast even
+on CPUs without dedicated crypto instructions. It's also a true
+wide-block mode, unlike XTS. It can also eliminate the need to derive
+per-file keys. However, it depends on the security of two primitives,
+XChaCha12 and AES-256, rather than just one. See the paper
+"Adiantum: length-preserving encryption for entry-level processors"
+(https://eprint.iacr.org/2018/720.pdf) for more details. To use
+Adiantum, CONFIG_CRYPTO_ADIANTUM must be enabled. Also, fast
+implementations of ChaCha and NHPoly1305 should be enabled, e.g.
+CONFIG_CRYPTO_CHACHA20_NEON and CONFIG_CRYPTO_NHPOLY1305_NEON for ARM.
New encryption modes can be added relatively easily, without changes
to individual filesystems. However, authenticated encryption (AE)
modes are not currently supported because of the difficulty of dealing
with ciphertext expansion.
+Contents encryption
+-------------------
+
For file contents, each filesystem block is encrypted independently.
Currently, only the case where the filesystem block size is equal to
-the system's page size (usually 4096 bytes) is supported. With the
-XTS mode of operation (recommended), the logical block number within
-the file is used as the IV. With the CBC mode of operation (not
-recommended), ESSIV is used; specifically, the IV for CBC is the
-logical block number encrypted with AES-256, where the AES-256 key is
-the SHA-256 hash of the inode's data encryption key.
+the system's page size (usually 4096 bytes) is supported.
-For filenames, the full filename is encrypted at once. Because of the
-requirements to retain support for efficient directory lookups and
-filenames of up to 255 bytes, a constant initialization vector (IV) is
-used. However, each encrypted directory uses a unique key, which
-limits IV reuse to within a single directory. Note that IV reuse in
-the context of CTS-CBC encryption means that when the original
-filenames share a common prefix at least as long as the cipher block
-size (16 bytes for AES), the corresponding encrypted filenames will
-also share a common prefix. This is undesirable; it may be fixed in
-the future by switching to an encryption mode that is a strong
-pseudorandom permutation on arbitrary-length messages, e.g. the HEH
-(Hash-Encrypt-Hash) mode.
+Each block's IV is set to the logical block number within the file as
+a little endian number, except that:
-Since filenames are encrypted with the CTS-CBC mode of operation, the
-plaintext and ciphertext filenames need not be multiples of the AES
-block size, i.e. 16 bytes. However, the minimum size that can be
-encrypted is 16 bytes, so shorter filenames are NUL-padded to 16 bytes
-before being encrypted. In addition, to reduce leakage of filename
-lengths via their ciphertexts, all filenames are NUL-padded to the
-next 4, 8, 16, or 32-byte boundary (configurable). 32 is recommended
-since this provides the best confidentiality, at the cost of making
-directory entries consume slightly more space. Note that since NUL
-(``\0``) is not otherwise a valid character in filenames, the padding
-will never produce duplicate plaintexts.
+- With CBC mode encryption, ESSIV is also used. Specifically, each IV
+ is encrypted with AES-256 where the AES-256 key is the SHA-256 hash
+ of the file's data encryption key.
+
+- In the "direct key" configuration (FSCRYPT_POLICY_FLAG_DIRECT_KEY
+ set in the fscrypt_policy), the file's nonce is also appended to the
+ IV. Currently this is only allowed with the Adiantum encryption
+ mode.
+
+Filenames encryption
+--------------------
+
+For filenames, each full filename is encrypted at once. Because of
+the requirements to retain support for efficient directory lookups and
+filenames of up to 255 bytes, the same IV is used for every filename
+in a directory.
+
+However, each encrypted directory still uses a unique key; or
+alternatively (for the "direct key" configuration) has the file's
+nonce included in the IVs. Thus, IV reuse is limited to within a
+single directory.
+
+With CTS-CBC, the IV reuse means that when the plaintext filenames
+share a common prefix at least as long as the cipher block size (16
+bytes for AES), the corresponding encrypted filenames will also share
+a common prefix. This is undesirable. Adiantum does not have this
+weakness, as it is a wide-block encryption mode.
+
+All supported filenames encryption modes accept any plaintext length
+>= 16 bytes; cipher block alignment is not required. However,
+filenames shorter than 16 bytes are NUL-padded to 16 bytes before
+being encrypted. In addition, to reduce leakage of filename lengths
+via their ciphertexts, all filenames are NUL-padded to the next 4, 8,
+16, or 32-byte boundary (configurable). 32 is recommended since this
+provides the best confidentiality, at the cost of making directory
+entries consume slightly more space. Note that since NUL (``\0``) is
+not otherwise a valid character in filenames, the padding will never
+produce duplicate plaintexts.
Symbolic link targets are considered a type of filename and are
-encrypted in the same way as filenames in directory entries. Each
-symlink also uses a unique key; hence, the hardcoded IV is not a
-problem for symlinks.
+encrypted in the same way as filenames in directory entries, except
+that IV reuse is not a problem as each symlink has its own inode.
User API
========
@@ -246,45 +386,77 @@
Setting an encryption policy
----------------------------
+FS_IOC_SET_ENCRYPTION_POLICY
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
The FS_IOC_SET_ENCRYPTION_POLICY ioctl sets an encryption policy on an
empty directory or verifies that a directory or regular file already
has the specified encryption policy. It takes in a pointer to a
-:c:type:`struct fscrypt_policy`, defined as follows::
+:c:type:`struct fscrypt_policy_v1` or a :c:type:`struct
+fscrypt_policy_v2`, defined as follows::
- #define FS_KEY_DESCRIPTOR_SIZE 8
-
- struct fscrypt_policy {
+ #define FSCRYPT_POLICY_V1 0
+ #define FSCRYPT_KEY_DESCRIPTOR_SIZE 8
+ struct fscrypt_policy_v1 {
__u8 version;
__u8 contents_encryption_mode;
__u8 filenames_encryption_mode;
__u8 flags;
- __u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ __u8 master_key_descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE];
+ };
+ #define fscrypt_policy fscrypt_policy_v1
+
+ #define FSCRYPT_POLICY_V2 2
+ #define FSCRYPT_KEY_IDENTIFIER_SIZE 16
+ struct fscrypt_policy_v2 {
+ __u8 version;
+ __u8 contents_encryption_mode;
+ __u8 filenames_encryption_mode;
+ __u8 flags;
+ __u8 __reserved[4];
+ __u8 master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE];
};
This structure must be initialized as follows:
-- ``version`` must be 0.
+- ``version`` must be FSCRYPT_POLICY_V1 (0) if the struct is
+ :c:type:`fscrypt_policy_v1` or FSCRYPT_POLICY_V2 (2) if the struct
+ is :c:type:`fscrypt_policy_v2`. (Note: we refer to the original
+ policy version as "v1", though its version code is really 0.) For
+ new encrypted directories, use v2 policies.
- ``contents_encryption_mode`` and ``filenames_encryption_mode`` must
- be set to constants from ``<linux/fs.h>`` which identify the
- encryption modes to use. If unsure, use
- FS_ENCRYPTION_MODE_AES_256_XTS (1) for ``contents_encryption_mode``
- and FS_ENCRYPTION_MODE_AES_256_CTS (4) for
- ``filenames_encryption_mode``.
+ be set to constants from ``<linux/fscrypt.h>`` which identify the
+ encryption modes to use. If unsure, use FSCRYPT_MODE_AES_256_XTS
+ (1) for ``contents_encryption_mode`` and FSCRYPT_MODE_AES_256_CTS
+ (4) for ``filenames_encryption_mode``.
-- ``flags`` must be set to a value from ``<linux/fs.h>`` which
+- ``flags`` must contain a value from ``<linux/fscrypt.h>`` which
identifies the amount of NUL-padding to use when encrypting
- filenames. If unsure, use FS_POLICY_FLAGS_PAD_32 (0x3).
+ filenames. If unsure, use FSCRYPT_POLICY_FLAGS_PAD_32 (0x3).
+ Additionally, if the encryption modes are both
+ FSCRYPT_MODE_ADIANTUM, this can contain
+ FSCRYPT_POLICY_FLAG_DIRECT_KEY; see `DIRECT_KEY and per-mode keys`_.
-- ``master_key_descriptor`` specifies how to find the master key in
- the keyring; see `Adding keys`_. It is up to userspace to choose a
- unique ``master_key_descriptor`` for each master key. The e4crypt
- and fscrypt tools use the first 8 bytes of
+- For v2 encryption policies, ``__reserved`` must be zeroed.
+
+- For v1 encryption policies, ``master_key_descriptor`` specifies how
+ to find the master key in a keyring; see `Adding keys`_. It is up
+ to userspace to choose a unique ``master_key_descriptor`` for each
+ master key. The e4crypt and fscrypt tools use the first 8 bytes of
``SHA-512(SHA-512(master_key))``, but this particular scheme is not
required. Also, the master key need not be in the keyring yet when
FS_IOC_SET_ENCRYPTION_POLICY is executed. However, it must be added
before any files can be created in the encrypted directory.
+ For v2 encryption policies, ``master_key_descriptor`` has been
+ replaced with ``master_key_identifier``, which is longer and cannot
+ be arbitrarily chosen. Instead, the key must first be added using
+ `FS_IOC_ADD_ENCRYPTION_KEY`_. Then, the ``key_spec.u.identifier``
+ the kernel returned in the :c:type:`struct fscrypt_add_key_arg` must
+ be used as the ``master_key_identifier`` in the :c:type:`struct
+ fscrypt_policy_v2`.
+
If the file is not yet encrypted, then FS_IOC_SET_ENCRYPTION_POLICY
verifies that the file is an empty directory. If so, the specified
encryption policy is assigned to the directory, turning it into an
@@ -300,6 +472,15 @@
returns 0. Otherwise, it fails with EEXIST. This works on both
regular files and directories, including nonempty directories.
+When a v2 encryption policy is assigned to a directory, it is also
+required that either the specified key has been added by the current
+user or that the caller has CAP_FOWNER in the initial user namespace.
+(This is needed to prevent a user from encrypting their data with
+another user's key.) The key must remain added while
+FS_IOC_SET_ENCRYPTION_POLICY is executing. However, if the new
+encrypted directory does not need to be accessed immediately, then the
+key can be removed right away afterwards.
+
Note that the ext4 filesystem does not allow the root directory to be
encrypted, even if it is empty. Users who want to encrypt an entire
filesystem with one key should consider using dm-crypt instead.
@@ -312,15 +493,19 @@
- ``EEXIST``: the file is already encrypted with an encryption policy
different from the one specified
- ``EINVAL``: an invalid encryption policy was specified (invalid
- version, mode(s), or flags)
+ version, mode(s), or flags; or reserved bits were set)
+- ``ENOKEY``: a v2 encryption policy was specified, but the key with
+ the specified ``master_key_identifier`` has not been added, nor does
+ the process have the CAP_FOWNER capability in the initial user
+ namespace
- ``ENOTDIR``: the file is unencrypted and is a regular file, not a
directory
- ``ENOTEMPTY``: the file is unencrypted and is a nonempty directory
- ``ENOTTY``: this type of filesystem does not implement encryption
- ``EOPNOTSUPP``: the kernel was not configured with encryption
- support for this filesystem, or the filesystem superblock has not
+ support for filesystems, or the filesystem superblock has not
had encryption enabled on it. (For example, to use encryption on an
- ext4 filesystem, CONFIG_EXT4_ENCRYPTION must be enabled in the
+ ext4 filesystem, CONFIG_FS_ENCRYPTION must be enabled in the
kernel config, and the superblock must have had the "encrypt"
feature flag enabled using ``tune2fs -O encrypt`` or ``mkfs.ext4 -O
encrypt``.)
@@ -331,25 +516,79 @@
Getting an encryption policy
----------------------------
-The FS_IOC_GET_ENCRYPTION_POLICY ioctl retrieves the :c:type:`struct
-fscrypt_policy`, if any, for a directory or regular file. See above
-for the struct definition. No additional permissions are required
-beyond the ability to open the file.
+Two ioctls are available to get a file's encryption policy:
-FS_IOC_GET_ENCRYPTION_POLICY can fail with the following errors:
+- `FS_IOC_GET_ENCRYPTION_POLICY_EX`_
+- `FS_IOC_GET_ENCRYPTION_POLICY`_
+
+The extended (_EX) version of the ioctl is more general and is
+recommended to use when possible. However, on older kernels only the
+original ioctl is available. Applications should try the extended
+version, and if it fails with ENOTTY fall back to the original
+version.
+
+FS_IOC_GET_ENCRYPTION_POLICY_EX
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The FS_IOC_GET_ENCRYPTION_POLICY_EX ioctl retrieves the encryption
+policy, if any, for a directory or regular file. No additional
+permissions are required beyond the ability to open the file. It
+takes in a pointer to a :c:type:`struct fscrypt_get_policy_ex_arg`,
+defined as follows::
+
+ struct fscrypt_get_policy_ex_arg {
+ __u64 policy_size; /* input/output */
+ union {
+ __u8 version;
+ struct fscrypt_policy_v1 v1;
+ struct fscrypt_policy_v2 v2;
+ } policy; /* output */
+ };
+
+The caller must initialize ``policy_size`` to the size available for
+the policy struct, i.e. ``sizeof(arg.policy)``.
+
+On success, the policy struct is returned in ``policy``, and its
+actual size is returned in ``policy_size``. ``policy.version`` should
+be checked to determine the version of policy returned. Note that the
+version code for the "v1" policy is actually 0 (FSCRYPT_POLICY_V1).
+
+FS_IOC_GET_ENCRYPTION_POLICY_EX can fail with the following errors:
- ``EINVAL``: the file is encrypted, but it uses an unrecognized
- encryption context format
+ encryption policy version
- ``ENODATA``: the file is not encrypted
-- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``ENOTTY``: this type of filesystem does not implement encryption,
+ or this kernel is too old to support FS_IOC_GET_ENCRYPTION_POLICY_EX
+ (try FS_IOC_GET_ENCRYPTION_POLICY instead)
- ``EOPNOTSUPP``: the kernel was not configured with encryption
- support for this filesystem
+ support for this filesystem, or the filesystem superblock has not
+ had encryption enabled on it
+- ``EOVERFLOW``: the file is encrypted and uses a recognized
+ encryption policy version, but the policy struct does not fit into
+ the provided buffer
Note: if you only need to know whether a file is encrypted or not, on
most filesystems it is also possible to use the FS_IOC_GETFLAGS ioctl
and check for FS_ENCRYPT_FL, or to use the statx() system call and
check for STATX_ATTR_ENCRYPTED in stx_attributes.
+FS_IOC_GET_ENCRYPTION_POLICY
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The FS_IOC_GET_ENCRYPTION_POLICY ioctl can also retrieve the
+encryption policy, if any, for a directory or regular file. However,
+unlike `FS_IOC_GET_ENCRYPTION_POLICY_EX`_,
+FS_IOC_GET_ENCRYPTION_POLICY only supports the original policy
+version. It takes in a pointer directly to a :c:type:`struct
+fscrypt_policy_v1` rather than a :c:type:`struct
+fscrypt_get_policy_ex_arg`.
+
+The error codes for FS_IOC_GET_ENCRYPTION_POLICY are the same as those
+for FS_IOC_GET_ENCRYPTION_POLICY_EX, except that
+FS_IOC_GET_ENCRYPTION_POLICY also returns ``EINVAL`` if the file is
+encrypted using a newer encryption policy version.
+
Getting the per-filesystem salt
-------------------------------
@@ -365,8 +604,115 @@
Adding keys
-----------
-To provide a master key, userspace must add it to an appropriate
-keyring using the add_key() system call (see:
+FS_IOC_ADD_ENCRYPTION_KEY
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The FS_IOC_ADD_ENCRYPTION_KEY ioctl adds a master encryption key to
+the filesystem, making all files on the filesystem which were
+encrypted using that key appear "unlocked", i.e. in plaintext form.
+It can be executed on any file or directory on the target filesystem,
+but using the filesystem's root directory is recommended. It takes in
+a pointer to a :c:type:`struct fscrypt_add_key_arg`, defined as
+follows::
+
+ struct fscrypt_add_key_arg {
+ struct fscrypt_key_specifier key_spec;
+ __u32 raw_size;
+ __u32 __reserved[9];
+ __u8 raw[];
+ };
+
+ #define FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR 1
+ #define FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER 2
+
+ struct fscrypt_key_specifier {
+ __u32 type; /* one of FSCRYPT_KEY_SPEC_TYPE_* */
+ __u32 __reserved;
+ union {
+ __u8 __reserved[32]; /* reserve some extra space */
+ __u8 descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE];
+ __u8 identifier[FSCRYPT_KEY_IDENTIFIER_SIZE];
+ } u;
+ };
+
+:c:type:`struct fscrypt_add_key_arg` must be zeroed, then initialized
+as follows:
+
+- If the key is being added for use by v1 encryption policies, then
+ ``key_spec.type`` must contain FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR, and
+ ``key_spec.u.descriptor`` must contain the descriptor of the key
+ being added, corresponding to the value in the
+ ``master_key_descriptor`` field of :c:type:`struct
+ fscrypt_policy_v1`. To add this type of key, the calling process
+ must have the CAP_SYS_ADMIN capability in the initial user
+ namespace.
+
+ Alternatively, if the key is being added for use by v2 encryption
+ policies, then ``key_spec.type`` must contain
+ FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER, and ``key_spec.u.identifier`` is
+ an *output* field which the kernel fills in with a cryptographic
+ hash of the key. To add this type of key, the calling process does
+ not need any privileges. However, the number of keys that can be
+ added is limited by the user's quota for the keyrings service (see
+ ``Documentation/security/keys/core.rst``).
+
+- ``raw_size`` must be the size of the ``raw`` key provided, in bytes.
+
+- ``raw`` is a variable-length field which must contain the actual
+ key, ``raw_size`` bytes long.
+
+For v2 policy keys, the kernel keeps track of which user (identified
+by effective user ID) added the key, and only allows the key to be
+removed by that user --- or by "root", if they use
+`FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS`_.
+
+However, if another user has added the key, it may be desirable to
+prevent that other user from unexpectedly removing it. Therefore,
+FS_IOC_ADD_ENCRYPTION_KEY may also be used to add a v2 policy key
+*again*, even if it's already added by other user(s). In this case,
+FS_IOC_ADD_ENCRYPTION_KEY will just install a claim to the key for the
+current user, rather than actually add the key again (but the raw key
+must still be provided, as a proof of knowledge).
+
+FS_IOC_ADD_ENCRYPTION_KEY returns 0 if either the key or a claim to
+the key was either added or already exists.
+
+FS_IOC_ADD_ENCRYPTION_KEY can fail with the following errors:
+
+- ``EACCES``: FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR was specified, but the
+ caller does not have the CAP_SYS_ADMIN capability in the initial
+ user namespace
+- ``EDQUOT``: the key quota for this user would be exceeded by adding
+ the key
+- ``EINVAL``: invalid key size or key specifier type, or reserved bits
+ were set
+- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``EOPNOTSUPP``: the kernel was not configured with encryption
+ support for this filesystem, or the filesystem superblock has not
+ had encryption enabled on it
+
+Legacy method
+~~~~~~~~~~~~~
+
+For v1 encryption policies, a master encryption key can also be
+provided by adding it to a process-subscribed keyring, e.g. to a
+session keyring, or to a user keyring if the user keyring is linked
+into the session keyring.
+
+This method is deprecated (and not supported for v2 encryption
+policies) for several reasons. First, it cannot be used in
+combination with FS_IOC_REMOVE_ENCRYPTION_KEY (see `Removing keys`_),
+so for removing a key a workaround such as keyctl_unlink() in
+combination with ``sync; echo 2 > /proc/sys/vm/drop_caches`` would
+have to be used. Second, it doesn't match the fact that the
+locked/unlocked status of encrypted files (i.e. whether they appear to
+be in plaintext form or in ciphertext form) is global. This mismatch
+has caused much confusion as well as real problems when processes
+running under different UIDs, such as a ``sudo`` command, need to
+access encrypted files.
+
+Nevertheless, to add a key to one of the process-subscribed keyrings,
+the add_key() system call can be used (see:
``Documentation/security/keys/core.rst``). The key type must be
"logon"; keys of this type are kept in kernel memory and cannot be
read back by userspace. The key description must be "fscrypt:"
@@ -374,12 +720,12 @@
``master_key_descriptor`` that was set in the encryption policy. The
key payload must conform to the following structure::
- #define FS_MAX_KEY_SIZE 64
+ #define FSCRYPT_MAX_KEY_SIZE 64
struct fscrypt_key {
- u32 mode;
- u8 raw[FS_MAX_KEY_SIZE];
- u32 size;
+ __u32 mode;
+ __u8 raw[FSCRYPT_MAX_KEY_SIZE];
+ __u32 size;
};
``mode`` is ignored; just set it to 0. The actual key is provided in
@@ -391,26 +737,194 @@
filesystem-specific prefixes are deprecated and should not be used in
new programs.
-There are several different types of keyrings in which encryption keys
-may be placed, such as a session keyring, a user session keyring, or a
-user keyring. Each key must be placed in a keyring that is "attached"
-to all processes that might need to access files encrypted with it, in
-the sense that request_key() will find the key. Generally, if only
-processes belonging to a specific user need to access a given
-encrypted directory and no session keyring has been installed, then
-that directory's key should be placed in that user's user session
-keyring or user keyring. Otherwise, a session keyring should be
-installed if needed, and the key should be linked into that session
-keyring, or in a keyring linked into that session keyring.
+Removing keys
+-------------
-Note: introducing the complex visibility semantics of keyrings here
-was arguably a mistake --- especially given that by design, after any
-process successfully opens an encrypted file (thereby setting up the
-per-file key), possessing the keyring key is not actually required for
-any process to read/write the file until its in-memory inode is
-evicted. In the future there probably should be a way to provide keys
-directly to the filesystem instead, which would make the intended
-semantics clearer.
+Two ioctls are available for removing a key that was added by
+`FS_IOC_ADD_ENCRYPTION_KEY`_:
+
+- `FS_IOC_REMOVE_ENCRYPTION_KEY`_
+- `FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS`_
+
+These two ioctls differ only in cases where v2 policy keys are added
+or removed by non-root users.
+
+These ioctls don't work on keys that were added via the legacy
+process-subscribed keyrings mechanism.
+
+Before using these ioctls, read the `Kernel memory compromise`_
+section for a discussion of the security goals and limitations of
+these ioctls.
+
+FS_IOC_REMOVE_ENCRYPTION_KEY
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The FS_IOC_REMOVE_ENCRYPTION_KEY ioctl removes a claim to a master
+encryption key from the filesystem, and possibly removes the key
+itself. It can be executed on any file or directory on the target
+filesystem, but using the filesystem's root directory is recommended.
+It takes in a pointer to a :c:type:`struct fscrypt_remove_key_arg`,
+defined as follows::
+
+ struct fscrypt_remove_key_arg {
+ struct fscrypt_key_specifier key_spec;
+ #define FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY 0x00000001
+ #define FSCRYPT_KEY_REMOVAL_STATUS_FLAG_OTHER_USERS 0x00000002
+ __u32 removal_status_flags; /* output */
+ __u32 __reserved[5];
+ };
+
+This structure must be zeroed, then initialized as follows:
+
+- The key to remove is specified by ``key_spec``:
+
+ - To remove a key used by v1 encryption policies, set
+ ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR and fill
+ in ``key_spec.u.descriptor``. To remove this type of key, the
+ calling process must have the CAP_SYS_ADMIN capability in the
+ initial user namespace.
+
+ - To remove a key used by v2 encryption policies, set
+ ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER and fill
+ in ``key_spec.u.identifier``.
+
+For v2 policy keys, this ioctl is usable by non-root users. However,
+to make this possible, it actually just removes the current user's
+claim to the key, undoing a single call to FS_IOC_ADD_ENCRYPTION_KEY.
+Only after all claims are removed is the key really removed.
+
+For example, if FS_IOC_ADD_ENCRYPTION_KEY was called with uid 1000,
+then the key will be "claimed" by uid 1000, and
+FS_IOC_REMOVE_ENCRYPTION_KEY will only succeed as uid 1000. Or, if
+both uids 1000 and 2000 added the key, then for each uid
+FS_IOC_REMOVE_ENCRYPTION_KEY will only remove their own claim. Only
+once *both* are removed is the key really removed. (Think of it like
+unlinking a file that may have hard links.)
+
+If FS_IOC_REMOVE_ENCRYPTION_KEY really removes the key, it will also
+try to "lock" all files that had been unlocked with the key. It won't
+lock files that are still in-use, so this ioctl is expected to be used
+in cooperation with userspace ensuring that none of the files are
+still open. However, if necessary, this ioctl can be executed again
+later to retry locking any remaining files.
+
+FS_IOC_REMOVE_ENCRYPTION_KEY returns 0 if either the key was removed
+(but may still have files remaining to be locked), the user's claim to
+the key was removed, or the key was already removed but had files
+remaining to be the locked so the ioctl retried locking them. In any
+of these cases, ``removal_status_flags`` is filled in with the
+following informational status flags:
+
+- ``FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY``: set if some file(s)
+ are still in-use. Not guaranteed to be set in the case where only
+ the user's claim to the key was removed.
+- ``FSCRYPT_KEY_REMOVAL_STATUS_FLAG_OTHER_USERS``: set if only the
+ user's claim to the key was removed, not the key itself
+
+FS_IOC_REMOVE_ENCRYPTION_KEY can fail with the following errors:
+
+- ``EACCES``: The FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR key specifier type
+ was specified, but the caller does not have the CAP_SYS_ADMIN
+ capability in the initial user namespace
+- ``EINVAL``: invalid key specifier type, or reserved bits were set
+- ``ENOKEY``: the key object was not found at all, i.e. it was never
+ added in the first place or was already fully removed including all
+ files locked; or, the user does not have a claim to the key (but
+ someone else does).
+- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``EOPNOTSUPP``: the kernel was not configured with encryption
+ support for this filesystem, or the filesystem superblock has not
+ had encryption enabled on it
+
+FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS is exactly the same as
+`FS_IOC_REMOVE_ENCRYPTION_KEY`_, except that for v2 policy keys, the
+ALL_USERS version of the ioctl will remove all users' claims to the
+key, not just the current user's. I.e., the key itself will always be
+removed, no matter how many users have added it. This difference is
+only meaningful if non-root users are adding and removing keys.
+
+Because of this, FS_IOC_REMOVE_ENCRYPTION_KEY_ALL_USERS also requires
+"root", namely the CAP_SYS_ADMIN capability in the initial user
+namespace. Otherwise it will fail with EACCES.
+
+Getting key status
+------------------
+
+FS_IOC_GET_ENCRYPTION_KEY_STATUS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The FS_IOC_GET_ENCRYPTION_KEY_STATUS ioctl retrieves the status of a
+master encryption key. It can be executed on any file or directory on
+the target filesystem, but using the filesystem's root directory is
+recommended. It takes in a pointer to a :c:type:`struct
+fscrypt_get_key_status_arg`, defined as follows::
+
+ struct fscrypt_get_key_status_arg {
+ /* input */
+ struct fscrypt_key_specifier key_spec;
+ __u32 __reserved[6];
+
+ /* output */
+ #define FSCRYPT_KEY_STATUS_ABSENT 1
+ #define FSCRYPT_KEY_STATUS_PRESENT 2
+ #define FSCRYPT_KEY_STATUS_INCOMPLETELY_REMOVED 3
+ __u32 status;
+ #define FSCRYPT_KEY_STATUS_FLAG_ADDED_BY_SELF 0x00000001
+ __u32 status_flags;
+ __u32 user_count;
+ __u32 __out_reserved[13];
+ };
+
+The caller must zero all input fields, then fill in ``key_spec``:
+
+ - To get the status of a key for v1 encryption policies, set
+ ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_DESCRIPTOR and fill
+ in ``key_spec.u.descriptor``.
+
+ - To get the status of a key for v2 encryption policies, set
+ ``key_spec.type`` to FSCRYPT_KEY_SPEC_TYPE_IDENTIFIER and fill
+ in ``key_spec.u.identifier``.
+
+On success, 0 is returned and the kernel fills in the output fields:
+
+- ``status`` indicates whether the key is absent, present, or
+ incompletely removed. Incompletely removed means that the master
+ secret has been removed, but some files are still in use; i.e.,
+ `FS_IOC_REMOVE_ENCRYPTION_KEY`_ returned 0 but set the informational
+ status flag FSCRYPT_KEY_REMOVAL_STATUS_FLAG_FILES_BUSY.
+
+- ``status_flags`` can contain the following flags:
+
+ - ``FSCRYPT_KEY_STATUS_FLAG_ADDED_BY_SELF`` indicates that the key
+ has added by the current user. This is only set for keys
+ identified by ``identifier`` rather than by ``descriptor``.
+
+- ``user_count`` specifies the number of users who have added the key.
+ This is only set for keys identified by ``identifier`` rather than
+ by ``descriptor``.
+
+FS_IOC_GET_ENCRYPTION_KEY_STATUS can fail with the following errors:
+
+- ``EINVAL``: invalid key specifier type, or reserved bits were set
+- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``EOPNOTSUPP``: the kernel was not configured with encryption
+ support for this filesystem, or the filesystem superblock has not
+ had encryption enabled on it
+
+Among other use cases, FS_IOC_GET_ENCRYPTION_KEY_STATUS can be useful
+for determining whether the key for a given encrypted directory needs
+to be added before prompting the user for the passphrase needed to
+derive the key.
+
+FS_IOC_GET_ENCRYPTION_KEY_STATUS can only get the status of keys in
+the filesystem-level keyring, i.e. the keyring managed by
+`FS_IOC_ADD_ENCRYPTION_KEY`_ and `FS_IOC_REMOVE_ENCRYPTION_KEY`_. It
+cannot get the status of a key that has only been added for use by v1
+encryption policies using the legacy mechanism involving
+process-subscribed keyrings.
Access semantics
================
@@ -426,10 +940,18 @@
- Unencrypted files, or files encrypted with a different encryption
policy (i.e. different key, modes, or flags), cannot be renamed or
linked into an encrypted directory; see `Encryption policy
- enforcement`_. Attempts to do so will fail with EPERM. However,
+ enforcement`_. Attempts to do so will fail with EXDEV. However,
encrypted files can be renamed within an encrypted directory, or
into an unencrypted directory.
+ Note: "moving" an unencrypted file into an encrypted directory, e.g.
+ with the `mv` program, is implemented in userspace by a copy
+ followed by a delete. Be aware that the original unencrypted data
+ may remain recoverable from free space on the disk; prefer to keep
+ all files encrypted from the very beginning. The `shred` program
+ may be used to overwrite the source files but isn't guaranteed to be
+ effective on all filesystems and storage devices.
+
- Direct I/O is not supported on encrypted files. Attempts to use
direct I/O on such files will fall back to buffered I/O.
@@ -465,7 +987,7 @@
Some filesystem operations may be performed on encrypted regular
files, directories, and symlinks even before their encryption key has
-been provided:
+been added, or after their encryption key has been removed:
- File metadata may be read, e.g. using stat().
@@ -516,7 +1038,7 @@
Except for those special files, it is forbidden to have unencrypted
files, or files encrypted with a different encryption policy, in an
encrypted directory tree. Attempts to link or rename such a file into
-an encrypted directory will fail with EPERM. This is also enforced
+an encrypted directory will fail with EXDEV. This is also enforced
during ->lookup() to provide limited protection against offline
attacks that try to disable or downgrade encryption in known locations
where applications may later write sensitive data. It is recommended
@@ -530,33 +1052,44 @@
------------------
An encryption policy is represented on-disk by a :c:type:`struct
-fscrypt_context`. It is up to individual filesystems to decide where
-to store it, but normally it would be stored in a hidden extended
-attribute. It should *not* be exposed by the xattr-related system
-calls such as getxattr() and setxattr() because of the special
-semantics of the encryption xattr. (In particular, there would be
-much confusion if an encryption policy were to be added to or removed
-from anything other than an empty directory.) The struct is defined
-as follows::
+fscrypt_context_v1` or a :c:type:`struct fscrypt_context_v2`. It is
+up to individual filesystems to decide where to store it, but normally
+it would be stored in a hidden extended attribute. It should *not* be
+exposed by the xattr-related system calls such as getxattr() and
+setxattr() because of the special semantics of the encryption xattr.
+(In particular, there would be much confusion if an encryption policy
+were to be added to or removed from anything other than an empty
+directory.) These structs are defined as follows::
- #define FS_KEY_DESCRIPTOR_SIZE 8
#define FS_KEY_DERIVATION_NONCE_SIZE 16
- struct fscrypt_context {
- u8 format;
+ #define FSCRYPT_KEY_DESCRIPTOR_SIZE 8
+ struct fscrypt_context_v1 {
+ u8 version;
u8 contents_encryption_mode;
u8 filenames_encryption_mode;
u8 flags;
- u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ u8 master_key_descriptor[FSCRYPT_KEY_DESCRIPTOR_SIZE];
u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE];
};
-Note that :c:type:`struct fscrypt_context` contains the same
-information as :c:type:`struct fscrypt_policy` (see `Setting an
-encryption policy`_), except that :c:type:`struct fscrypt_context`
-also contains a nonce. The nonce is randomly generated by the kernel
-and is used to derive the inode's encryption key as described in
-`Per-file keys`_.
+ #define FSCRYPT_KEY_IDENTIFIER_SIZE 16
+ struct fscrypt_context_v2 {
+ u8 version;
+ u8 contents_encryption_mode;
+ u8 filenames_encryption_mode;
+ u8 flags;
+ u8 __reserved[4];
+ u8 master_key_identifier[FSCRYPT_KEY_IDENTIFIER_SIZE];
+ u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE];
+ };
+
+The context structs contain the same information as the corresponding
+policy structs (see `Setting an encryption policy`_), except that the
+context structs also contain a nonce. The nonce is randomly generated
+by the kernel and is used as KDF input or as a tweak to cause
+different files to be encrypted differently; see `Per-file keys`_ and
+`DIRECT_KEY and per-mode keys`_.
Data path changes
-----------------
@@ -614,3 +1147,42 @@
without the key is subject to change in the future. It is only meant
as a way to temporarily present valid filenames so that commands like
``rm -r`` work as expected on encrypted directories.
+
+Tests
+=====
+
+To test fscrypt, use xfstests, which is Linux's de facto standard
+filesystem test suite. First, run all the tests in the "encrypt"
+group on the relevant filesystem(s). For example, to test ext4 and
+f2fs encryption using `kvm-xfstests
+<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
+
+ kvm-xfstests -c ext4,f2fs -g encrypt
+
+UBIFS encryption can also be tested this way, but it should be done in
+a separate command, and it takes some time for kvm-xfstests to set up
+emulated UBI volumes::
+
+ kvm-xfstests -c ubifs -g encrypt
+
+No tests should fail. However, tests that use non-default encryption
+modes (e.g. generic/549 and generic/550) will be skipped if the needed
+algorithms were not built into the kernel's crypto API. Also, tests
+that access the raw block device (e.g. generic/399, generic/548,
+generic/549, generic/550) will be skipped on UBIFS.
+
+Besides running the "encrypt" group tests, for ext4 and f2fs it's also
+possible to run most xfstests with the "test_dummy_encryption" mount
+option. This option causes all new files to be automatically
+encrypted with a dummy key, without having to make any API calls.
+This tests the encrypted I/O paths more thoroughly. To do this with
+kvm-xfstests, use the "encrypt" filesystem configuration::
+
+ kvm-xfstests -c ext4/encrypt,f2fs/encrypt -g auto
+
+Because this runs many more tests than "-g encrypt" does, it takes
+much longer to run; so also consider using `gce-xfstests
+<https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md>`_
+instead of kvm-xfstests::
+
+ gce-xfstests -c ext4/encrypt,f2fs/encrypt -g auto
diff --git a/Documentation/filesystems/fsverity.rst b/Documentation/filesystems/fsverity.rst
new file mode 100644
index 0000000..42a0b6d
--- /dev/null
+++ b/Documentation/filesystems/fsverity.rst
@@ -0,0 +1,726 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _fsverity:
+
+=======================================================
+fs-verity: read-only file-based authenticity protection
+=======================================================
+
+Introduction
+============
+
+fs-verity (``fs/verity/``) is a support layer that filesystems can
+hook into to support transparent integrity and authenticity protection
+of read-only files. Currently, it is supported by the ext4 and f2fs
+filesystems. Like fscrypt, not too much filesystem-specific code is
+needed to support fs-verity.
+
+fs-verity is similar to `dm-verity
+<https://www.kernel.org/doc/Documentation/device-mapper/verity.txt>`_
+but works on files rather than block devices. On regular files on
+filesystems supporting fs-verity, userspace can execute an ioctl that
+causes the filesystem to build a Merkle tree for the file and persist
+it to a filesystem-specific location associated with the file.
+
+After this, the file is made readonly, and all reads from the file are
+automatically verified against the file's Merkle tree. Reads of any
+corrupted data, including mmap reads, will fail.
+
+Userspace can use another ioctl to retrieve the root hash (actually
+the "file measurement", which is a hash that includes the root hash)
+that fs-verity is enforcing for the file. This ioctl executes in
+constant time, regardless of the file size.
+
+fs-verity is essentially a way to hash a file in constant time,
+subject to the caveat that reads which would violate the hash will
+fail at runtime.
+
+Use cases
+=========
+
+By itself, the base fs-verity feature only provides integrity
+protection, i.e. detection of accidental (non-malicious) corruption.
+
+However, because fs-verity makes retrieving the file hash extremely
+efficient, it's primarily meant to be used as a tool to support
+authentication (detection of malicious modifications) or auditing
+(logging file hashes before use).
+
+Trusted userspace code (e.g. operating system code running on a
+read-only partition that is itself authenticated by dm-verity) can
+authenticate the contents of an fs-verity file by using the
+`FS_IOC_MEASURE_VERITY`_ ioctl to retrieve its hash, then verifying a
+digital signature of it.
+
+A standard file hash could be used instead of fs-verity. However,
+this is inefficient if the file is large and only a small portion may
+be accessed. This is often the case for Android application package
+(APK) files, for example. These typically contain many translations,
+classes, and other resources that are infrequently or even never
+accessed on a particular device. It would be slow and wasteful to
+read and hash the entire file before starting the application.
+
+Unlike an ahead-of-time hash, fs-verity also re-verifies data each
+time it's paged in. This ensures that malicious disk firmware can't
+undetectably change the contents of the file at runtime.
+
+fs-verity does not replace or obsolete dm-verity. dm-verity should
+still be used on read-only filesystems. fs-verity is for files that
+must live on a read-write filesystem because they are independently
+updated and potentially user-installed, so dm-verity cannot be used.
+
+The base fs-verity feature is a hashing mechanism only; actually
+authenticating the files is up to userspace. However, to meet some
+users' needs, fs-verity optionally supports a simple signature
+verification mechanism where users can configure the kernel to require
+that all fs-verity files be signed by a key loaded into a keyring; see
+`Built-in signature verification`_. Support for fs-verity file hashes
+in IMA (Integrity Measurement Architecture) policies is also planned.
+
+User API
+========
+
+FS_IOC_ENABLE_VERITY
+--------------------
+
+The FS_IOC_ENABLE_VERITY ioctl enables fs-verity on a file. It takes
+in a pointer to a :c:type:`struct fsverity_enable_arg`, defined as
+follows::
+
+ struct fsverity_enable_arg {
+ __u32 version;
+ __u32 hash_algorithm;
+ __u32 block_size;
+ __u32 salt_size;
+ __u64 salt_ptr;
+ __u32 sig_size;
+ __u32 __reserved1;
+ __u64 sig_ptr;
+ __u64 __reserved2[11];
+ };
+
+This structure contains the parameters of the Merkle tree to build for
+the file, and optionally contains a signature. It must be initialized
+as follows:
+
+- ``version`` must be 1.
+- ``hash_algorithm`` must be the identifier for the hash algorithm to
+ use for the Merkle tree, such as FS_VERITY_HASH_ALG_SHA256. See
+ ``include/uapi/linux/fsverity.h`` for the list of possible values.
+- ``block_size`` must be the Merkle tree block size. Currently, this
+ must be equal to the system page size, which is usually 4096 bytes.
+ Other sizes may be supported in the future. This value is not
+ necessarily the same as the filesystem block size.
+- ``salt_size`` is the size of the salt in bytes, or 0 if no salt is
+ provided. The salt is a value that is prepended to every hashed
+ block; it can be used to personalize the hashing for a particular
+ file or device. Currently the maximum salt size is 32 bytes.
+- ``salt_ptr`` is the pointer to the salt, or NULL if no salt is
+ provided.
+- ``sig_size`` is the size of the signature in bytes, or 0 if no
+ signature is provided. Currently the signature is (somewhat
+ arbitrarily) limited to 16128 bytes. See `Built-in signature
+ verification`_ for more information.
+- ``sig_ptr`` is the pointer to the signature, or NULL if no
+ signature is provided.
+- All reserved fields must be zeroed.
+
+FS_IOC_ENABLE_VERITY causes the filesystem to build a Merkle tree for
+the file and persist it to a filesystem-specific location associated
+with the file, then mark the file as a verity file. This ioctl may
+take a long time to execute on large files, and it is interruptible by
+fatal signals.
+
+FS_IOC_ENABLE_VERITY checks for write access to the inode. However,
+it must be executed on an O_RDONLY file descriptor and no processes
+can have the file open for writing. Attempts to open the file for
+writing while this ioctl is executing will fail with ETXTBSY. (This
+is necessary to guarantee that no writable file descriptors will exist
+after verity is enabled, and to guarantee that the file's contents are
+stable while the Merkle tree is being built over it.)
+
+On success, FS_IOC_ENABLE_VERITY returns 0, and the file becomes a
+verity file. On failure (including the case of interruption by a
+fatal signal), no changes are made to the file.
+
+FS_IOC_ENABLE_VERITY can fail with the following errors:
+
+- ``EACCES``: the process does not have write access to the file
+- ``EBADMSG``: the signature is malformed
+- ``EBUSY``: this ioctl is already running on the file
+- ``EEXIST``: the file already has verity enabled
+- ``EFAULT``: the caller provided inaccessible memory
+- ``EINTR``: the operation was interrupted by a fatal signal
+- ``EINVAL``: unsupported version, hash algorithm, or block size; or
+ reserved bits are set; or the file descriptor refers to neither a
+ regular file nor a directory.
+- ``EISDIR``: the file descriptor refers to a directory
+- ``EKEYREJECTED``: the signature doesn't match the file
+- ``EMSGSIZE``: the salt or signature is too long
+- ``ENOKEY``: the fs-verity keyring doesn't contain the certificate
+ needed to verify the signature
+- ``ENOPKG``: fs-verity recognizes the hash algorithm, but it's not
+ available in the kernel's crypto API as currently configured (e.g.
+ for SHA-512, missing CONFIG_CRYPTO_SHA512).
+- ``ENOTTY``: this type of filesystem does not implement fs-verity
+- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
+ support; or the filesystem superblock has not had the 'verity'
+ feature enabled on it; or the filesystem does not support fs-verity
+ on this file. (See `Filesystem support`_.)
+- ``EPERM``: the file is append-only; or, a signature is required and
+ one was not provided.
+- ``EROFS``: the filesystem is read-only
+- ``ETXTBSY``: someone has the file open for writing. This can be the
+ caller's file descriptor, another open file descriptor, or the file
+ reference held by a writable memory map.
+
+FS_IOC_MEASURE_VERITY
+---------------------
+
+The FS_IOC_MEASURE_VERITY ioctl retrieves the measurement of a verity
+file. The file measurement is a digest that cryptographically
+identifies the file contents that are being enforced on reads.
+
+This ioctl takes in a pointer to a variable-length structure::
+
+ struct fsverity_digest {
+ __u16 digest_algorithm;
+ __u16 digest_size; /* input/output */
+ __u8 digest[];
+ };
+
+``digest_size`` is an input/output field. On input, it must be
+initialized to the number of bytes allocated for the variable-length
+``digest`` field.
+
+On success, 0 is returned and the kernel fills in the structure as
+follows:
+
+- ``digest_algorithm`` will be the hash algorithm used for the file
+ measurement. It will match ``fsverity_enable_arg::hash_algorithm``.
+- ``digest_size`` will be the size of the digest in bytes, e.g. 32
+ for SHA-256. (This can be redundant with ``digest_algorithm``.)
+- ``digest`` will be the actual bytes of the digest.
+
+FS_IOC_MEASURE_VERITY is guaranteed to execute in constant time,
+regardless of the size of the file.
+
+FS_IOC_MEASURE_VERITY can fail with the following errors:
+
+- ``EFAULT``: the caller provided inaccessible memory
+- ``ENODATA``: the file is not a verity file
+- ``ENOTTY``: this type of filesystem does not implement fs-verity
+- ``EOPNOTSUPP``: the kernel was not configured with fs-verity
+ support, or the filesystem superblock has not had the 'verity'
+ feature enabled on it. (See `Filesystem support`_.)
+- ``EOVERFLOW``: the digest is longer than the specified
+ ``digest_size`` bytes. Try providing a larger buffer.
+
+FS_IOC_GETFLAGS
+---------------
+
+The existing ioctl FS_IOC_GETFLAGS (which isn't specific to fs-verity)
+can also be used to check whether a file has fs-verity enabled or not.
+To do so, check for FS_VERITY_FL (0x00100000) in the returned flags.
+
+The verity flag is not settable via FS_IOC_SETFLAGS. You must use
+FS_IOC_ENABLE_VERITY instead, since parameters must be provided.
+
+Accessing verity files
+======================
+
+Applications can transparently access a verity file just like a
+non-verity one, with the following exceptions:
+
+- Verity files are readonly. They cannot be opened for writing or
+ truncate()d, even if the file mode bits allow it. Attempts to do
+ one of these things will fail with EPERM. However, changes to
+ metadata such as owner, mode, timestamps, and xattrs are still
+ allowed, since these are not measured by fs-verity. Verity files
+ can also still be renamed, deleted, and linked to.
+
+- Direct I/O is not supported on verity files. Attempts to use direct
+ I/O on such files will fall back to buffered I/O.
+
+- DAX (Direct Access) is not supported on verity files, because this
+ would circumvent the data verification.
+
+- Reads of data that doesn't match the verity Merkle tree will fail
+ with EIO (for read()) or SIGBUS (for mmap() reads).
+
+- If the sysctl "fs.verity.require_signatures" is set to 1 and the
+ file's verity measurement is not signed by a key in the fs-verity
+ keyring, then opening the file will fail. See `Built-in signature
+ verification`_.
+
+Direct access to the Merkle tree is not supported. Therefore, if a
+verity file is copied, or is backed up and restored, then it will lose
+its "verity"-ness. fs-verity is primarily meant for files like
+executables that are managed by a package manager.
+
+File measurement computation
+============================
+
+This section describes how fs-verity hashes the file contents using a
+Merkle tree to produce the "file measurement" which cryptographically
+identifies the file contents. This algorithm is the same for all
+filesystems that support fs-verity.
+
+Userspace only needs to be aware of this algorithm if it needs to
+compute the file measurement itself, e.g. in order to sign the file.
+
+.. _fsverity_merkle_tree:
+
+Merkle tree
+-----------
+
+The file contents is divided into blocks, where the block size is
+configurable but is usually 4096 bytes. The end of the last block is
+zero-padded if needed. Each block is then hashed, producing the first
+level of hashes. Then, the hashes in this first level are grouped
+into 'blocksize'-byte blocks (zero-padding the ends as needed) and
+these blocks are hashed, producing the second level of hashes. This
+proceeds up the tree until only a single block remains. The hash of
+this block is the "Merkle tree root hash".
+
+If the file fits in one block and is nonempty, then the "Merkle tree
+root hash" is simply the hash of the single data block. If the file
+is empty, then the "Merkle tree root hash" is all zeroes.
+
+The "blocks" here are not necessarily the same as "filesystem blocks".
+
+If a salt was specified, then it's zero-padded to the closest multiple
+of the input size of the hash algorithm's compression function, e.g.
+64 bytes for SHA-256 or 128 bytes for SHA-512. The padded salt is
+prepended to every data or Merkle tree block that is hashed.
+
+The purpose of the block padding is to cause every hash to be taken
+over the same amount of data, which simplifies the implementation and
+keeps open more possibilities for hardware acceleration. The purpose
+of the salt padding is to make the salting "free" when the salted hash
+state is precomputed, then imported for each hash.
+
+Example: in the recommended configuration of SHA-256 and 4K blocks,
+128 hash values fit in each block. Thus, each level of the Merkle
+tree is approximately 128 times smaller than the previous, and for
+large files the Merkle tree's size converges to approximately 1/127 of
+the original file size. However, for small files, the padding is
+significant, making the space overhead proportionally more.
+
+.. _fsverity_descriptor:
+
+fs-verity descriptor
+--------------------
+
+By itself, the Merkle tree root hash is ambiguous. For example, it
+can't a distinguish a large file from a small second file whose data
+is exactly the top-level hash block of the first file. Ambiguities
+also arise from the convention of padding to the next block boundary.
+
+To solve this problem, the verity file measurement is actually
+computed as a hash of the following structure, which contains the
+Merkle tree root hash as well as other fields such as the file size::
+
+ struct fsverity_descriptor {
+ __u8 version; /* must be 1 */
+ __u8 hash_algorithm; /* Merkle tree hash algorithm */
+ __u8 log_blocksize; /* log2 of size of data and tree blocks */
+ __u8 salt_size; /* size of salt in bytes; 0 if none */
+ __le32 sig_size; /* must be 0 */
+ __le64 data_size; /* size of file the Merkle tree is built over */
+ __u8 root_hash[64]; /* Merkle tree root hash */
+ __u8 salt[32]; /* salt prepended to each hashed block */
+ __u8 __reserved[144]; /* must be 0's */
+ };
+
+Note that the ``sig_size`` field must be set to 0 for the purpose of
+computing the file measurement, even if a signature was provided (or
+will be provided) to `FS_IOC_ENABLE_VERITY`_.
+
+Built-in signature verification
+===============================
+
+With CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y, fs-verity supports putting
+a portion of an authentication policy (see `Use cases`_) in the
+kernel. Specifically, it adds support for:
+
+1. At fs-verity module initialization time, a keyring ".fs-verity" is
+ created. The root user can add trusted X.509 certificates to this
+ keyring using the add_key() system call, then (when done)
+ optionally use keyctl_restrict_keyring() to prevent additional
+ certificates from being added.
+
+2. `FS_IOC_ENABLE_VERITY`_ accepts a pointer to a PKCS#7 formatted
+ detached signature in DER format of the file measurement. On
+ success, this signature is persisted alongside the Merkle tree.
+ Then, any time the file is opened, the kernel will verify the
+ file's actual measurement against this signature, using the
+ certificates in the ".fs-verity" keyring.
+
+3. A new sysctl "fs.verity.require_signatures" is made available.
+ When set to 1, the kernel requires that all verity files have a
+ correctly signed file measurement as described in (2).
+
+File measurements must be signed in the following format, which is
+similar to the structure used by `FS_IOC_MEASURE_VERITY`_::
+
+ struct fsverity_signed_digest {
+ char magic[8]; /* must be "FSVerity" */
+ __le16 digest_algorithm;
+ __le16 digest_size;
+ __u8 digest[];
+ };
+
+fs-verity's built-in signature verification support is meant as a
+relatively simple mechanism that can be used to provide some level of
+authenticity protection for verity files, as an alternative to doing
+the signature verification in userspace or using IMA-appraisal.
+However, with this mechanism, userspace programs still need to check
+that the verity bit is set, and there is no protection against verity
+files being swapped around.
+
+Filesystem support
+==================
+
+fs-verity is currently supported by the ext4 and f2fs filesystems.
+The CONFIG_FS_VERITY kconfig option must be enabled to use fs-verity
+on either filesystem.
+
+``include/linux/fsverity.h`` declares the interface between the
+``fs/verity/`` support layer and filesystems. Briefly, filesystems
+must provide an ``fsverity_operations`` structure that provides
+methods to read and write the verity metadata to a filesystem-specific
+location, including the Merkle tree blocks and
+``fsverity_descriptor``. Filesystems must also call functions in
+``fs/verity/`` at certain times, such as when a file is opened or when
+pages have been read into the pagecache. (See `Verifying data`_.)
+
+ext4
+----
+
+ext4 supports fs-verity since Linux TODO and e2fsprogs v1.45.2.
+
+To create verity files on an ext4 filesystem, the filesystem must have
+been formatted with ``-O verity`` or had ``tune2fs -O verity`` run on
+it. "verity" is an RO_COMPAT filesystem feature, so once set, old
+kernels will only be able to mount the filesystem readonly, and old
+versions of e2fsck will be unable to check the filesystem. Moreover,
+currently ext4 only supports mounting a filesystem with the "verity"
+feature when its block size is equal to PAGE_SIZE (often 4096 bytes).
+
+ext4 sets the EXT4_VERITY_FL on-disk inode flag on verity files. It
+can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be cleared.
+
+ext4 also supports encryption, which can be used simultaneously with
+fs-verity. In this case, the plaintext data is verified rather than
+the ciphertext. This is necessary in order to make the file
+measurement meaningful, since every file is encrypted differently.
+
+ext4 stores the verity metadata (Merkle tree and fsverity_descriptor)
+past the end of the file, starting at the first 64K boundary beyond
+i_size. This approach works because (a) verity files are readonly,
+and (b) pages fully beyond i_size aren't visible to userspace but can
+be read/written internally by ext4 with only some relatively small
+changes to ext4. This approach avoids having to depend on the
+EA_INODE feature and on rearchitecturing ext4's xattr support to
+support paging multi-gigabyte xattrs into memory, and to support
+encrypting xattrs. Note that the verity metadata *must* be encrypted
+when the file is, since it contains hashes of the plaintext data.
+
+Currently, ext4 verity only supports the case where the Merkle tree
+block size, filesystem block size, and page size are all the same. It
+also only supports extent-based files.
+
+f2fs
+----
+
+f2fs supports fs-verity since Linux TODO and f2fs-tools v1.11.0.
+
+To create verity files on an f2fs filesystem, the filesystem must have
+been formatted with ``-O verity``.
+
+f2fs sets the FADVISE_VERITY_BIT on-disk inode flag on verity files.
+It can only be set by `FS_IOC_ENABLE_VERITY`_, and it cannot be
+cleared.
+
+Like ext4, f2fs stores the verity metadata (Merkle tree and
+fsverity_descriptor) past the end of the file, starting at the first
+64K boundary beyond i_size. See explanation for ext4 above.
+Moreover, f2fs supports at most 4096 bytes of xattr entries per inode
+which wouldn't be enough for even a single Merkle tree block.
+
+Currently, f2fs verity only supports a Merkle tree block size of 4096.
+Also, f2fs doesn't support enabling verity on files that currently
+have atomic or volatile writes pending.
+
+Implementation details
+======================
+
+Verifying data
+--------------
+
+fs-verity ensures that all reads of a verity file's data are verified,
+regardless of which syscall is used to do the read (e.g. mmap(),
+read(), pread()) and regardless of whether it's the first read or a
+later read (unless the later read can return cached data that was
+already verified). Below, we describe how filesystems implement this.
+
+Pagecache
+~~~~~~~~~
+
+For filesystems using Linux's pagecache, the ``->readpage()`` and
+``->readpages()`` methods must be modified to verify pages before they
+are marked Uptodate. Merely hooking ``->read_iter()`` would be
+insufficient, since ``->read_iter()`` is not used for memory maps.
+
+Therefore, fs/verity/ provides a function fsverity_verify_page() which
+verifies a page that has been read into the pagecache of a verity
+inode, but is still locked and not Uptodate, so it's not yet readable
+by userspace. As needed to do the verification,
+fsverity_verify_page() will call back into the filesystem to read
+Merkle tree pages via fsverity_operations::read_merkle_tree_page().
+
+fsverity_verify_page() returns false if verification failed; in this
+case, the filesystem must not set the page Uptodate. Following this,
+as per the usual Linux pagecache behavior, attempts by userspace to
+read() from the part of the file containing the page will fail with
+EIO, and accesses to the page within a memory map will raise SIGBUS.
+
+fsverity_verify_page() currently only supports the case where the
+Merkle tree block size is equal to PAGE_SIZE (often 4096 bytes).
+
+In principle, fsverity_verify_page() verifies the entire path in the
+Merkle tree from the data page to the root hash. However, for
+efficiency the filesystem may cache the hash pages. Therefore,
+fsverity_verify_page() only ascends the tree reading hash pages until
+an already-verified hash page is seen, as indicated by the PageChecked
+bit being set. It then verifies the path to that page.
+
+This optimization, which is also used by dm-verity, results in
+excellent sequential read performance. This is because usually (e.g.
+127 in 128 times for 4K blocks and SHA-256) the hash page from the
+bottom level of the tree will already be cached and checked from
+reading a previous data page. However, random reads perform worse.
+
+Block device based filesystems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Block device based filesystems (e.g. ext4 and f2fs) in Linux also use
+the pagecache, so the above subsection applies too. However, they
+also usually read many pages from a file at once, grouped into a
+structure called a "bio". To make it easier for these types of
+filesystems to support fs-verity, fs/verity/ also provides a function
+fsverity_verify_bio() which verifies all pages in a bio.
+
+ext4 and f2fs also support encryption. If a verity file is also
+encrypted, the pages must be decrypted before being verified. To
+support this, these filesystems allocate a "post-read context" for
+each bio and store it in ``->bi_private``::
+
+ struct bio_post_read_ctx {
+ struct bio *bio;
+ struct work_struct work;
+ unsigned int cur_step;
+ unsigned int enabled_steps;
+ };
+
+``enabled_steps`` is a bitmask that specifies whether decryption,
+verity, or both is enabled. After the bio completes, for each needed
+postprocessing step the filesystem enqueues the bio_post_read_ctx on a
+workqueue, and then the workqueue work does the decryption or
+verification. Finally, pages where no decryption or verity error
+occurred are marked Uptodate, and the pages are unlocked.
+
+Files on ext4 and f2fs may contain holes. Normally, ``->readpages()``
+simply zeroes holes and sets the corresponding pages Uptodate; no bios
+are issued. To prevent this case from bypassing fs-verity, these
+filesystems use fsverity_verify_page() to verify hole pages.
+
+ext4 and f2fs disable direct I/O on verity files, since otherwise
+direct I/O would bypass fs-verity. (They also do the same for
+encrypted files.)
+
+Userspace utility
+=================
+
+This document focuses on the kernel, but a userspace utility for
+fs-verity can be found at:
+
+ https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/fsverity-utils.git
+
+See the README.md file in the fsverity-utils source tree for details,
+including examples of setting up fs-verity protected files.
+
+Tests
+=====
+
+To test fs-verity, use xfstests. For example, using `kvm-xfstests
+<https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md>`_::
+
+ kvm-xfstests -c ext4,f2fs -g verity
+
+FAQ
+===
+
+This section answers frequently asked questions about fs-verity that
+weren't already directly answered in other parts of this document.
+
+:Q: Why isn't fs-verity part of IMA?
+:A: fs-verity and IMA (Integrity Measurement Architecture) have
+ different focuses. fs-verity is a filesystem-level mechanism for
+ hashing individual files using a Merkle tree. In contrast, IMA
+ specifies a system-wide policy that specifies which files are
+ hashed and what to do with those hashes, such as log them,
+ authenticate them, or add them to a measurement list.
+
+ IMA is planned to support the fs-verity hashing mechanism as an
+ alternative to doing full file hashes, for people who want the
+ performance and security benefits of the Merkle tree based hash.
+ But it doesn't make sense to force all uses of fs-verity to be
+ through IMA. As a standalone filesystem feature, fs-verity
+ already meets many users' needs, and it's testable like other
+ filesystem features e.g. with xfstests.
+
+:Q: Isn't fs-verity useless because the attacker can just modify the
+ hashes in the Merkle tree, which is stored on-disk?
+:A: To verify the authenticity of an fs-verity file you must verify
+ the authenticity of the "file measurement", which is basically the
+ root hash of the Merkle tree. See `Use cases`_.
+
+:Q: Isn't fs-verity useless because the attacker can just replace a
+ verity file with a non-verity one?
+:A: See `Use cases`_. In the initial use case, it's really trusted
+ userspace code that authenticates the files; fs-verity is just a
+ tool to do this job efficiently and securely. The trusted
+ userspace code will consider non-verity files to be inauthentic.
+
+:Q: Why does the Merkle tree need to be stored on-disk? Couldn't you
+ store just the root hash?
+:A: If the Merkle tree wasn't stored on-disk, then you'd have to
+ compute the entire tree when the file is first accessed, even if
+ just one byte is being read. This is a fundamental consequence of
+ how Merkle tree hashing works. To verify a leaf node, you need to
+ verify the whole path to the root hash, including the root node
+ (the thing which the root hash is a hash of). But if the root
+ node isn't stored on-disk, you have to compute it by hashing its
+ children, and so on until you've actually hashed the entire file.
+
+ That defeats most of the point of doing a Merkle tree-based hash,
+ since if you have to hash the whole file ahead of time anyway,
+ then you could simply do sha256(file) instead. That would be much
+ simpler, and a bit faster too.
+
+ It's true that an in-memory Merkle tree could still provide the
+ advantage of verification on every read rather than just on the
+ first read. However, it would be inefficient because every time a
+ hash page gets evicted (you can't pin the entire Merkle tree into
+ memory, since it may be very large), in order to restore it you
+ again need to hash everything below it in the tree. This again
+ defeats most of the point of doing a Merkle tree-based hash, since
+ a single block read could trigger re-hashing gigabytes of data.
+
+:Q: But couldn't you store just the leaf nodes and compute the rest?
+:A: See previous answer; this really just moves up one level, since
+ one could alternatively interpret the data blocks as being the
+ leaf nodes of the Merkle tree. It's true that the tree can be
+ computed much faster if the leaf level is stored rather than just
+ the data, but that's only because each level is less than 1% the
+ size of the level below (assuming the recommended settings of
+ SHA-256 and 4K blocks). For the exact same reason, by storing
+ "just the leaf nodes" you'd already be storing over 99% of the
+ tree, so you might as well simply store the whole tree.
+
+:Q: Can the Merkle tree be built ahead of time, e.g. distributed as
+ part of a package that is installed to many computers?
+:A: This isn't currently supported. It was part of the original
+ design, but was removed to simplify the kernel UAPI and because it
+ wasn't a critical use case. Files are usually installed once and
+ used many times, and cryptographic hashing is somewhat fast on
+ most modern processors.
+
+:Q: Why doesn't fs-verity support writes?
+:A: Write support would be very difficult and would require a
+ completely different design, so it's well outside the scope of
+ fs-verity. Write support would require:
+
+ - A way to maintain consistency between the data and hashes,
+ including all levels of hashes, since corruption after a crash
+ (especially of potentially the entire file!) is unacceptable.
+ The main options for solving this are data journalling,
+ copy-on-write, and log-structured volume. But it's very hard to
+ retrofit existing filesystems with new consistency mechanisms.
+ Data journalling is available on ext4, but is very slow.
+
+ - Rebuilding the the Merkle tree after every write, which would be
+ extremely inefficient. Alternatively, a different authenticated
+ dictionary structure such as an "authenticated skiplist" could
+ be used. However, this would be far more complex.
+
+ Compare it to dm-verity vs. dm-integrity. dm-verity is very
+ simple: the kernel just verifies read-only data against a
+ read-only Merkle tree. In contrast, dm-integrity supports writes
+ but is slow, is much more complex, and doesn't actually support
+ full-device authentication since it authenticates each sector
+ independently, i.e. there is no "root hash". It doesn't really
+ make sense for the same device-mapper target to support these two
+ very different cases; the same applies to fs-verity.
+
+:Q: Since verity files are immutable, why isn't the immutable bit set?
+:A: The existing "immutable" bit (FS_IMMUTABLE_FL) already has a
+ specific set of semantics which not only make the file contents
+ read-only, but also prevent the file from being deleted, renamed,
+ linked to, or having its owner or mode changed. These extra
+ properties are unwanted for fs-verity, so reusing the immutable
+ bit isn't appropriate.
+
+:Q: Why does the API use ioctls instead of setxattr() and getxattr()?
+:A: Abusing the xattr interface for basically arbitrary syscalls is
+ heavily frowned upon by most of the Linux filesystem developers.
+ An xattr should really just be an xattr on-disk, not an API to
+ e.g. magically trigger construction of a Merkle tree.
+
+:Q: Does fs-verity support remote filesystems?
+:A: Only ext4 and f2fs support is implemented currently, but in
+ principle any filesystem that can store per-file verity metadata
+ can support fs-verity, regardless of whether it's local or remote.
+ Some filesystems may have fewer options of where to store the
+ verity metadata; one possibility is to store it past the end of
+ the file and "hide" it from userspace by manipulating i_size. The
+ data verification functions provided by ``fs/verity/`` also assume
+ that the filesystem uses the Linux pagecache, but both local and
+ remote filesystems normally do so.
+
+:Q: Why is anything filesystem-specific at all? Shouldn't fs-verity
+ be implemented entirely at the VFS level?
+:A: There are many reasons why this is not possible or would be very
+ difficult, including the following:
+
+ - To prevent bypassing verification, pages must not be marked
+ Uptodate until they've been verified. Currently, each
+ filesystem is responsible for marking pages Uptodate via
+ ``->readpages()``. Therefore, currently it's not possible for
+ the VFS to do the verification on its own. Changing this would
+ require significant changes to the VFS and all filesystems.
+
+ - It would require defining a filesystem-independent way to store
+ the verity metadata. Extended attributes don't work for this
+ because (a) the Merkle tree may be gigabytes, but many
+ filesystems assume that all xattrs fit into a single 4K
+ filesystem block, and (b) ext4 and f2fs encryption doesn't
+ encrypt xattrs, yet the Merkle tree *must* be encrypted when the
+ file contents are, because it stores hashes of the plaintext
+ file contents.
+
+ So the verity metadata would have to be stored in an actual
+ file. Using a separate file would be very ugly, since the
+ metadata is fundamentally part of the file to be protected, and
+ it could cause problems where users could delete the real file
+ but not the metadata file or vice versa. On the other hand,
+ having it be in the same file would break applications unless
+ filesystems' notion of i_size were divorced from the VFS's,
+ which would be complex and require changes to all filesystems.
+
+ - It's desirable that FS_IOC_ENABLE_VERITY uses the filesystem's
+ transaction mechanism so that either the file ends up with
+ verity enabled, or no changes were made. Allowing intermediate
+ states to occur after a crash may cause problems.
diff --git a/Documentation/filesystems/index.rst b/Documentation/filesystems/index.rst
index 46d1b1b..2c3a9f7 100644
--- a/Documentation/filesystems/index.rst
+++ b/Documentation/filesystems/index.rst
@@ -1,361 +1,49 @@
-=====================
-Linux Filesystems API
-=====================
+===============================
+Filesystems in the Linux kernel
+===============================
-The Linux VFS
-=============
+This under-development manual will, some glorious day, provide
+comprehensive information on how the Linux virtual filesystem (VFS) layer
+works, along with the filesystems that sit below it. For now, what we have
+can be found below.
-The Filesystem types
---------------------
-
-.. kernel-doc:: include/linux/fs.h
- :internal:
-
-The Directory Cache
--------------------
-
-.. kernel-doc:: fs/dcache.c
- :export:
-
-.. kernel-doc:: include/linux/dcache.h
- :internal:
-
-Inode Handling
---------------
-
-.. kernel-doc:: fs/inode.c
- :export:
-
-.. kernel-doc:: fs/bad_inode.c
- :export:
-
-Registration and Superblocks
-----------------------------
-
-.. kernel-doc:: fs/super.c
- :export:
-
-File Locks
-----------
-
-.. kernel-doc:: fs/locks.c
- :export:
-
-.. kernel-doc:: fs/locks.c
- :internal:
-
-Other Functions
----------------
-
-.. kernel-doc:: fs/mpage.c
- :export:
-
-.. kernel-doc:: fs/namei.c
- :export:
-
-.. kernel-doc:: fs/buffer.c
- :export:
-
-.. kernel-doc:: block/bio.c
- :export:
-
-.. kernel-doc:: fs/seq_file.c
- :export:
-
-.. kernel-doc:: fs/filesystems.c
- :export:
-
-.. kernel-doc:: fs/fs-writeback.c
- :export:
-
-.. kernel-doc:: fs/block_dev.c
- :export:
-
-.. kernel-doc:: fs/anon_inodes.c
- :export:
-
-.. kernel-doc:: fs/attr.c
- :export:
-
-.. kernel-doc:: fs/d_path.c
- :export:
-
-.. kernel-doc:: fs/dax.c
- :export:
-
-.. kernel-doc:: fs/direct-io.c
- :export:
-
-.. kernel-doc:: fs/file_table.c
- :export:
-
-.. kernel-doc:: fs/libfs.c
- :export:
-
-.. kernel-doc:: fs/posix_acl.c
- :export:
-
-.. kernel-doc:: fs/stat.c
- :export:
-
-.. kernel-doc:: fs/sync.c
- :export:
-
-.. kernel-doc:: fs/xattr.c
- :export:
-
-The proc filesystem
-===================
-
-sysctl interface
-----------------
-
-.. kernel-doc:: kernel/sysctl.c
- :export:
-
-proc filesystem interface
--------------------------
-
-.. kernel-doc:: fs/proc/base.c
- :internal:
-
-Events based on file descriptors
-================================
-
-.. kernel-doc:: fs/eventfd.c
- :export:
-
-The Filesystem for Exporting Kernel Objects
-===========================================
-
-.. kernel-doc:: fs/sysfs/file.c
- :export:
-
-.. kernel-doc:: fs/sysfs/symlink.c
- :export:
-
-The debugfs filesystem
+Core VFS documentation
======================
-debugfs interface
------------------
-
-.. kernel-doc:: fs/debugfs/inode.c
- :export:
-
-.. kernel-doc:: fs/debugfs/file.c
- :export:
-
-The Linux Journalling API
-=========================
-
-Overview
---------
-
-Details
-~~~~~~~
-
-The journalling layer is easy to use. You need to first of all create a
-journal_t data structure. There are two calls to do this dependent on
-how you decide to allocate the physical media on which the journal
-resides. The :c:func:`jbd2_journal_init_inode` call is for journals stored in
-filesystem inodes, or the :c:func:`jbd2_journal_init_dev` call can be used
-for journal stored on a raw device (in a continuous range of blocks). A
-journal_t is a typedef for a struct pointer, so when you are finally
-finished make sure you call :c:func:`jbd2_journal_destroy` on it to free up
-any used kernel memory.
-
-Once you have got your journal_t object you need to 'mount' or load the
-journal file. The journalling layer expects the space for the journal
-was already allocated and initialized properly by the userspace tools.
-When loading the journal you must call :c:func:`jbd2_journal_load` to process
-journal contents. If the client file system detects the journal contents
-does not need to be processed (or even need not have valid contents), it
-may call :c:func:`jbd2_journal_wipe` to clear the journal contents before
-calling :c:func:`jbd2_journal_load`.
-
-Note that jbd2_journal_wipe(..,0) calls
-:c:func:`jbd2_journal_skip_recovery` for you if it detects any outstanding
-transactions in the journal and similarly :c:func:`jbd2_journal_load` will
-call :c:func:`jbd2_journal_recover` if necessary. I would advise reading
-:c:func:`ext4_load_journal` in fs/ext4/super.c for examples on this stage.
-
-Now you can go ahead and start modifying the underlying filesystem.
-Almost.
-
-You still need to actually journal your filesystem changes, this is done
-by wrapping them into transactions. Additionally you also need to wrap
-the modification of each of the buffers with calls to the journal layer,
-so it knows what the modifications you are actually making are. To do
-this use :c:func:`jbd2_journal_start` which returns a transaction handle.
-
-:c:func:`jbd2_journal_start` and its counterpart :c:func:`jbd2_journal_stop`,
-which indicates the end of a transaction are nestable calls, so you can
-reenter a transaction if necessary, but remember you must call
-:c:func:`jbd2_journal_stop` the same number of times as
-:c:func:`jbd2_journal_start` before the transaction is completed (or more
-accurately leaves the update phase). Ext4/VFS makes use of this feature to
-simplify handling of inode dirtying, quota support, etc.
-
-Inside each transaction you need to wrap the modifications to the
-individual buffers (blocks). Before you start to modify a buffer you
-need to call :c:func:`jbd2_journal_get_create_access()` /
-:c:func:`jbd2_journal_get_write_access()` /
-:c:func:`jbd2_journal_get_undo_access()` as appropriate, this allows the
-journalling layer to copy the unmodified
-data if it needs to. After all the buffer may be part of a previously
-uncommitted transaction. At this point you are at last ready to modify a
-buffer, and once you are have done so you need to call
-:c:func:`jbd2_journal_dirty_metadata`. Or if you've asked for access to a
-buffer you now know is now longer required to be pushed back on the
-device you can call :c:func:`jbd2_journal_forget` in much the same way as you
-might have used :c:func:`bforget` in the past.
-
-A :c:func:`jbd2_journal_flush` may be called at any time to commit and
-checkpoint all your transactions.
-
-Then at umount time , in your :c:func:`put_super` you can then call
-:c:func:`jbd2_journal_destroy` to clean up your in-core journal object.
-
-Unfortunately there a couple of ways the journal layer can cause a
-deadlock. The first thing to note is that each task can only have a
-single outstanding transaction at any one time, remember nothing commits
-until the outermost :c:func:`jbd2_journal_stop`. This means you must complete
-the transaction at the end of each file/inode/address etc. operation you
-perform, so that the journalling system isn't re-entered on another
-journal. Since transactions can't be nested/batched across differing
-journals, and another filesystem other than yours (say ext4) may be
-modified in a later syscall.
-
-The second case to bear in mind is that :c:func:`jbd2_journal_start` can block
-if there isn't enough space in the journal for your transaction (based
-on the passed nblocks param) - when it blocks it merely(!) needs to wait
-for transactions to complete and be committed from other tasks, so
-essentially we are waiting for :c:func:`jbd2_journal_stop`. So to avoid
-deadlocks you must treat :c:func:`jbd2_journal_start` /
-:c:func:`jbd2_journal_stop` as if they were semaphores and include them in
-your semaphore ordering rules to prevent
-deadlocks. Note that :c:func:`jbd2_journal_extend` has similar blocking
-behaviour to :c:func:`jbd2_journal_start` so you can deadlock here just as
-easily as on :c:func:`jbd2_journal_start`.
-
-Try to reserve the right number of blocks the first time. ;-). This will
-be the maximum number of blocks you are going to touch in this
-transaction. I advise having a look at at least ext4_jbd.h to see the
-basis on which ext4 uses to make these decisions.
-
-Another wriggle to watch out for is your on-disk block allocation
-strategy. Why? Because, if you do a delete, you need to ensure you
-haven't reused any of the freed blocks until the transaction freeing
-these blocks commits. If you reused these blocks and crash happens,
-there is no way to restore the contents of the reallocated blocks at the
-end of the last fully committed transaction. One simple way of doing
-this is to mark blocks as free in internal in-memory block allocation
-structures only after the transaction freeing them commits. Ext4 uses
-journal commit callback for this purpose.
-
-With journal commit callbacks you can ask the journalling layer to call
-a callback function when the transaction is finally committed to disk,
-so that you can do some of your own management. You ask the journalling
-layer for calling the callback by simply setting
-``journal->j_commit_callback`` function pointer and that function is
-called after each transaction commit. You can also use
-``transaction->t_private_list`` for attaching entries to a transaction
-that need processing when the transaction commits.
-
-JBD2 also provides a way to block all transaction updates via
-:c:func:`jbd2_journal_lock_updates()` /
-:c:func:`jbd2_journal_unlock_updates()`. Ext4 uses this when it wants a
-window with a clean and stable fs for a moment. E.g.
-
-::
-
-
- jbd2_journal_lock_updates() //stop new stuff happening..
- jbd2_journal_flush() // checkpoint everything.
- ..do stuff on stable fs
- jbd2_journal_unlock_updates() // carry on with filesystem use.
-
-The opportunities for abuse and DOS attacks with this should be obvious,
-if you allow unprivileged userspace to trigger codepaths containing
-these calls.
-
-Summary
-~~~~~~~
-
-Using the journal is a matter of wrapping the different context changes,
-being each mount, each modification (transaction) and each changed
-buffer to tell the journalling layer about them.
-
-Data Types
-----------
-
-The journalling layer uses typedefs to 'hide' the concrete definitions
-of the structures used. As a client of the JBD2 layer you can just rely
-on the using the pointer as a magic cookie of some sort. Obviously the
-hiding is not enforced as this is 'C'.
-
-Structures
-~~~~~~~~~~
-
-.. kernel-doc:: include/linux/jbd2.h
- :internal:
-
-Functions
----------
-
-The functions here are split into two groups those that affect a journal
-as a whole, and those which are used to manage transactions
-
-Journal Level
-~~~~~~~~~~~~~
-
-.. kernel-doc:: fs/jbd2/journal.c
- :export:
-
-.. kernel-doc:: fs/jbd2/recovery.c
- :internal:
-
-Transasction Level
-~~~~~~~~~~~~~~~~~~
-
-.. kernel-doc:: fs/jbd2/transaction.c
-
-See also
---------
-
-`Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen
-Tweedie <http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz>`__
-
-`Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen
-Tweedie <http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html>`__
-
-splice API
-==========
-
-splice is a method for moving blocks of data around inside the kernel,
-without continually transferring them between the kernel and user space.
-
-.. kernel-doc:: fs/splice.c
-
-pipes API
-=========
-
-Pipe interfaces are all for in-kernel (builtin image) use. They are not
-exported for use by modules.
-
-.. kernel-doc:: include/linux/pipe_fs_i.h
- :internal:
-
-.. kernel-doc:: fs/pipe.c
-
-Encryption API
-==============
-
-A library which filesystems can hook into to support transparent
-encryption of files and directories.
+See these manuals for documentation about the VFS layer itself and how its
+algorithms work.
.. toctree::
- :maxdepth: 2
+ :maxdepth: 2
- fscrypt
+ vfs
+ path-lookup
+ api-summary
+ splice
+ locking
+ directory-locking
+
+ porting
+
+Filesystem support layers
+=========================
+
+Documentation for the support code within the filesystem layer for use in
+filesystem implementations.
+
+.. toctree::
+ :maxdepth: 2
+
+ journalling
+ fscrypt
+ fsverity
+
+Filesystems
+===========
+
+Documentation for filesystem implementations.
+
+.. toctree::
+ :maxdepth: 2
+
+ virtiofs
diff --git a/Documentation/filesystems/jfs.txt b/Documentation/filesystems/jfs.txt
deleted file mode 100644
index 41fd757..0000000
--- a/Documentation/filesystems/jfs.txt
+++ /dev/null
@@ -1,52 +0,0 @@
-IBM's Journaled File System (JFS) for Linux
-
-JFS Homepage: http://jfs.sourceforge.net/
-
-The following mount options are supported:
-(*) == default
-
-iocharset=name Character set to use for converting from Unicode to
- ASCII. The default is to do no conversion. Use
- iocharset=utf8 for UTF-8 translations. This requires
- CONFIG_NLS_UTF8 to be set in the kernel .config file.
- iocharset=none specifies the default behavior explicitly.
-
-resize=value Resize the volume to <value> blocks. JFS only supports
- growing a volume, not shrinking it. This option is only
- valid during a remount, when the volume is mounted
- read-write. The resize keyword with no value will grow
- the volume to the full size of the partition.
-
-nointegrity Do not write to the journal. The primary use of this option
- is to allow for higher performance when restoring a volume
- from backup media. The integrity of the volume is not
- guaranteed if the system abnormally abends.
-
-integrity(*) Commit metadata changes to the journal. Use this option to
- remount a volume where the nointegrity option was
- previously specified in order to restore normal behavior.
-
-errors=continue Keep going on a filesystem error.
-errors=remount-ro(*) Remount the filesystem read-only on an error.
-errors=panic Panic and halt the machine if an error occurs.
-
-uid=value Override on-disk uid with specified value
-gid=value Override on-disk gid with specified value
-umask=value Override on-disk umask with specified octal value. For
- directories, the execute bit will be set if the corresponding
- read bit is set.
-
-discard=minlen This enables/disables the use of discard/TRIM commands.
-discard The discard/TRIM commands are sent to the underlying
-nodiscard(*) block device when blocks are freed. This is useful for SSD
- devices and sparse/thinly-provisioned LUNs. The FITRIM ioctl
- command is also available together with the nodiscard option.
- The value of minlen specifies the minimum blockcount, when
- a TRIM command to the block device is considered useful.
- When no value is given to the discard option, it defaults to
- 64 blocks, which means 256KiB in JFS.
- The minlen value of discard overrides the minlen value given
- on an FITRIM ioctl().
-
-The JFS mailing list can be subscribed to by using the link labeled
-"Mail list Subscribe" at our web page http://jfs.sourceforge.net/
diff --git a/Documentation/filesystems/journalling.rst b/Documentation/filesystems/journalling.rst
new file mode 100644
index 0000000..58ce6b3
--- /dev/null
+++ b/Documentation/filesystems/journalling.rst
@@ -0,0 +1,184 @@
+The Linux Journalling API
+=========================
+
+Overview
+--------
+
+Details
+~~~~~~~
+
+The journalling layer is easy to use. You need to first of all create a
+journal_t data structure. There are two calls to do this dependent on
+how you decide to allocate the physical media on which the journal
+resides. The :c:func:`jbd2_journal_init_inode` call is for journals stored in
+filesystem inodes, or the :c:func:`jbd2_journal_init_dev` call can be used
+for journal stored on a raw device (in a continuous range of blocks). A
+journal_t is a typedef for a struct pointer, so when you are finally
+finished make sure you call :c:func:`jbd2_journal_destroy` on it to free up
+any used kernel memory.
+
+Once you have got your journal_t object you need to 'mount' or load the
+journal file. The journalling layer expects the space for the journal
+was already allocated and initialized properly by the userspace tools.
+When loading the journal you must call :c:func:`jbd2_journal_load` to process
+journal contents. If the client file system detects the journal contents
+does not need to be processed (or even need not have valid contents), it
+may call :c:func:`jbd2_journal_wipe` to clear the journal contents before
+calling :c:func:`jbd2_journal_load`.
+
+Note that jbd2_journal_wipe(..,0) calls
+:c:func:`jbd2_journal_skip_recovery` for you if it detects any outstanding
+transactions in the journal and similarly :c:func:`jbd2_journal_load` will
+call :c:func:`jbd2_journal_recover` if necessary. I would advise reading
+:c:func:`ext4_load_journal` in fs/ext4/super.c for examples on this stage.
+
+Now you can go ahead and start modifying the underlying filesystem.
+Almost.
+
+You still need to actually journal your filesystem changes, this is done
+by wrapping them into transactions. Additionally you also need to wrap
+the modification of each of the buffers with calls to the journal layer,
+so it knows what the modifications you are actually making are. To do
+this use :c:func:`jbd2_journal_start` which returns a transaction handle.
+
+:c:func:`jbd2_journal_start` and its counterpart :c:func:`jbd2_journal_stop`,
+which indicates the end of a transaction are nestable calls, so you can
+reenter a transaction if necessary, but remember you must call
+:c:func:`jbd2_journal_stop` the same number of times as
+:c:func:`jbd2_journal_start` before the transaction is completed (or more
+accurately leaves the update phase). Ext4/VFS makes use of this feature to
+simplify handling of inode dirtying, quota support, etc.
+
+Inside each transaction you need to wrap the modifications to the
+individual buffers (blocks). Before you start to modify a buffer you
+need to call :c:func:`jbd2_journal_get_create_access()` /
+:c:func:`jbd2_journal_get_write_access()` /
+:c:func:`jbd2_journal_get_undo_access()` as appropriate, this allows the
+journalling layer to copy the unmodified
+data if it needs to. After all the buffer may be part of a previously
+uncommitted transaction. At this point you are at last ready to modify a
+buffer, and once you are have done so you need to call
+:c:func:`jbd2_journal_dirty_metadata`. Or if you've asked for access to a
+buffer you now know is now longer required to be pushed back on the
+device you can call :c:func:`jbd2_journal_forget` in much the same way as you
+might have used :c:func:`bforget` in the past.
+
+A :c:func:`jbd2_journal_flush` may be called at any time to commit and
+checkpoint all your transactions.
+
+Then at umount time , in your :c:func:`put_super` you can then call
+:c:func:`jbd2_journal_destroy` to clean up your in-core journal object.
+
+Unfortunately there a couple of ways the journal layer can cause a
+deadlock. The first thing to note is that each task can only have a
+single outstanding transaction at any one time, remember nothing commits
+until the outermost :c:func:`jbd2_journal_stop`. This means you must complete
+the transaction at the end of each file/inode/address etc. operation you
+perform, so that the journalling system isn't re-entered on another
+journal. Since transactions can't be nested/batched across differing
+journals, and another filesystem other than yours (say ext4) may be
+modified in a later syscall.
+
+The second case to bear in mind is that :c:func:`jbd2_journal_start` can block
+if there isn't enough space in the journal for your transaction (based
+on the passed nblocks param) - when it blocks it merely(!) needs to wait
+for transactions to complete and be committed from other tasks, so
+essentially we are waiting for :c:func:`jbd2_journal_stop`. So to avoid
+deadlocks you must treat :c:func:`jbd2_journal_start` /
+:c:func:`jbd2_journal_stop` as if they were semaphores and include them in
+your semaphore ordering rules to prevent
+deadlocks. Note that :c:func:`jbd2_journal_extend` has similar blocking
+behaviour to :c:func:`jbd2_journal_start` so you can deadlock here just as
+easily as on :c:func:`jbd2_journal_start`.
+
+Try to reserve the right number of blocks the first time. ;-). This will
+be the maximum number of blocks you are going to touch in this
+transaction. I advise having a look at at least ext4_jbd.h to see the
+basis on which ext4 uses to make these decisions.
+
+Another wriggle to watch out for is your on-disk block allocation
+strategy. Why? Because, if you do a delete, you need to ensure you
+haven't reused any of the freed blocks until the transaction freeing
+these blocks commits. If you reused these blocks and crash happens,
+there is no way to restore the contents of the reallocated blocks at the
+end of the last fully committed transaction. One simple way of doing
+this is to mark blocks as free in internal in-memory block allocation
+structures only after the transaction freeing them commits. Ext4 uses
+journal commit callback for this purpose.
+
+With journal commit callbacks you can ask the journalling layer to call
+a callback function when the transaction is finally committed to disk,
+so that you can do some of your own management. You ask the journalling
+layer for calling the callback by simply setting
+``journal->j_commit_callback`` function pointer and that function is
+called after each transaction commit. You can also use
+``transaction->t_private_list`` for attaching entries to a transaction
+that need processing when the transaction commits.
+
+JBD2 also provides a way to block all transaction updates via
+:c:func:`jbd2_journal_lock_updates()` /
+:c:func:`jbd2_journal_unlock_updates()`. Ext4 uses this when it wants a
+window with a clean and stable fs for a moment. E.g.
+
+::
+
+
+ jbd2_journal_lock_updates() //stop new stuff happening..
+ jbd2_journal_flush() // checkpoint everything.
+ ..do stuff on stable fs
+ jbd2_journal_unlock_updates() // carry on with filesystem use.
+
+The opportunities for abuse and DOS attacks with this should be obvious,
+if you allow unprivileged userspace to trigger codepaths containing
+these calls.
+
+Summary
+~~~~~~~
+
+Using the journal is a matter of wrapping the different context changes,
+being each mount, each modification (transaction) and each changed
+buffer to tell the journalling layer about them.
+
+Data Types
+----------
+
+The journalling layer uses typedefs to 'hide' the concrete definitions
+of the structures used. As a client of the JBD2 layer you can just rely
+on the using the pointer as a magic cookie of some sort. Obviously the
+hiding is not enforced as this is 'C'.
+
+Structures
+~~~~~~~~~~
+
+.. kernel-doc:: include/linux/jbd2.h
+ :internal:
+
+Functions
+---------
+
+The functions here are split into two groups those that affect a journal
+as a whole, and those which are used to manage transactions
+
+Journal Level
+~~~~~~~~~~~~~
+
+.. kernel-doc:: fs/jbd2/journal.c
+ :export:
+
+.. kernel-doc:: fs/jbd2/recovery.c
+ :internal:
+
+Transasction Level
+~~~~~~~~~~~~~~~~~~
+
+.. kernel-doc:: fs/jbd2/transaction.c
+
+See also
+--------
+
+`Journaling the Linux ext2fs Filesystem, LinuxExpo 98, Stephen
+Tweedie <http://kernel.org/pub/linux/kernel/people/sct/ext3/journal-design.ps.gz>`__
+
+`Ext3 Journalling FileSystem, OLS 2000, Dr. Stephen
+Tweedie <http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html>`__
+
diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/locking.rst
similarity index 78%
rename from Documentation/filesystems/Locking
rename to Documentation/filesystems/locking.rst
index efea228..fc3a070 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/locking.rst
@@ -1,14 +1,22 @@
- The text below describes the locking rules for VFS-related methods.
+=======
+Locking
+=======
+
+The text below describes the locking rules for VFS-related methods.
It is (believed to be) up-to-date. *Please*, if you change anything in
prototypes or locking protocols - update this file. And update the relevant
instances in the tree, don't leave that to maintainers of filesystems/devices/
etc. At the very least, put the list of dubious cases in the end of this file.
Don't turn it into log - maintainers of out-of-the-tree code are supposed to
be able to use diff(1).
- Thing currently missing here: socket operations. Alexey?
---------------------------- dentry_operations --------------------------
-prototypes:
+Thing currently missing here: socket operations. Alexey?
+
+dentry_operations
+=================
+
+prototypes::
+
int (*d_revalidate)(struct dentry *, unsigned int);
int (*d_weak_revalidate)(struct dentry *, unsigned int);
int (*d_hash)(const struct dentry *, struct qstr *);
@@ -24,23 +32,30 @@
struct dentry *(*d_real)(struct dentry *, const struct inode *);
locking rules:
- rename_lock ->d_lock may block rcu-walk
-d_revalidate: no no yes (ref-walk) maybe
-d_weak_revalidate:no no yes no
-d_hash no no no maybe
-d_compare: yes no no maybe
-d_delete: no yes no no
-d_init: no no yes no
-d_release: no no yes no
-d_prune: no yes no no
-d_iput: no no yes no
-d_dname: no no no no
-d_automount: no no yes no
-d_manage: no no yes (ref-walk) maybe
-d_real no no yes no
---------------------------- inode_operations ---------------------------
-prototypes:
+================== =========== ======== ============== ========
+ops rename_lock ->d_lock may block rcu-walk
+================== =========== ======== ============== ========
+d_revalidate: no no yes (ref-walk) maybe
+d_weak_revalidate: no no yes no
+d_hash no no no maybe
+d_compare: yes no no maybe
+d_delete: no yes no no
+d_init: no no yes no
+d_release: no no yes no
+d_prune: no yes no no
+d_iput: no no yes no
+d_dname: no no no no
+d_automount: no no yes no
+d_manage: no no yes (ref-walk) maybe
+d_real no no yes no
+================== =========== ======== ============== ========
+
+inode_operations
+================
+
+prototypes::
+
int (*create) (struct inode *,struct dentry *,umode_t, bool);
struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int);
int (*link) (struct dentry *,struct inode *,struct dentry *);
@@ -52,7 +67,7 @@
int (*rename) (struct inode *, struct dentry *,
struct inode *, struct dentry *, unsigned int);
int (*readlink) (struct dentry *, char __user *,int);
- const char *(*get_link) (struct dentry *, struct inode *, void **);
+ const char *(*get_link) (struct dentry *, struct inode *, struct delayed_call *);
void (*truncate) (struct inode *);
int (*permission) (struct inode *, int, unsigned int);
int (*get_acl)(struct inode *, int);
@@ -68,7 +83,10 @@
locking rules:
all may block
- i_rwsem(inode)
+
+============ =============================================
+ops i_rwsem(inode)
+============ =============================================
lookup: shared
create: exclusive
link: exclusive (both)
@@ -89,17 +107,21 @@
update_time: no
atomic_open: exclusive
tmpfile: no
+============ =============================================
Additionally, ->rmdir(), ->unlink() and ->rename() have ->i_rwsem
exclusive on victim.
cross-directory ->rename() has (per-superblock) ->s_vfs_rename_sem.
-See Documentation/filesystems/directory-locking for more detailed discussion
+See Documentation/filesystems/directory-locking.rst for more detailed discussion
of the locking scheme for directory operations.
------------------------ xattr_handler operations -----------------------
-prototypes:
+xattr_handler operations
+========================
+
+prototypes::
+
bool (*list)(struct dentry *dentry);
int (*get)(const struct xattr_handler *handler, struct dentry *dentry,
struct inode *inode, const char *name, void *buffer,
@@ -110,14 +132,22 @@
locking rules:
all may block
- i_rwsem(inode)
+
+===== ==============
+ops i_rwsem(inode)
+===== ==============
list: no
get: no
set: exclusive
+===== ==============
---------------------------- super_operations ---------------------------
-prototypes:
+super_operations
+================
+
+prototypes::
+
struct inode *(*alloc_inode)(struct super_block *sb);
+ void (*free_inode)(struct inode *);
void (*destroy_inode)(struct inode *);
void (*dirty_inode) (struct inode *, int flags);
int (*write_inode) (struct inode *, struct writeback_control *wbc);
@@ -137,8 +167,12 @@
locking rules:
All may block [not true, see below]
- s_umount
+
+====================== ============ ========================
+ops s_umount note
+====================== ============ ========================
alloc_inode:
+free_inode: called from RCU callback
destroy_inode:
dirty_inode:
write_inode:
@@ -155,6 +189,7 @@
quota_read: no (see below)
quota_write: no (see below)
bdev_try_to_free_page: no (see below)
+====================== ============ ========================
->statfs() has s_umount (shared) when called by ustat(2) (native or
compat), but that's an accident of bad API; s_umount is used to pin
@@ -162,31 +197,44 @@
identify the superblock. Everything else (statfs(), fstatfs(), etc.)
doesn't hold it when calling ->statfs() - superblock is pinned down
by resolving the pathname passed to syscall.
+
->quota_read() and ->quota_write() functions are both guaranteed to
be the only ones operating on the quota file by the quota code (via
dqio_sem) (unless an admin really wants to screw up something and
writes to quota files with quotas on). For other details about locking
see also dquot_operations section.
+
->bdev_try_to_free_page is called from the ->releasepage handler of
the block device inode. See there for more details.
---------------------------- file_system_type ---------------------------
-prototypes:
+file_system_type
+================
+
+prototypes::
+
struct dentry *(*mount) (struct file_system_type *, int,
const char *, void *);
void (*kill_sb) (struct super_block *);
+
locking rules:
- may block
+
+======= =========
+ops may block
+======= =========
mount yes
kill_sb yes
+======= =========
->mount() returns ERR_PTR or the root dentry; its superblock should be locked
on return.
+
->kill_sb() takes a write-locked superblock, does all shutdown work on it,
unlocks and drops the reference.
---------------------------- address_space_operations --------------------------
-prototypes:
+address_space_operations
+========================
+prototypes::
+
int (*writepage)(struct page *page, struct writeback_control *wbc);
int (*readpage)(struct file *, struct page *);
int (*writepages)(struct address_space *, struct writeback_control *);
@@ -216,14 +264,16 @@
locking rules:
All except set_page_dirty and freepage may block
- PageLocked(page) i_rwsem
+====================== ======================== =========
+ops PageLocked(page) i_rwsem
+====================== ======================== =========
writepage: yes, unlocks (see below)
readpage: yes, unlocks
writepages:
set_page_dirty no
readpages:
-write_begin: locks the page exclusive
-write_end: yes, unlocks exclusive
+write_begin: locks the page exclusive
+write_end: yes, unlocks exclusive
bmap:
invalidatepage: yes
releasepage: yes
@@ -237,17 +287,18 @@
error_remove_page: yes
swap_activate: no
swap_deactivate: no
+====================== ======================== =========
- ->write_begin(), ->write_end() and ->readpage() may be called from
+->write_begin(), ->write_end() and ->readpage() may be called from
the request handler (/dev/loop).
- ->readpage() unlocks the page, either synchronously or via I/O
+->readpage() unlocks the page, either synchronously or via I/O
completion.
- ->readpages() populates the pagecache with the passed pages and starts
+->readpages() populates the pagecache with the passed pages and starts
I/O against them. They come unlocked upon I/O completion.
- ->writepage() is used for two purposes: for "memory cleansing" and for
+->writepage() is used for two purposes: for "memory cleansing" and for
"sync". These are quite different operations and the behaviour may differ
depending upon the mode.
@@ -295,72 +346,81 @@
radix tree. This incoherency can lead to all sorts of hard-to-debug problems
in the filesystem like having dirty inodes at umount and losing written data.
- ->writepages() is used for periodic writeback and for syscall-initiated
+->writepages() is used for periodic writeback and for syscall-initiated
sync operations. The address_space should start I/O against at least
-*nr_to_write pages. *nr_to_write must be decremented for each page which is
-written. The address_space implementation may write more (or less) pages
-than *nr_to_write asks for, but it should try to be reasonably close. If
-nr_to_write is NULL, all dirty pages must be written.
+``*nr_to_write`` pages. ``*nr_to_write`` must be decremented for each page
+which is written. The address_space implementation may write more (or less)
+pages than ``*nr_to_write`` asks for, but it should try to be reasonably close.
+If nr_to_write is NULL, all dirty pages must be written.
writepages should _only_ write pages which are present on
mapping->io_pages.
- ->set_page_dirty() is called from various places in the kernel
+->set_page_dirty() is called from various places in the kernel
when the target page is marked as needing writeback. It may be called
under spinlock (it cannot block) and is sometimes called with the page
not locked.
- ->bmap() is currently used by legacy ioctl() (FIBMAP) provided by some
+->bmap() is currently used by legacy ioctl() (FIBMAP) provided by some
filesystems and by the swapper. The latter will eventually go away. Please,
keep it that way and don't breed new callers.
- ->invalidatepage() is called when the filesystem must attempt to drop
+->invalidatepage() is called when the filesystem must attempt to drop
some or all of the buffers from the page when it is being truncated. It
returns zero on success. If ->invalidatepage is zero, the kernel uses
block_invalidatepage() instead.
- ->releasepage() is called when the kernel is about to try to drop the
+->releasepage() is called when the kernel is about to try to drop the
buffers from the page in preparation for freeing it. It returns zero to
indicate that the buffers are (or may be) freeable. If ->releasepage is zero,
the kernel assumes that the fs has no private interest in the buffers.
- ->freepage() is called when the kernel is done dropping the page
+->freepage() is called when the kernel is done dropping the page
from the page cache.
- ->launder_page() may be called prior to releasing a page if
+->launder_page() may be called prior to releasing a page if
it is still found to be dirty. It returns zero if the page was successfully
cleaned, or an error value if not. Note that in order to prevent the page
getting mapped back in and redirtied, it needs to be kept locked
across the entire operation.
- ->swap_activate will be called with a non-zero argument on
+->swap_activate will be called with a non-zero argument on
files backing (non block device backed) swapfiles. A return value
of zero indicates success, in which case this file can be used for
backing swapspace. The swapspace operations will be proxied to the
address space operations.
- ->swap_deactivate() will be called in the sys_swapoff()
+->swap_deactivate() will be called in the sys_swapoff()
path after ->swap_activate() returned success.
------------------------ file_lock_operations ------------------------------
-prototypes:
+file_lock_operations
+====================
+
+prototypes::
+
void (*fl_copy_lock)(struct file_lock *, struct file_lock *);
void (*fl_release_private)(struct file_lock *);
locking rules:
- inode->i_lock may block
+
+=================== ============= =========
+ops inode->i_lock may block
+=================== ============= =========
fl_copy_lock: yes no
-fl_release_private: maybe maybe[1]
+fl_release_private: maybe maybe[1]_
+=================== ============= =========
-[1]: ->fl_release_private for flock or POSIX locks is currently allowed
-to block. Leases however can still be freed while the i_lock is held and
-so fl_release_private called on a lease should not block.
+.. [1]:
+ ->fl_release_private for flock or POSIX locks is currently allowed
+ to block. Leases however can still be freed while the i_lock is held and
+ so fl_release_private called on a lease should not block.
------------------------ lock_manager_operations ---------------------------
-prototypes:
- int (*lm_compare_owner)(struct file_lock *, struct file_lock *);
- unsigned long (*lm_owner_key)(struct file_lock *);
+lock_manager_operations
+=======================
+
+prototypes::
+
void (*lm_notify)(struct file_lock *); /* unblock callback */
int (*lm_grant)(struct file_lock *, struct file_lock *, int);
void (*lm_break)(struct file_lock *); /* break_lease callback */
@@ -368,36 +428,33 @@
locking rules:
- inode->i_lock blocked_lock_lock may block
-lm_compare_owner: yes[1] maybe no
-lm_owner_key yes[1] yes no
+========== ============= ================= =========
+ops inode->i_lock blocked_lock_lock may block
+========== ============= ================= =========
lm_notify: yes yes no
lm_grant: no no no
lm_break: yes no no
lm_change yes no no
+========== ============= ================= =========
-[1]: ->lm_compare_owner and ->lm_owner_key are generally called with
-*an* inode->i_lock held. It may not be the i_lock of the inode
-associated with either file_lock argument! This is the case with deadlock
-detection, since the code has to chase down the owners of locks that may
-be entirely unrelated to the one on which the lock is being acquired.
-For deadlock detection however, the blocked_lock_lock is also held. The
-fact that these locks are held ensures that the file_locks do not
-disappear out from under you while doing the comparison or generating an
-owner key.
+buffer_head
+===========
---------------------------- buffer_head -----------------------------------
-prototypes:
+prototypes::
+
void (*b_end_io)(struct buffer_head *bh, int uptodate);
locking rules:
- called from interrupts. In other words, extreme care is needed here.
+
+called from interrupts. In other words, extreme care is needed here.
bh is locked, but that's all warranties we have here. Currently only RAID1,
highmem, fs/buffer.c, and fs/ntfs/aops.c are providing these. Block devices
call this method upon the IO completion.
---------------------------- block_device_operations -----------------------
-prototypes:
+block_device_operations
+=======================
+prototypes::
+
int (*open) (struct block_device *, fmode_t);
int (*release) (struct gendisk *, fmode_t);
int (*ioctl) (struct block_device *, fmode_t, unsigned, unsigned long);
@@ -411,7 +468,10 @@
void (*swap_slot_free_notify) (struct block_device *, unsigned long);
locking rules:
- bd_mutex
+
+======================= ===================
+ops bd_mutex
+======================= ===================
open: yes
release: yes
ioctl: no
@@ -422,6 +482,7 @@
revalidate_disk: no
getgeo: no
swap_slot_free_notify: no (see below)
+======================= ===================
media_changed, unlock_native_capacity and revalidate_disk are called only from
check_disk_change().
@@ -430,8 +491,11 @@
held.
---------------------------- file_operations -------------------------------
-prototypes:
+file_operations
+===============
+
+prototypes::
+
loff_t (*llseek) (struct file *, loff_t, int);
ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
@@ -467,7 +531,6 @@
size_t, unsigned int);
int (*setlease)(struct file *, long, struct file_lock **, void **);
long (*fallocate)(struct file *, int, loff_t, loff_t);
-};
locking rules:
All may block.
@@ -502,8 +565,11 @@
the lease within the individual filesystem to record the result of the
operation
---------------------------- dquot_operations -------------------------------
-prototypes:
+dquot_operations
+================
+
+prototypes::
+
int (*write_dquot) (struct dquot *);
int (*acquire_dquot) (struct dquot *);
int (*release_dquot) (struct dquot *);
@@ -515,20 +581,26 @@
What filesystem should expect from the generic quota functions:
- FS recursion Held locks when called
+============== ============ =========================
+ops FS recursion Held locks when called
+============== ============ =========================
write_dquot: yes dqonoff_sem or dqptr_sem
acquire_dquot: yes dqonoff_sem or dqptr_sem
release_dquot: yes dqonoff_sem or dqptr_sem
mark_dirty: no -
write_info: yes dqonoff_sem
+============== ============ =========================
FS recursion means calling ->quota_read() and ->quota_write() from superblock
operations.
More details about quota locking can be found in fs/dquot.c.
---------------------------- vm_operations_struct -----------------------------
-prototypes:
+vm_operations_struct
+====================
+
+prototypes::
+
void (*open)(struct vm_area_struct*);
void (*close)(struct vm_area_struct*);
vm_fault_t (*fault)(struct vm_area_struct*, struct vm_fault *);
@@ -537,7 +609,10 @@
int (*access)(struct vm_area_struct *, unsigned long, void*, int, int);
locking rules:
- mmap_sem PageLocked(page)
+
+============= ======== ===========================
+ops mmap_sem PageLocked(page)
+============= ======== ===========================
open: yes
close: yes
fault: yes can return with page locked
@@ -545,8 +620,9 @@
page_mkwrite: yes can return with page locked
pfn_mkwrite: yes
access: yes
+============= ======== ===========================
- ->fault() is called when a previously not present pte is about
+->fault() is called when a previously not present pte is about
to be faulted in. The filesystem must find and return the page associated
with the passed in "pgoff" in the vm_fault structure. If it is possible that
the page may be truncated and/or invalidated, then the filesystem must lock
@@ -554,7 +630,7 @@
subsequent truncate), and then return with VM_FAULT_LOCKED, and the page
locked. The VM will unlock the page.
- ->map_pages() is called when VM asks to map easy accessible pages.
+->map_pages() is called when VM asks to map easy accessible pages.
Filesystem should find and map pages associated with offsets from "start_pgoff"
till "end_pgoff". ->map_pages() is called with page table locked and must
not block. If it's not possible to reach a page without blocking,
@@ -563,25 +639,26 @@
"pte" field in vm_fault structure. Pointers to entries for other offsets
should be calculated relative to "pte".
- ->page_mkwrite() is called when a previously read-only pte is
+->page_mkwrite() is called when a previously read-only pte is
about to become writeable. The filesystem again must ensure that there are
no truncate/invalidate races, and then return with the page locked. If
the page has been truncated, the filesystem should not look up a new page
like the ->fault() handler, but simply return with VM_FAULT_NOPAGE, which
will cause the VM to retry the fault.
- ->pfn_mkwrite() is the same as page_mkwrite but when the pte is
+->pfn_mkwrite() is the same as page_mkwrite but when the pte is
VM_PFNMAP or VM_MIXEDMAP with a page-less entry. Expected return is
VM_FAULT_NOPAGE. Or one of the VM_FAULT_ERROR types. The default behavior
after this call is to make the pte read-write, unless pfn_mkwrite returns
an error.
- ->access() is called when get_user_pages() fails in
+->access() is called when get_user_pages() fails in
access_process_vm(), typically used to debug a process through
/proc/pid/mem or ptrace. This function is needed only for
VM_IO | VM_PFNMAP VMAs.
-================================================================================
+--------------------------------------------------------------------------------
+
Dubious stuff
(if you break something or notice that it is broken and do not fix it yourself
diff --git a/Documentation/filesystems/mandatory-locking.txt b/Documentation/filesystems/mandatory-locking.txt
index 0979d1d..a251ca3 100644
--- a/Documentation/filesystems/mandatory-locking.txt
+++ b/Documentation/filesystems/mandatory-locking.txt
@@ -169,3 +169,13 @@
permissions (remove the setgid bit) before trying to read or write to it.
Of course, that might be a bit tricky if the system is hung :-(
+7. The "mand" mount option
+--------------------------
+Mandatory locking is disabled on all filesystems by default, and must be
+administratively enabled by mounting with "-o mand". That mount option
+is only allowed if the mounting task has the CAP_SYS_ADMIN capability.
+
+Since kernel v4.5, it is possible to disable mandatory locking
+altogether by setting CONFIG_MANDATORY_FILE_LOCKING to "n". A kernel
+with this disabled will reject attempts to mount filesystems with the
+"mand" mount option with the error status EPERM.
diff --git a/Documentation/filesystems/mount_api.txt b/Documentation/filesystems/mount_api.txt
new file mode 100644
index 0000000..00ff0cf
--- /dev/null
+++ b/Documentation/filesystems/mount_api.txt
@@ -0,0 +1,732 @@
+ ====================
+ FILESYSTEM MOUNT API
+ ====================
+
+CONTENTS
+
+ (1) Overview.
+
+ (2) The filesystem context.
+
+ (3) The filesystem context operations.
+
+ (4) Filesystem context security.
+
+ (5) VFS filesystem context API.
+
+ (6) Superblock creation helpers.
+
+ (7) Parameter description.
+
+ (8) Parameter helper functions.
+
+
+========
+OVERVIEW
+========
+
+The creation of new mounts is now to be done in a multistep process:
+
+ (1) Create a filesystem context.
+
+ (2) Parse the parameters and attach them to the context. Parameters are
+ expected to be passed individually from userspace, though legacy binary
+ parameters can also be handled.
+
+ (3) Validate and pre-process the context.
+
+ (4) Get or create a superblock and mountable root.
+
+ (5) Perform the mount.
+
+ (6) Return an error message attached to the context.
+
+ (7) Destroy the context.
+
+To support this, the file_system_type struct gains two new fields:
+
+ int (*init_fs_context)(struct fs_context *fc);
+ const struct fs_parameter_description *parameters;
+
+The first is invoked to set up the filesystem-specific parts of a filesystem
+context, including the additional space, and the second points to the
+parameter description for validation at registration time and querying by a
+future system call.
+
+Note that security initialisation is done *after* the filesystem is called so
+that the namespaces may be adjusted first.
+
+
+======================
+THE FILESYSTEM CONTEXT
+======================
+
+The creation and reconfiguration of a superblock is governed by a filesystem
+context. This is represented by the fs_context structure:
+
+ struct fs_context {
+ const struct fs_context_operations *ops;
+ struct file_system_type *fs_type;
+ void *fs_private;
+ struct dentry *root;
+ struct user_namespace *user_ns;
+ struct net *net_ns;
+ const struct cred *cred;
+ char *source;
+ char *subtype;
+ void *security;
+ void *s_fs_info;
+ unsigned int sb_flags;
+ unsigned int sb_flags_mask;
+ unsigned int s_iflags;
+ unsigned int lsm_flags;
+ enum fs_context_purpose purpose:8;
+ ...
+ };
+
+The fs_context fields are as follows:
+
+ (*) const struct fs_context_operations *ops
+
+ These are operations that can be done on a filesystem context (see
+ below). This must be set by the ->init_fs_context() file_system_type
+ operation.
+
+ (*) struct file_system_type *fs_type
+
+ A pointer to the file_system_type of the filesystem that is being
+ constructed or reconfigured. This retains a reference on the type owner.
+
+ (*) void *fs_private
+
+ A pointer to the file system's private data. This is where the filesystem
+ will need to store any options it parses.
+
+ (*) struct dentry *root
+
+ A pointer to the root of the mountable tree (and indirectly, the
+ superblock thereof). This is filled in by the ->get_tree() op. If this
+ is set, an active reference on root->d_sb must also be held.
+
+ (*) struct user_namespace *user_ns
+ (*) struct net *net_ns
+
+ There are a subset of the namespaces in use by the invoking process. They
+ retain references on each namespace. The subscribed namespaces may be
+ replaced by the filesystem to reflect other sources, such as the parent
+ mount superblock on an automount.
+
+ (*) const struct cred *cred
+
+ The mounter's credentials. This retains a reference on the credentials.
+
+ (*) char *source
+
+ This specifies the source. It may be a block device (e.g. /dev/sda1) or
+ something more exotic, such as the "host:/path" that NFS desires.
+
+ (*) char *subtype
+
+ This is a string to be added to the type displayed in /proc/mounts to
+ qualify it (used by FUSE). This is available for the filesystem to set if
+ desired.
+
+ (*) void *security
+
+ A place for the LSMs to hang their security data for the superblock. The
+ relevant security operations are described below.
+
+ (*) void *s_fs_info
+
+ The proposed s_fs_info for a new superblock, set in the superblock by
+ sget_fc(). This can be used to distinguish superblocks.
+
+ (*) unsigned int sb_flags
+ (*) unsigned int sb_flags_mask
+
+ Which bits SB_* flags are to be set/cleared in super_block::s_flags.
+
+ (*) unsigned int s_iflags
+
+ These will be bitwise-OR'd with s->s_iflags when a superblock is created.
+
+ (*) enum fs_context_purpose
+
+ This indicates the purpose for which the context is intended. The
+ available values are:
+
+ FS_CONTEXT_FOR_MOUNT, -- New superblock for explicit mount
+ FS_CONTEXT_FOR_SUBMOUNT -- New automatic submount of extant mount
+ FS_CONTEXT_FOR_RECONFIGURE -- Change an existing mount
+
+The mount context is created by calling vfs_new_fs_context() or
+vfs_dup_fs_context() and is destroyed with put_fs_context(). Note that the
+structure is not refcounted.
+
+VFS, security and filesystem mount options are set individually with
+vfs_parse_mount_option(). Options provided by the old mount(2) system call as
+a page of data can be parsed with generic_parse_monolithic().
+
+When mounting, the filesystem is allowed to take data from any of the pointers
+and attach it to the superblock (or whatever), provided it clears the pointer
+in the mount context.
+
+The filesystem is also allowed to allocate resources and pin them with the
+mount context. For instance, NFS might pin the appropriate protocol version
+module.
+
+
+=================================
+THE FILESYSTEM CONTEXT OPERATIONS
+=================================
+
+The filesystem context points to a table of operations:
+
+ struct fs_context_operations {
+ void (*free)(struct fs_context *fc);
+ int (*dup)(struct fs_context *fc, struct fs_context *src_fc);
+ int (*parse_param)(struct fs_context *fc,
+ struct struct fs_parameter *param);
+ int (*parse_monolithic)(struct fs_context *fc, void *data);
+ int (*get_tree)(struct fs_context *fc);
+ int (*reconfigure)(struct fs_context *fc);
+ };
+
+These operations are invoked by the various stages of the mount procedure to
+manage the filesystem context. They are as follows:
+
+ (*) void (*free)(struct fs_context *fc);
+
+ Called to clean up the filesystem-specific part of the filesystem context
+ when the context is destroyed. It should be aware that parts of the
+ context may have been removed and NULL'd out by ->get_tree().
+
+ (*) int (*dup)(struct fs_context *fc, struct fs_context *src_fc);
+
+ Called when a filesystem context has been duplicated to duplicate the
+ filesystem-private data. An error may be returned to indicate failure to
+ do this.
+
+ [!] Note that even if this fails, put_fs_context() will be called
+ immediately thereafter, so ->dup() *must* make the
+ filesystem-private data safe for ->free().
+
+ (*) int (*parse_param)(struct fs_context *fc,
+ struct struct fs_parameter *param);
+
+ Called when a parameter is being added to the filesystem context. param
+ points to the key name and maybe a value object. VFS-specific options
+ will have been weeded out and fc->sb_flags updated in the context.
+ Security options will also have been weeded out and fc->security updated.
+
+ The parameter can be parsed with fs_parse() and fs_lookup_param(). Note
+ that the source(s) are presented as parameters named "source".
+
+ If successful, 0 should be returned or a negative error code otherwise.
+
+ (*) int (*parse_monolithic)(struct fs_context *fc, void *data);
+
+ Called when the mount(2) system call is invoked to pass the entire data
+ page in one go. If this is expected to be just a list of "key[=val]"
+ items separated by commas, then this may be set to NULL.
+
+ The return value is as for ->parse_param().
+
+ If the filesystem (e.g. NFS) needs to examine the data first and then
+ finds it's the standard key-val list then it may pass it off to
+ generic_parse_monolithic().
+
+ (*) int (*get_tree)(struct fs_context *fc);
+
+ Called to get or create the mountable root and superblock, using the
+ information stored in the filesystem context (reconfiguration goes via a
+ different vector). It may detach any resources it desires from the
+ filesystem context and transfer them to the superblock it creates.
+
+ On success it should set fc->root to the mountable root and return 0. In
+ the case of an error, it should return a negative error code.
+
+ The phase on a userspace-driven context will be set to only allow this to
+ be called once on any particular context.
+
+ (*) int (*reconfigure)(struct fs_context *fc);
+
+ Called to effect reconfiguration of a superblock using information stored
+ in the filesystem context. It may detach any resources it desires from
+ the filesystem context and transfer them to the superblock. The
+ superblock can be found from fc->root->d_sb.
+
+ On success it should return 0. In the case of an error, it should return
+ a negative error code.
+
+ [NOTE] reconfigure is intended as a replacement for remount_fs.
+
+
+===========================
+FILESYSTEM CONTEXT SECURITY
+===========================
+
+The filesystem context contains a security pointer that the LSMs can use for
+building up a security context for the superblock to be mounted. There are a
+number of operations used by the new mount code for this purpose:
+
+ (*) int security_fs_context_alloc(struct fs_context *fc,
+ struct dentry *reference);
+
+ Called to initialise fc->security (which is preset to NULL) and allocate
+ any resources needed. It should return 0 on success or a negative error
+ code on failure.
+
+ reference will be non-NULL if the context is being created for superblock
+ reconfiguration (FS_CONTEXT_FOR_RECONFIGURE) in which case it indicates
+ the root dentry of the superblock to be reconfigured. It will also be
+ non-NULL in the case of a submount (FS_CONTEXT_FOR_SUBMOUNT) in which case
+ it indicates the automount point.
+
+ (*) int security_fs_context_dup(struct fs_context *fc,
+ struct fs_context *src_fc);
+
+ Called to initialise fc->security (which is preset to NULL) and allocate
+ any resources needed. The original filesystem context is pointed to by
+ src_fc and may be used for reference. It should return 0 on success or a
+ negative error code on failure.
+
+ (*) void security_fs_context_free(struct fs_context *fc);
+
+ Called to clean up anything attached to fc->security. Note that the
+ contents may have been transferred to a superblock and the pointer cleared
+ during get_tree.
+
+ (*) int security_fs_context_parse_param(struct fs_context *fc,
+ struct fs_parameter *param);
+
+ Called for each mount parameter, including the source. The arguments are
+ as for the ->parse_param() method. It should return 0 to indicate that
+ the parameter should be passed on to the filesystem, 1 to indicate that
+ the parameter should be discarded or an error to indicate that the
+ parameter should be rejected.
+
+ The value pointed to by param may be modified (if a string) or stolen
+ (provided the value pointer is NULL'd out). If it is stolen, 1 must be
+ returned to prevent it being passed to the filesystem.
+
+ (*) int security_fs_context_validate(struct fs_context *fc);
+
+ Called after all the options have been parsed to validate the collection
+ as a whole and to do any necessary allocation so that
+ security_sb_get_tree() and security_sb_reconfigure() are less likely to
+ fail. It should return 0 or a negative error code.
+
+ In the case of reconfiguration, the target superblock will be accessible
+ via fc->root.
+
+ (*) int security_sb_get_tree(struct fs_context *fc);
+
+ Called during the mount procedure to verify that the specified superblock
+ is allowed to be mounted and to transfer the security data there. It
+ should return 0 or a negative error code.
+
+ (*) void security_sb_reconfigure(struct fs_context *fc);
+
+ Called to apply any reconfiguration to an LSM's context. It must not
+ fail. Error checking and resource allocation must be done in advance by
+ the parameter parsing and validation hooks.
+
+ (*) int security_sb_mountpoint(struct fs_context *fc, struct path *mountpoint,
+ unsigned int mnt_flags);
+
+ Called during the mount procedure to verify that the root dentry attached
+ to the context is permitted to be attached to the specified mountpoint.
+ It should return 0 on success or a negative error code on failure.
+
+
+==========================
+VFS FILESYSTEM CONTEXT API
+==========================
+
+There are four operations for creating a filesystem context and one for
+destroying a context:
+
+ (*) struct fs_context *fs_context_for_mount(
+ struct file_system_type *fs_type,
+ unsigned int sb_flags);
+
+ Allocate a filesystem context for the purpose of setting up a new mount,
+ whether that be with a new superblock or sharing an existing one. This
+ sets the superblock flags, initialises the security and calls
+ fs_type->init_fs_context() to initialise the filesystem private data.
+
+ fs_type specifies the filesystem type that will manage the context and
+ sb_flags presets the superblock flags stored therein.
+
+ (*) struct fs_context *fs_context_for_reconfigure(
+ struct dentry *dentry,
+ unsigned int sb_flags,
+ unsigned int sb_flags_mask);
+
+ Allocate a filesystem context for the purpose of reconfiguring an
+ existing superblock. dentry provides a reference to the superblock to be
+ configured. sb_flags and sb_flags_mask indicate which superblock flags
+ need changing and to what.
+
+ (*) struct fs_context *fs_context_for_submount(
+ struct file_system_type *fs_type,
+ struct dentry *reference);
+
+ Allocate a filesystem context for the purpose of creating a new mount for
+ an automount point or other derived superblock. fs_type specifies the
+ filesystem type that will manage the context and the reference dentry
+ supplies the parameters. Namespaces are propagated from the reference
+ dentry's superblock also.
+
+ Note that it's not a requirement that the reference dentry be of the same
+ filesystem type as fs_type.
+
+ (*) struct fs_context *vfs_dup_fs_context(struct fs_context *src_fc);
+
+ Duplicate a filesystem context, copying any options noted and duplicating
+ or additionally referencing any resources held therein. This is available
+ for use where a filesystem has to get a mount within a mount, such as NFS4
+ does by internally mounting the root of the target server and then doing a
+ private pathwalk to the target directory.
+
+ The purpose in the new context is inherited from the old one.
+
+ (*) void put_fs_context(struct fs_context *fc);
+
+ Destroy a filesystem context, releasing any resources it holds. This
+ calls the ->free() operation. This is intended to be called by anyone who
+ created a filesystem context.
+
+ [!] filesystem contexts are not refcounted, so this causes unconditional
+ destruction.
+
+In all the above operations, apart from the put op, the return is a mount
+context pointer or a negative error code.
+
+For the remaining operations, if an error occurs, a negative error code will be
+returned.
+
+ (*) int vfs_parse_fs_param(struct fs_context *fc,
+ struct fs_parameter *param);
+
+ Supply a single mount parameter to the filesystem context. This include
+ the specification of the source/device which is specified as the "source"
+ parameter (which may be specified multiple times if the filesystem
+ supports that).
+
+ param specifies the parameter key name and the value. The parameter is
+ first checked to see if it corresponds to a standard mount flag (in which
+ case it is used to set an SB_xxx flag and consumed) or a security option
+ (in which case the LSM consumes it) before it is passed on to the
+ filesystem.
+
+ The parameter value is typed and can be one of:
+
+ fs_value_is_flag, Parameter not given a value.
+ fs_value_is_string, Value is a string
+ fs_value_is_blob, Value is a binary blob
+ fs_value_is_filename, Value is a filename* + dirfd
+ fs_value_is_filename_empty, Value is a filename* + dirfd + AT_EMPTY_PATH
+ fs_value_is_file, Value is an open file (file*)
+
+ If there is a value, that value is stored in a union in the struct in one
+ of param->{string,blob,name,file}. Note that the function may steal and
+ clear the pointer, but then becomes responsible for disposing of the
+ object.
+
+ (*) int vfs_parse_fs_string(struct fs_context *fc, const char *key,
+ const char *value, size_t v_size);
+
+ A wrapper around vfs_parse_fs_param() that copies the value string it is
+ passed.
+
+ (*) int generic_parse_monolithic(struct fs_context *fc, void *data);
+
+ Parse a sys_mount() data page, assuming the form to be a text list
+ consisting of key[=val] options separated by commas. Each item in the
+ list is passed to vfs_mount_option(). This is the default when the
+ ->parse_monolithic() method is NULL.
+
+ (*) int vfs_get_tree(struct fs_context *fc);
+
+ Get or create the mountable root and superblock, using the parameters in
+ the filesystem context to select/configure the superblock. This invokes
+ the ->get_tree() method.
+
+ (*) struct vfsmount *vfs_create_mount(struct fs_context *fc);
+
+ Create a mount given the parameters in the specified filesystem context.
+ Note that this does not attach the mount to anything.
+
+
+===========================
+SUPERBLOCK CREATION HELPERS
+===========================
+
+A number of VFS helpers are available for use by filesystems for the creation
+or looking up of superblocks.
+
+ (*) struct super_block *
+ sget_fc(struct fs_context *fc,
+ int (*test)(struct super_block *sb, struct fs_context *fc),
+ int (*set)(struct super_block *sb, struct fs_context *fc));
+
+ This is the core routine. If test is non-NULL, it searches for an
+ existing superblock matching the criteria held in the fs_context, using
+ the test function to match them. If no match is found, a new superblock
+ is created and the set function is called to set it up.
+
+ Prior to the set function being called, fc->s_fs_info will be transferred
+ to sb->s_fs_info - and fc->s_fs_info will be cleared if set returns
+ success (ie. 0).
+
+The following helpers all wrap sget_fc():
+
+ (*) int vfs_get_super(struct fs_context *fc,
+ enum vfs_get_super_keying keying,
+ int (*fill_super)(struct super_block *sb,
+ struct fs_context *fc))
+
+ This creates/looks up a deviceless superblock. The keying indicates how
+ many superblocks of this type may exist and in what manner they may be
+ shared:
+
+ (1) vfs_get_single_super
+
+ Only one such superblock may exist in the system. Any further
+ attempt to get a new superblock gets this one (and any parameter
+ differences are ignored).
+
+ (2) vfs_get_keyed_super
+
+ Multiple superblocks of this type may exist and they're keyed on
+ their s_fs_info pointer (for example this may refer to a
+ namespace).
+
+ (3) vfs_get_independent_super
+
+ Multiple independent superblocks of this type may exist. This
+ function never matches an existing one and always creates a new
+ one.
+
+
+=====================
+PARAMETER DESCRIPTION
+=====================
+
+Parameters are described using structures defined in linux/fs_parser.h.
+There's a core description struct that links everything together:
+
+ struct fs_parameter_description {
+ const char name[16];
+ const struct fs_parameter_spec *specs;
+ const struct fs_parameter_enum *enums;
+ };
+
+For example:
+
+ enum {
+ Opt_autocell,
+ Opt_bar,
+ Opt_dyn,
+ Opt_foo,
+ Opt_source,
+ };
+
+ static const struct fs_parameter_description afs_fs_parameters = {
+ .name = "kAFS",
+ .specs = afs_param_specs,
+ .enums = afs_param_enums,
+ };
+
+The members are as follows:
+
+ (1) const char name[16];
+
+ The name to be used in error messages generated by the parse helper
+ functions.
+
+ (2) const struct fs_parameter_specification *specs;
+
+ Table of parameter specifications, terminated with a null entry, where the
+ entries are of type:
+
+ struct fs_parameter_spec {
+ const char *name;
+ u8 opt;
+ enum fs_parameter_type type:8;
+ unsigned short flags;
+ };
+
+ The 'name' field is a string to match exactly to the parameter key (no
+ wildcards, patterns and no case-independence) and 'opt' is the value that
+ will be returned by the fs_parser() function in the case of a successful
+ match.
+
+ The 'type' field indicates the desired value type and must be one of:
+
+ TYPE NAME EXPECTED VALUE RESULT IN
+ ======================= ======================= =====================
+ fs_param_is_flag No value n/a
+ fs_param_is_bool Boolean value result->boolean
+ fs_param_is_u32 32-bit unsigned int result->uint_32
+ fs_param_is_u32_octal 32-bit octal int result->uint_32
+ fs_param_is_u32_hex 32-bit hex int result->uint_32
+ fs_param_is_s32 32-bit signed int result->int_32
+ fs_param_is_u64 64-bit unsigned int result->uint_64
+ fs_param_is_enum Enum value name result->uint_32
+ fs_param_is_string Arbitrary string param->string
+ fs_param_is_blob Binary blob param->blob
+ fs_param_is_blockdev Blockdev path * Needs lookup
+ fs_param_is_path Path * Needs lookup
+ fs_param_is_fd File descriptor result->int_32
+
+ Note that if the value is of fs_param_is_bool type, fs_parse() will try
+ to match any string value against "0", "1", "no", "yes", "false", "true".
+
+ Each parameter can also be qualified with 'flags':
+
+ fs_param_v_optional The value is optional
+ fs_param_neg_with_no result->negated set if key is prefixed with "no"
+ fs_param_neg_with_empty result->negated set if value is ""
+ fs_param_deprecated The parameter is deprecated.
+
+ These are wrapped with a number of convenience wrappers:
+
+ MACRO SPECIFIES
+ ======================= ===============================================
+ fsparam_flag() fs_param_is_flag
+ fsparam_flag_no() fs_param_is_flag, fs_param_neg_with_no
+ fsparam_bool() fs_param_is_bool
+ fsparam_u32() fs_param_is_u32
+ fsparam_u32oct() fs_param_is_u32_octal
+ fsparam_u32hex() fs_param_is_u32_hex
+ fsparam_s32() fs_param_is_s32
+ fsparam_u64() fs_param_is_u64
+ fsparam_enum() fs_param_is_enum
+ fsparam_string() fs_param_is_string
+ fsparam_blob() fs_param_is_blob
+ fsparam_bdev() fs_param_is_blockdev
+ fsparam_path() fs_param_is_path
+ fsparam_fd() fs_param_is_fd
+
+ all of which take two arguments, name string and option number - for
+ example:
+
+ static const struct fs_parameter_spec afs_param_specs[] = {
+ fsparam_flag ("autocell", Opt_autocell),
+ fsparam_flag ("dyn", Opt_dyn),
+ fsparam_string ("source", Opt_source),
+ fsparam_flag_no ("foo", Opt_foo),
+ {}
+ };
+
+ An addition macro, __fsparam() is provided that takes an additional pair
+ of arguments to specify the type and the flags for anything that doesn't
+ match one of the above macros.
+
+ (6) const struct fs_parameter_enum *enums;
+
+ Table of enum value names to integer mappings, terminated with a null
+ entry. This is of type:
+
+ struct fs_parameter_enum {
+ u8 opt;
+ char name[14];
+ u8 value;
+ };
+
+ Where the array is an unsorted list of { parameter ID, name }-keyed
+ elements that indicate the value to map to, e.g.:
+
+ static const struct fs_parameter_enum afs_param_enums[] = {
+ { Opt_bar, "x", 1},
+ { Opt_bar, "y", 23},
+ { Opt_bar, "z", 42},
+ };
+
+ If a parameter of type fs_param_is_enum is encountered, fs_parse() will
+ try to look the value up in the enum table and the result will be stored
+ in the parse result.
+
+The parser should be pointed to by the parser pointer in the file_system_type
+struct as this will provide validation on registration (if
+CONFIG_VALIDATE_FS_PARSER=y) and will allow the description to be queried from
+userspace using the fsinfo() syscall.
+
+
+==========================
+PARAMETER HELPER FUNCTIONS
+==========================
+
+A number of helper functions are provided to help a filesystem or an LSM
+process the parameters it is given.
+
+ (*) int lookup_constant(const struct constant_table tbl[],
+ const char *name, int not_found);
+
+ Look up a constant by name in a table of name -> integer mappings. The
+ table is an array of elements of the following type:
+
+ struct constant_table {
+ const char *name;
+ int value;
+ };
+
+ If a match is found, the corresponding value is returned. If a match
+ isn't found, the not_found value is returned instead.
+
+ (*) bool validate_constant_table(const struct constant_table *tbl,
+ size_t tbl_size,
+ int low, int high, int special);
+
+ Validate a constant table. Checks that all the elements are appropriately
+ ordered, that there are no duplicates and that the values are between low
+ and high inclusive, though provision is made for one allowable special
+ value outside of that range. If no special value is required, special
+ should just be set to lie inside the low-to-high range.
+
+ If all is good, true is returned. If the table is invalid, errors are
+ logged to dmesg and false is returned.
+
+ (*) bool fs_validate_description(const struct fs_parameter_description *desc);
+
+ This performs some validation checks on a parameter description. It
+ returns true if the description is good and false if it is not. It will
+ log errors to dmesg if validation fails.
+
+ (*) int fs_parse(struct fs_context *fc,
+ const struct fs_parameter_description *desc,
+ struct fs_parameter *param,
+ struct fs_parse_result *result);
+
+ This is the main interpreter of parameters. It uses the parameter
+ description to look up a parameter by key name and to convert that to an
+ option number (which it returns).
+
+ If successful, and if the parameter type indicates the result is a
+ boolean, integer or enum type, the value is converted by this function and
+ the result stored in result->{boolean,int_32,uint_32,uint_64}.
+
+ If a match isn't initially made, the key is prefixed with "no" and no
+ value is present then an attempt will be made to look up the key with the
+ prefix removed. If this matches a parameter for which the type has flag
+ fs_param_neg_with_no set, then a match will be made and result->negated
+ will be set to true.
+
+ If the parameter isn't matched, -ENOPARAM will be returned; if the
+ parameter is matched, but the value is erroneous, -EINVAL will be
+ returned; otherwise the parameter's option number will be returned.
+
+ (*) int fs_lookup_param(struct fs_context *fc,
+ struct fs_parameter *value,
+ bool want_bdev,
+ struct path *_path);
+
+ This takes a parameter that carries a string or filename type and attempts
+ to do a path lookup on it. If the parameter expects a blockdev, a check
+ is made that the inode actually represents one.
+
+ Returns 0 if successful and *_path will be set; returns a negative error
+ code if not.
diff --git a/Documentation/filesystems/nfs/00-INDEX b/Documentation/filesystems/nfs/00-INDEX
deleted file mode 100644
index 53f3b59..0000000
--- a/Documentation/filesystems/nfs/00-INDEX
+++ /dev/null
@@ -1,26 +0,0 @@
-00-INDEX
- - this file (nfs-related documentation).
-Exporting
- - explanation of how to make filesystems exportable.
-fault_injection.txt
- - information for using fault injection on the server
-knfsd-stats.txt
- - statistics which the NFS server makes available to user space.
-nfs.txt
- - nfs client, and DNS resolution for fs_locations.
-nfs41-server.txt
- - info on the Linux server implementation of NFSv4 minor version 1.
-nfs-rdma.txt
- - how to install and setup the Linux NFS/RDMA client and server software
-nfsd-admin-interfaces.txt
- - Administrative interfaces for nfsd.
-nfsroot.txt
- - short guide on setting up a diskless box with NFS root filesystem.
-pnfs.txt
- - short explanation of some of the internals of the pnfs client code
-rpc-cache.txt
- - introduction to the caching mechanisms in the sunrpc layer.
-idmapper.txt
- - information for configuring request-keys to be used by idmapper
-rpc-server-gss.txt
- - Information on GSS authentication support in the NFS Server
diff --git a/Documentation/filesystems/nfs/Exporting b/Documentation/filesystems/nfs/exporting.rst
similarity index 90%
rename from Documentation/filesystems/nfs/Exporting
rename to Documentation/filesystems/nfs/exporting.rst
index 6388914..33d588a 100644
--- a/Documentation/filesystems/nfs/Exporting
+++ b/Documentation/filesystems/nfs/exporting.rst
@@ -1,3 +1,4 @@
+:orphan:
Making Filesystems Exportable
=============================
@@ -42,9 +43,9 @@
for the object. This leads to two related but distinct features of
the dcache that are not needed for normal filesystem access.
-1/ The dcache must sometimes contain objects that are not part of the
+1. The dcache must sometimes contain objects that are not part of the
proper prefix. i.e that are not connected to the root.
-2/ The dcache must be prepared for a newly found (via ->lookup) directory
+2. The dcache must be prepared for a newly found (via ->lookup) directory
to already have a (non-connected) dentry, and must be able to move
that dentry into place (based on the parent and name in the
->lookup). This is particularly needed for directories as
@@ -52,7 +53,7 @@
To implement these features, the dcache has:
-a/ A dentry flag DCACHE_DISCONNECTED which is set on
+a. A dentry flag DCACHE_DISCONNECTED which is set on
any dentry that might not be part of the proper prefix.
This is set when anonymous dentries are created, and cleared when a
dentry is noticed to be a child of a dentry which is in the proper
@@ -71,48 +72,52 @@
dentries. That guarantees that we won't need to hunt them down upon
umount.
-b/ A primitive for creation of secondary roots - d_obtain_root(inode).
+b. A primitive for creation of secondary roots - d_obtain_root(inode).
Those do _not_ bear DCACHE_DISCONNECTED. They are placed on the
per-superblock list (->s_roots), so they can be located at umount
time for eviction purposes.
-c/ Helper routines to allocate anonymous dentries, and to help attach
+c. Helper routines to allocate anonymous dentries, and to help attach
loose directory dentries at lookup time. They are:
+
d_obtain_alias(inode) will return a dentry for the given inode.
If the inode already has a dentry, one of those is returned.
+
If it doesn't, a new anonymous (IS_ROOT and
- DCACHE_DISCONNECTED) dentry is allocated and attached.
+ DCACHE_DISCONNECTED) dentry is allocated and attached.
+
In the case of a directory, care is taken that only one dentry
can ever be attached.
+
d_splice_alias(inode, dentry) will introduce a new dentry into the tree;
either the passed-in dentry or a preexisting alias for the given inode
(such as an anonymous one created by d_obtain_alias), if appropriate.
It returns NULL when the passed-in dentry is used, following the calling
convention of ->lookup.
-
+
Filesystem Issues
-----------------
For a filesystem to be exportable it must:
-
- 1/ provide the filehandle fragment routines described below.
- 2/ make sure that d_splice_alias is used rather than d_add
+
+ 1. provide the filehandle fragment routines described below.
+ 2. make sure that d_splice_alias is used rather than d_add
when ->lookup finds an inode for a given parent and name.
- If inode is NULL, d_splice_alias(inode, dentry) is equivalent to
+ If inode is NULL, d_splice_alias(inode, dentry) is equivalent to::
d_add(dentry, inode), NULL
Similarly, d_splice_alias(ERR_PTR(err), dentry) = ERR_PTR(err)
- Typically the ->lookup routine will simply end with a:
+ Typically the ->lookup routine will simply end with a::
return d_splice_alias(inode, dentry);
}
- A file system implementation declares that instances of the filesystem
+A file system implementation declares that instances of the filesystem
are exportable by setting the s_export_op field in the struct
super_block. This field must point to a "struct export_operations"
struct which has the following members:
diff --git a/Documentation/filesystems/nfs/nfsroot.txt b/Documentation/filesystems/nfs/nfsroot.txt
index d296312..ae43324 100644
--- a/Documentation/filesystems/nfs/nfsroot.txt
+++ b/Documentation/filesystems/nfs/nfsroot.txt
@@ -239,7 +239,7 @@
A description of the process of mounting the root file system can be
found in:
- Documentation/early-userspace/README
+ Documentation/driver-api/early-userspace/early_userspace_support.rst
diff --git a/Documentation/filesystems/nfs/rpc-cache.txt b/Documentation/filesystems/nfs/rpc-cache.txt
index ebcaaee..c4dac82 100644
--- a/Documentation/filesystems/nfs/rpc-cache.txt
+++ b/Documentation/filesystems/nfs/rpc-cache.txt
@@ -84,7 +84,7 @@
A message from user space has arrived to fill out a
cache entry. It is in 'buf' of length 'len'.
cache_parse should parse this, find the item in the
- cache with sunrpc_cache_lookup, and update the item
+ cache with sunrpc_cache_lookup_rcu, and update the item
with sunrpc_cache_update.
@@ -95,7 +95,7 @@
Using a cache
-------------
-To find a value in a cache, call sunrpc_cache_lookup passing a pointer
+To find a value in a cache, call sunrpc_cache_lookup_rcu passing a pointer
to the cache_head in a sample item with the 'key' fields filled in.
This will be passed to ->match to identify the target entry. If no
entry is found, a new entry will be create, added to the cache, and
@@ -116,7 +116,7 @@
revisited (->revisit). It is expected that this method will
reschedule the request for processing.
-The value returned by sunrpc_cache_lookup can also be passed to
+The value returned by sunrpc_cache_lookup_rcu can also be passed to
sunrpc_cache_update to set the content for the item. A second item is
passed which should hold the content. If the item found by _lookup
has valid data, then it is discarded and a new item is created. This
diff --git a/Documentation/filesystems/overlayfs.txt b/Documentation/filesystems/overlayfs.txt
index eef7d9d..845d689 100644
--- a/Documentation/filesystems/overlayfs.txt
+++ b/Documentation/filesystems/overlayfs.txt
@@ -302,7 +302,7 @@
Using an upper layer path and/or a workdir path that are already used by
another overlay mount is not allowed and may fail with EBUSY. Using
-partially overlapping paths is not allowed but will not fail with EBUSY.
+partially overlapping paths is not allowed and may fail with EBUSY.
If files are accessed from two overlayfs mounts which share or overlap the
upper layer and/or workdir path the behavior of the overlay is undefined,
though it will not result in a crash or deadlock.
@@ -336,8 +336,20 @@
Non-standard behavior
---------------------
-Overlayfs can now act as a POSIX compliant filesystem with the following
-features turned on:
+Current version of overlayfs can act as a mostly POSIX compliant
+filesystem.
+
+This is the list of cases that overlayfs doesn't currently handle:
+
+a) POSIX mandates updating st_atime for reads. This is currently not
+done in the case when the file resides on a lower layer.
+
+b) If a file residing on a lower layer is opened for read-only and then
+memory mapped with MAP_SHARED, then subsequent changes to the file are not
+reflected in the memory mapping.
+
+The following options allow overlayfs to act more like a standards
+compliant filesystem:
1) "redirect_dir"
diff --git a/Documentation/filesystems/path-lookup.md b/Documentation/filesystems/path-lookup.md
deleted file mode 100644
index e2edd45..0000000
--- a/Documentation/filesystems/path-lookup.md
+++ /dev/null
@@ -1,1297 +0,0 @@
-<head>
-<style> p { max-width:50em} ol, ul {max-width: 40em}</style>
-</head>
-
-Pathname lookup in Linux.
-=========================
-
-This write-up is based on three articles published at lwn.net:
-
-- <https://lwn.net/Articles/649115/> Pathname lookup in Linux
-- <https://lwn.net/Articles/649729/> RCU-walk: faster pathname lookup in Linux
-- <https://lwn.net/Articles/650786/> A walk among the symlinks
-
-Written by Neil Brown with help from Al Viro and Jon Corbet.
-
-Introduction
-------------
-
-The most obvious aspect of pathname lookup, which very little
-exploration is needed to discover, is that it is complex. There are
-many rules, special cases, and implementation alternatives that all
-combine to confuse the unwary reader. Computer science has long been
-acquainted with such complexity and has tools to help manage it. One
-tool that we will make extensive use of is "divide and conquer". For
-the early parts of the analysis we will divide off symlinks - leaving
-them until the final part. Well before we get to symlinks we have
-another major division based on the VFS's approach to locking which
-will allow us to review "REF-walk" and "RCU-walk" separately. But we
-are getting ahead of ourselves. There are some important low level
-distinctions we need to clarify first.
-
-There are two sorts of ...
---------------------------
-
-[`openat()`]: http://man7.org/linux/man-pages/man2/openat.2.html
-
-Pathnames (sometimes "file names"), used to identify objects in the
-filesystem, will be familiar to most readers. They contain two sorts
-of elements: "slashes" that are sequences of one or more "`/`"
-characters, and "components" that are sequences of one or more
-non-"`/`" characters. These form two kinds of paths. Those that
-start with slashes are "absolute" and start from the filesystem root.
-The others are "relative" and start from the current directory, or
-from some other location specified by a file descriptor given to a
-"xxx`at`" system call such as "[`openat()`]".
-
-[`execveat()`]: http://man7.org/linux/man-pages/man2/execveat.2.html
-
-It is tempting to describe the second kind as starting with a
-component, but that isn't always accurate: a pathname can lack both
-slashes and components, it can be empty, in other words. This is
-generally forbidden in POSIX, but some of those "xxx`at`" system calls
-in Linux permit it when the `AT_EMPTY_PATH` flag is given. For
-example, if you have an open file descriptor on an executable file you
-can execute it by calling [`execveat()`] passing the file descriptor,
-an empty path, and the `AT_EMPTY_PATH` flag.
-
-These paths can be divided into two sections: the final component and
-everything else. The "everything else" is the easy bit. In all cases
-it must identify a directory that already exists, otherwise an error
-such as `ENOENT` or `ENOTDIR` will be reported.
-
-The final component is not so simple. Not only do different system
-calls interpret it quite differently (e.g. some create it, some do
-not), but it might not even exist: neither the empty pathname nor the
-pathname that is just slashes have a final component. If it does
-exist, it could be "`.`" or "`..`" which are handled quite differently
-from other components.
-
-[POSIX]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_12
-
-If a pathname ends with a slash, such as "`/tmp/foo/`" it might be
-tempting to consider that to have an empty final component. In many
-ways that would lead to correct results, but not always. In
-particular, `mkdir()` and `rmdir()` each create or remove a directory named
-by the final component, and they are required to work with pathnames
-ending in "`/`". According to [POSIX]
-
-> A pathname that contains at least one non- <slash> character and
-> that ends with one or more trailing <slash> characters shall not
-> be resolved successfully unless the last pathname component before
-> the trailing <slash> characters names an existing directory or a
-> directory entry that is to be created for a directory immediately
-> after the pathname is resolved.
-
-The Linux pathname walking code (mostly in `fs/namei.c`) deals with
-all of these issues: breaking the path into components, handling the
-"everything else" quite separately from the final component, and
-checking that the trailing slash is not used where it isn't
-permitted. It also addresses the important issue of concurrent
-access.
-
-While one process is looking up a pathname, another might be making
-changes that affect that lookup. One fairly extreme case is that if
-"a/b" were renamed to "a/c/b" while another process were looking up
-"a/b/..", that process might successfully resolve on "a/c".
-Most races are much more subtle, and a big part of the task of
-pathname lookup is to prevent them from having damaging effects. Many
-of the possible races are seen most clearly in the context of the
-"dcache" and an understanding of that is central to understanding
-pathname lookup.
-
-More than just a cache.
------------------------
-
-The "dcache" caches information about names in each filesystem to
-make them quickly available for lookup. Each entry (known as a
-"dentry") contains three significant fields: a component name, a
-pointer to a parent dentry, and a pointer to the "inode" which
-contains further information about the object in that parent with
-the given name. The inode pointer can be `NULL` indicating that the
-name doesn't exist in the parent. While there can be linkage in the
-dentry of a directory to the dentries of the children, that linkage is
-not used for pathname lookup, and so will not be considered here.
-
-The dcache has a number of uses apart from accelerating lookup. One
-that will be particularly relevant is that it is closely integrated
-with the mount table that records which filesystem is mounted where.
-What the mount table actually stores is which dentry is mounted on top
-of which other dentry.
-
-When considering the dcache, we have another of our "two types"
-distinctions: there are two types of filesystems.
-
-Some filesystems ensure that the information in the dcache is always
-completely accurate (though not necessarily complete). This can allow
-the VFS to determine if a particular file does or doesn't exist
-without checking with the filesystem, and means that the VFS can
-protect the filesystem against certain races and other problems.
-These are typically "local" filesystems such as ext3, XFS, and Btrfs.
-
-Other filesystems don't provide that guarantee because they cannot.
-These are typically filesystems that are shared across a network,
-whether remote filesystems like NFS and 9P, or cluster filesystems
-like ocfs2 or cephfs. These filesystems allow the VFS to revalidate
-cached information, and must provide their own protection against
-awkward races. The VFS can detect these filesystems by the
-`DCACHE_OP_REVALIDATE` flag being set in the dentry.
-
-REF-walk: simple concurrency management with refcounts and spinlocks
---------------------------------------------------------------------
-
-With all of those divisions carefully classified, we can now start
-looking at the actual process of walking along a path. In particular
-we will start with the handling of the "everything else" part of a
-pathname, and focus on the "REF-walk" approach to concurrency
-management. This code is found in the `link_path_walk()` function, if
-you ignore all the places that only run when "`LOOKUP_RCU`"
-(indicating the use of RCU-walk) is set.
-
-[Meet the Lockers]: https://lwn.net/Articles/453685/
-
-REF-walk is fairly heavy-handed with locks and reference counts. Not
-as heavy-handed as in the old "big kernel lock" days, but certainly not
-afraid of taking a lock when one is needed. It uses a variety of
-different concurrency controls. A background understanding of the
-various primitives is assumed, or can be gleaned from elsewhere such
-as in [Meet the Lockers].
-
-The locking mechanisms used by REF-walk include:
-
-### dentry->d_lockref ###
-
-This uses the lockref primitive to provide both a spinlock and a
-reference count. The special-sauce of this primitive is that the
-conceptual sequence "lock; inc_ref; unlock;" can often be performed
-with a single atomic memory operation.
-
-Holding a reference on a dentry ensures that the dentry won't suddenly
-be freed and used for something else, so the values in various fields
-will behave as expected. It also protects the `->d_inode` reference
-to the inode to some extent.
-
-The association between a dentry and its inode is fairly permanent.
-For example, when a file is renamed, the dentry and inode move
-together to the new location. When a file is created the dentry will
-initially be negative (i.e. `d_inode` is `NULL`), and will be assigned
-to the new inode as part of the act of creation.
-
-When a file is deleted, this can be reflected in the cache either by
-setting `d_inode` to `NULL`, or by removing it from the hash table
-(described shortly) used to look up the name in the parent directory.
-If the dentry is still in use the second option is used as it is
-perfectly legal to keep using an open file after it has been deleted
-and having the dentry around helps. If the dentry is not otherwise in
-use (i.e. if the refcount in `d_lockref` is one), only then will
-`d_inode` be set to `NULL`. Doing it this way is more efficient for a
-very common case.
-
-So as long as a counted reference is held to a dentry, a non-`NULL` `->d_inode`
-value will never be changed.
-
-### dentry->d_lock ###
-
-`d_lock` is a synonym for the spinlock that is part of `d_lockref` above.
-For our purposes, holding this lock protects against the dentry being
-renamed or unlinked. In particular, its parent (`d_parent`), and its
-name (`d_name`) cannot be changed, and it cannot be removed from the
-dentry hash table.
-
-When looking for a name in a directory, REF-walk takes `d_lock` on
-each candidate dentry that it finds in the hash table and then checks
-that the parent and name are correct. So it doesn't lock the parent
-while searching in the cache; it only locks children.
-
-When looking for the parent for a given name (to handle "`..`"),
-REF-walk can take `d_lock` to get a stable reference to `d_parent`,
-but it first tries a more lightweight approach. As seen in
-`dget_parent()`, if a reference can be claimed on the parent, and if
-subsequently `d_parent` can be seen to have not changed, then there is
-no need to actually take the lock on the child.
-
-### rename_lock ###
-
-Looking up a given name in a given directory involves computing a hash
-from the two values (the name and the dentry of the directory),
-accessing that slot in a hash table, and searching the linked list
-that is found there.
-
-When a dentry is renamed, the name and the parent dentry can both
-change so the hash will almost certainly change too. This would move the
-dentry to a different chain in the hash table. If a filename search
-happened to be looking at a dentry that was moved in this way,
-it might end up continuing the search down the wrong chain,
-and so miss out on part of the correct chain.
-
-The name-lookup process (`d_lookup()`) does _not_ try to prevent this
-from happening, but only to detect when it happens.
-`rename_lock` is a seqlock that is updated whenever any dentry is
-renamed. If `d_lookup` finds that a rename happened while it
-unsuccessfully scanned a chain in the hash table, it simply tries
-again.
-
-### inode->i_mutex ###
-
-`i_mutex` is a mutex that serializes all changes to a particular
-directory. This ensures that, for example, an `unlink()` and a `rename()`
-cannot both happen at the same time. It also keeps the directory
-stable while the filesystem is asked to look up a name that is not
-currently in the dcache.
-
-This has a complementary role to that of `d_lock`: `i_mutex` on a
-directory protects all of the names in that directory, while `d_lock`
-on a name protects just one name in a directory. Most changes to the
-dcache hold `i_mutex` on the relevant directory inode and briefly take
-`d_lock` on one or more the dentries while the change happens. One
-exception is when idle dentries are removed from the dcache due to
-memory pressure. This uses `d_lock`, but `i_mutex` plays no role.
-
-The mutex affects pathname lookup in two distinct ways. Firstly it
-serializes lookup of a name in a directory. `walk_component()` uses
-`lookup_fast()` first which, in turn, checks to see if the name is in the cache,
-using only `d_lock` locking. If the name isn't found, then `walk_component()`
-falls back to `lookup_slow()` which takes `i_mutex`, checks again that
-the name isn't in the cache, and then calls in to the filesystem to get a
-definitive answer. A new dentry will be added to the cache regardless of
-the result.
-
-Secondly, when pathname lookup reaches the final component, it will
-sometimes need to take `i_mutex` before performing the last lookup so
-that the required exclusion can be achieved. How path lookup chooses
-to take, or not take, `i_mutex` is one of the
-issues addressed in a subsequent section.
-
-### mnt->mnt_count ###
-
-`mnt_count` is a per-CPU reference counter on "`mount`" structures.
-Per-CPU here means that incrementing the count is cheap as it only
-uses CPU-local memory, but checking if the count is zero is expensive as
-it needs to check with every CPU. Taking a `mnt_count` reference
-prevents the mount structure from disappearing as the result of regular
-unmount operations, but does not prevent a "lazy" unmount. So holding
-`mnt_count` doesn't ensure that the mount remains in the namespace and,
-in particular, doesn't stabilize the link to the mounted-on dentry. It
-does, however, ensure that the `mount` data structure remains coherent,
-and it provides a reference to the root dentry of the mounted
-filesystem. So a reference through `->mnt_count` provides a stable
-reference to the mounted dentry, but not the mounted-on dentry.
-
-### mount_lock ###
-
-`mount_lock` is a global seqlock, a bit like `rename_lock`. It can be used to
-check if any change has been made to any mount points.
-
-While walking down the tree (away from the root) this lock is used when
-crossing a mount point to check that the crossing was safe. That is,
-the value in the seqlock is read, then the code finds the mount that
-is mounted on the current directory, if there is one, and increments
-the `mnt_count`. Finally the value in `mount_lock` is checked against
-the old value. If there is no change, then the crossing was safe. If there
-was a change, the `mnt_count` is decremented and the whole process is
-retried.
-
-When walking up the tree (towards the root) by following a ".." link,
-a little more care is needed. In this case the seqlock (which
-contains both a counter and a spinlock) is fully locked to prevent
-any changes to any mount points while stepping up. This locking is
-needed to stabilize the link to the mounted-on dentry, which the
-refcount on the mount itself doesn't ensure.
-
-### RCU ###
-
-Finally the global (but extremely lightweight) RCU read lock is held
-from time to time to ensure certain data structures don't get freed
-unexpectedly.
-
-In particular it is held while scanning chains in the dcache hash
-table, and the mount point hash table.
-
-Bringing it together with `struct nameidata`
---------------------------------------------
-
-[First edition Unix]: http://minnie.tuhs.org/cgi-bin/utree.pl?file=V1/u2.s
-
-Throughout the process of walking a path, the current status is stored
-in a `struct nameidata`, "namei" being the traditional name - dating
-all the way back to [First Edition Unix] - of the function that
-converts a "name" to an "inode". `struct nameidata` contains (among
-other fields):
-
-### `struct path path` ###
-
-A `path` contains a `struct vfsmount` (which is
-embedded in a `struct mount`) and a `struct dentry`. Together these
-record the current status of the walk. They start out referring to the
-starting point (the current working directory, the root directory, or some other
-directory identified by a file descriptor), and are updated on each
-step. A reference through `d_lockref` and `mnt_count` is always
-held.
-
-### `struct qstr last` ###
-
-This is a string together with a length (i.e. _not_ `nul` terminated)
-that is the "next" component in the pathname.
-
-### `int last_type` ###
-
-This is one of `LAST_NORM`, `LAST_ROOT`, `LAST_DOT`, `LAST_DOTDOT`, or
-`LAST_BIND`. The `last` field is only valid if the type is
-`LAST_NORM`. `LAST_BIND` is used when following a symlink and no
-components of the symlink have been processed yet. Others should be
-fairly self-explanatory.
-
-### `struct path root` ###
-
-This is used to hold a reference to the effective root of the
-filesystem. Often that reference won't be needed, so this field is
-only assigned the first time it is used, or when a non-standard root
-is requested. Keeping a reference in the `nameidata` ensures that
-only one root is in effect for the entire path walk, even if it races
-with a `chroot()` system call.
-
-The root is needed when either of two conditions holds: (1) either the
-pathname or a symbolic link starts with a "'/'", or (2) a "`..`"
-component is being handled, since "`..`" from the root must always stay
-at the root. The value used is usually the current root directory of
-the calling process. An alternate root can be provided as when
-`sysctl()` calls `file_open_root()`, and when NFSv4 or Btrfs call
-`mount_subtree()`. In each case a pathname is being looked up in a very
-specific part of the filesystem, and the lookup must not be allowed to
-escape that subtree. It works a bit like a local `chroot()`.
-
-Ignoring the handling of symbolic links, we can now describe the
-"`link_path_walk()`" function, which handles the lookup of everything
-except the final component as:
-
-> Given a path (`name`) and a nameidata structure (`nd`), check that the
-> current directory has execute permission and then advance `name`
-> over one component while updating `last_type` and `last`. If that
-> was the final component, then return, otherwise call
-> `walk_component()` and repeat from the top.
-
-`walk_component()` is even easier. If the component is `LAST_DOTS`,
-it calls `handle_dots()` which does the necessary locking as already
-described. If it finds a `LAST_NORM` component it first calls
-"`lookup_fast()`" which only looks in the dcache, but will ask the
-filesystem to revalidate the result if it is that sort of filesystem.
-If that doesn't get a good result, it calls "`lookup_slow()`" which
-takes the `i_mutex`, rechecks the cache, and then asks the filesystem
-to find a definitive answer. Each of these will call
-`follow_managed()` (as described below) to handle any mount points.
-
-In the absence of symbolic links, `walk_component()` creates a new
-`struct path` containing a counted reference to the new dentry and a
-reference to the new `vfsmount` which is only counted if it is
-different from the previous `vfsmount`. It then calls
-`path_to_nameidata()` to install the new `struct path` in the
-`struct nameidata` and drop the unneeded references.
-
-This "hand-over-hand" sequencing of getting a reference to the new
-dentry before dropping the reference to the previous dentry may
-seem obvious, but is worth pointing out so that we will recognize its
-analogue in the "RCU-walk" version.
-
-Handling the final component.
------------------------------
-
-`link_path_walk()` only walks as far as setting `nd->last` and
-`nd->last_type` to refer to the final component of the path. It does
-not call `walk_component()` that last time. Handling that final
-component remains for the caller to sort out. Those callers are
-`path_lookupat()`, `path_parentat()`, `path_mountpoint()` and
-`path_openat()` each of which handles the differing requirements of
-different system calls.
-
-`path_parentat()` is clearly the simplest - it just wraps a little bit
-of housekeeping around `link_path_walk()` and returns the parent
-directory and final component to the caller. The caller will be either
-aiming to create a name (via `filename_create()`) or remove or rename
-a name (in which case `user_path_parent()` is used). They will use
-`i_mutex` to exclude other changes while they validate and then
-perform their operation.
-
-`path_lookupat()` is nearly as simple - it is used when an existing
-object is wanted such as by `stat()` or `chmod()`. It essentially just
-calls `walk_component()` on the final component through a call to
-`lookup_last()`. `path_lookupat()` returns just the final dentry.
-
-`path_mountpoint()` handles the special case of unmounting which must
-not try to revalidate the mounted filesystem. It effectively
-contains, through a call to `mountpoint_last()`, an alternate
-implementation of `lookup_slow()` which skips that step. This is
-important when unmounting a filesystem that is inaccessible, such as
-one provided by a dead NFS server.
-
-Finally `path_openat()` is used for the `open()` system call; it
-contains, in support functions starting with "`do_last()`", all the
-complexity needed to handle the different subtleties of O_CREAT (with
-or without O_EXCL), final "`/`" characters, and trailing symbolic
-links. We will revisit this in the final part of this series, which
-focuses on those symbolic links. "`do_last()`" will sometimes, but
-not always, take `i_mutex`, depending on what it finds.
-
-Each of these, or the functions which call them, need to be alert to
-the possibility that the final component is not `LAST_NORM`. If the
-goal of the lookup is to create something, then any value for
-`last_type` other than `LAST_NORM` will result in an error. For
-example if `path_parentat()` reports `LAST_DOTDOT`, then the caller
-won't try to create that name. They also check for trailing slashes
-by testing `last.name[last.len]`. If there is any character beyond
-the final component, it must be a trailing slash.
-
-Revalidation and automounts
----------------------------
-
-Apart from symbolic links, there are only two parts of the "REF-walk"
-process not yet covered. One is the handling of stale cache entries
-and the other is automounts.
-
-On filesystems that require it, the lookup routines will call the
-`->d_revalidate()` dentry method to ensure that the cached information
-is current. This will often confirm validity or update a few details
-from a server. In some cases it may find that there has been change
-further up the path and that something that was thought to be valid
-previously isn't really. When this happens the lookup of the whole
-path is aborted and retried with the "`LOOKUP_REVAL`" flag set. This
-forces revalidation to be more thorough. We will see more details of
-this retry process in the next article.
-
-Automount points are locations in the filesystem where an attempt to
-lookup a name can trigger changes to how that lookup should be
-handled, in particular by mounting a filesystem there. These are
-covered in greater detail in autofs.txt in the Linux documentation
-tree, but a few notes specifically related to path lookup are in order
-here.
-
-The Linux VFS has a concept of "managed" dentries which is reflected
-in function names such as "`follow_managed()`". There are three
-potentially interesting things about these dentries corresponding
-to three different flags that might be set in `dentry->d_flags`:
-
-### `DCACHE_MANAGE_TRANSIT` ###
-
-If this flag has been set, then the filesystem has requested that the
-`d_manage()` dentry operation be called before handling any possible
-mount point. This can perform two particular services:
-
-It can block to avoid races. If an automount point is being
-unmounted, the `d_manage()` function will usually wait for that
-process to complete before letting the new lookup proceed and possibly
-trigger a new automount.
-
-It can selectively allow only some processes to transit through a
-mount point. When a server process is managing automounts, it may
-need to access a directory without triggering normal automount
-processing. That server process can identify itself to the `autofs`
-filesystem, which will then give it a special pass through
-`d_manage()` by returning `-EISDIR`.
-
-### `DCACHE_MOUNTED` ###
-
-This flag is set on every dentry that is mounted on. As Linux
-supports multiple filesystem namespaces, it is possible that the
-dentry may not be mounted on in *this* namespace, just in some
-other. So this flag is seen as a hint, not a promise.
-
-If this flag is set, and `d_manage()` didn't return `-EISDIR`,
-`lookup_mnt()` is called to examine the mount hash table (honoring the
-`mount_lock` described earlier) and possibly return a new `vfsmount`
-and a new `dentry` (both with counted references).
-
-### `DCACHE_NEED_AUTOMOUNT` ###
-
-If `d_manage()` allowed us to get this far, and `lookup_mnt()` didn't
-find a mount point, then this flag causes the `d_automount()` dentry
-operation to be called.
-
-The `d_automount()` operation can be arbitrarily complex and may
-communicate with server processes etc. but it should ultimately either
-report that there was an error, that there was nothing to mount, or
-should provide an updated `struct path` with new `dentry` and `vfsmount`.
-
-In the latter case, `finish_automount()` will be called to safely
-install the new mount point into the mount table.
-
-There is no new locking of import here and it is important that no
-locks (only counted references) are held over this processing due to
-the very real possibility of extended delays.
-This will become more important next time when we examine RCU-walk
-which is particularly sensitive to delays.
-
-RCU-walk - faster pathname lookup in Linux
-==========================================
-
-RCU-walk is another algorithm for performing pathname lookup in Linux.
-It is in many ways similar to REF-walk and the two share quite a bit
-of code. The significant difference in RCU-walk is how it allows for
-the possibility of concurrent access.
-
-We noted that REF-walk is complex because there are numerous details
-and special cases. RCU-walk reduces this complexity by simply
-refusing to handle a number of cases -- it instead falls back to
-REF-walk. The difficulty with RCU-walk comes from a different
-direction: unfamiliarity. The locking rules when depending on RCU are
-quite different from traditional locking, so we will spend a little extra
-time when we come to those.
-
-Clear demarcation of roles
---------------------------
-
-The easiest way to manage concurrency is to forcibly stop any other
-thread from changing the data structures that a given thread is
-looking at. In cases where no other thread would even think of
-changing the data and lots of different threads want to read at the
-same time, this can be very costly. Even when using locks that permit
-multiple concurrent readers, the simple act of updating the count of
-the number of current readers can impose an unwanted cost. So the
-goal when reading a shared data structure that no other process is
-changing is to avoid writing anything to memory at all. Take no
-locks, increment no counts, leave no footprints.
-
-The REF-walk mechanism already described certainly doesn't follow this
-principle, but then it is really designed to work when there may well
-be other threads modifying the data. RCU-walk, in contrast, is
-designed for the common situation where there are lots of frequent
-readers and only occasional writers. This may not be common in all
-parts of the filesystem tree, but in many parts it will be. For the
-other parts it is important that RCU-walk can quickly fall back to
-using REF-walk.
-
-Pathname lookup always starts in RCU-walk mode but only remains there
-as long as what it is looking for is in the cache and is stable. It
-dances lightly down the cached filesystem image, leaving no footprints
-and carefully watching where it is, to be sure it doesn't trip. If it
-notices that something has changed or is changing, or if something
-isn't in the cache, then it tries to stop gracefully and switch to
-REF-walk.
-
-This stopping requires getting a counted reference on the current
-`vfsmount` and `dentry`, and ensuring that these are still valid -
-that a path walk with REF-walk would have found the same entries.
-This is an invariant that RCU-walk must guarantee. It can only make
-decisions, such as selecting the next step, that are decisions which
-REF-walk could also have made if it were walking down the tree at the
-same time. If the graceful stop succeeds, the rest of the path is
-processed with the reliable, if slightly sluggish, REF-walk. If
-RCU-walk finds it cannot stop gracefully, it simply gives up and
-restarts from the top with REF-walk.
-
-This pattern of "try RCU-walk, if that fails try REF-walk" can be
-clearly seen in functions like `filename_lookup()`,
-`filename_parentat()`, `filename_mountpoint()`,
-`do_filp_open()`, and `do_file_open_root()`. These five
-correspond roughly to the four `path_`* functions we met earlier,
-each of which calls `link_path_walk()`. The `path_*` functions are
-called using different mode flags until a mode is found which works.
-They are first called with `LOOKUP_RCU` set to request "RCU-walk". If
-that fails with the error `ECHILD` they are called again with no
-special flag to request "REF-walk". If either of those report the
-error `ESTALE` a final attempt is made with `LOOKUP_REVAL` set (and no
-`LOOKUP_RCU`) to ensure that entries found in the cache are forcibly
-revalidated - normally entries are only revalidated if the filesystem
-determines that they are too old to trust.
-
-The `LOOKUP_RCU` attempt may drop that flag internally and switch to
-REF-walk, but will never then try to switch back to RCU-walk. Places
-that trip up RCU-walk are much more likely to be near the leaves and
-so it is very unlikely that there will be much, if any, benefit from
-switching back.
-
-RCU and seqlocks: fast and light
---------------------------------
-
-RCU is, unsurprisingly, critical to RCU-walk mode. The
-`rcu_read_lock()` is held for the entire time that RCU-walk is walking
-down a path. The particular guarantee it provides is that the key
-data structures - dentries, inodes, super_blocks, and mounts - will
-not be freed while the lock is held. They might be unlinked or
-invalidated in one way or another, but the memory will not be
-repurposed so values in various fields will still be meaningful. This
-is the only guarantee that RCU provides; everything else is done using
-seqlocks.
-
-As we saw above, REF-walk holds a counted reference to the current
-dentry and the current vfsmount, and does not release those references
-before taking references to the "next" dentry or vfsmount. It also
-sometimes takes the `d_lock` spinlock. These references and locks are
-taken to prevent certain changes from happening. RCU-walk must not
-take those references or locks and so cannot prevent such changes.
-Instead, it checks to see if a change has been made, and aborts or
-retries if it has.
-
-To preserve the invariant mentioned above (that RCU-walk may only make
-decisions that REF-walk could have made), it must make the checks at
-or near the same places that REF-walk holds the references. So, when
-REF-walk increments a reference count or takes a spinlock, RCU-walk
-samples the status of a seqlock using `read_seqcount_begin()` or a
-similar function. When REF-walk decrements the count or drops the
-lock, RCU-walk checks if the sampled status is still valid using
-`read_seqcount_retry()` or similar.
-
-However, there is a little bit more to seqlocks than that. If
-RCU-walk accesses two different fields in a seqlock-protected
-structure, or accesses the same field twice, there is no a priori
-guarantee of any consistency between those accesses. When consistency
-is needed - which it usually is - RCU-walk must take a copy and then
-use `read_seqcount_retry()` to validate that copy.
-
-`read_seqcount_retry()` not only checks the sequence number, but also
-imposes a memory barrier so that no memory-read instruction from
-*before* the call can be delayed until *after* the call, either by the
-CPU or by the compiler. A simple example of this can be seen in
-`slow_dentry_cmp()` which, for filesystems which do not use simple
-byte-wise name equality, calls into the filesystem to compare a name
-against a dentry. The length and name pointer are copied into local
-variables, then `read_seqcount_retry()` is called to confirm the two
-are consistent, and only then is `->d_compare()` called. When
-standard filename comparison is used, `dentry_cmp()` is called
-instead. Notably it does _not_ use `read_seqcount_retry()`, but
-instead has a large comment explaining why the consistency guarantee
-isn't necessary. A subsequent `read_seqcount_retry()` will be
-sufficient to catch any problem that could occur at this point.
-
-With that little refresher on seqlocks out of the way we can look at
-the bigger picture of how RCU-walk uses seqlocks.
-
-### `mount_lock` and `nd->m_seq` ###
-
-We already met the `mount_lock` seqlock when REF-walk used it to
-ensure that crossing a mount point is performed safely. RCU-walk uses
-it for that too, but for quite a bit more.
-
-Instead of taking a counted reference to each `vfsmount` as it
-descends the tree, RCU-walk samples the state of `mount_lock` at the
-start of the walk and stores this initial sequence number in the
-`struct nameidata` in the `m_seq` field. This one lock and one
-sequence number are used to validate all accesses to all `vfsmounts`,
-and all mount point crossings. As changes to the mount table are
-relatively rare, it is reasonable to fall back on REF-walk any time
-that any "mount" or "unmount" happens.
-
-`m_seq` is checked (using `read_seqretry()`) at the end of an RCU-walk
-sequence, whether switching to REF-walk for the rest of the path or
-when the end of the path is reached. It is also checked when stepping
-down over a mount point (in `__follow_mount_rcu()`) or up (in
-`follow_dotdot_rcu()`). If it is ever found to have changed, the
-whole RCU-walk sequence is aborted and the path is processed again by
-REF-walk.
-
-If RCU-walk finds that `mount_lock` hasn't changed then it can be sure
-that, had REF-walk taken counted references on each vfsmount, the
-results would have been the same. This ensures the invariant holds,
-at least for vfsmount structures.
-
-### `dentry->d_seq` and `nd->seq`. ###
-
-In place of taking a count or lock on `d_reflock`, RCU-walk samples
-the per-dentry `d_seq` seqlock, and stores the sequence number in the
-`seq` field of the nameidata structure, so `nd->seq` should always be
-the current sequence number of `nd->dentry`. This number needs to be
-revalidated after copying, and before using, the name, parent, or
-inode of the dentry.
-
-The handling of the name we have already looked at, and the parent is
-only accessed in `follow_dotdot_rcu()` which fairly trivially follows
-the required pattern, though it does so for three different cases.
-
-When not at a mount point, `d_parent` is followed and its `d_seq` is
-collected. When we are at a mount point, we instead follow the
-`mnt->mnt_mountpoint` link to get a new dentry and collect its
-`d_seq`. Then, after finally finding a `d_parent` to follow, we must
-check if we have landed on a mount point and, if so, must find that
-mount point and follow the `mnt->mnt_root` link. This would imply a
-somewhat unusual, but certainly possible, circumstance where the
-starting point of the path lookup was in part of the filesystem that
-was mounted on, and so not visible from the root.
-
-The inode pointer, stored in `->d_inode`, is a little more
-interesting. The inode will always need to be accessed at least
-twice, once to determine if it is NULL and once to verify access
-permissions. Symlink handling requires a validated inode pointer too.
-Rather than revalidating on each access, a copy is made on the first
-access and it is stored in the `inode` field of `nameidata` from where
-it can be safely accessed without further validation.
-
-`lookup_fast()` is the only lookup routine that is used in RCU-mode,
-`lookup_slow()` being too slow and requiring locks. It is in
-`lookup_fast()` that we find the important "hand over hand" tracking
-of the current dentry.
-
-The current `dentry` and current `seq` number are passed to
-`__d_lookup_rcu()` which, on success, returns a new `dentry` and a
-new `seq` number. `lookup_fast()` then copies the inode pointer and
-revalidates the new `seq` number. It then validates the old `dentry`
-with the old `seq` number one last time and only then continues. This
-process of getting the `seq` number of the new dentry and then
-checking the `seq` number of the old exactly mirrors the process of
-getting a counted reference to the new dentry before dropping that for
-the old dentry which we saw in REF-walk.
-
-### No `inode->i_mutex` or even `rename_lock` ###
-
-A mutex is a fairly heavyweight lock that can only be taken when it is
-permissible to sleep. As `rcu_read_lock()` forbids sleeping,
-`inode->i_mutex` plays no role in RCU-walk. If some other thread does
-take `i_mutex` and modifies the directory in a way that RCU-walk needs
-to notice, the result will be either that RCU-walk fails to find the
-dentry that it is looking for, or it will find a dentry which
-`read_seqretry()` won't validate. In either case it will drop down to
-REF-walk mode which can take whatever locks are needed.
-
-Though `rename_lock` could be used by RCU-walk as it doesn't require
-any sleeping, RCU-walk doesn't bother. REF-walk uses `rename_lock` to
-protect against the possibility of hash chains in the dcache changing
-while they are being searched. This can result in failing to find
-something that actually is there. When RCU-walk fails to find
-something in the dentry cache, whether it is really there or not, it
-already drops down to REF-walk and tries again with appropriate
-locking. This neatly handles all cases, so adding extra checks on
-rename_lock would bring no significant value.
-
-`unlazy walk()` and `complete_walk()`
--------------------------------------
-
-That "dropping down to REF-walk" typically involves a call to
-`unlazy_walk()`, so named because "RCU-walk" is also sometimes
-referred to as "lazy walk". `unlazy_walk()` is called when
-following the path down to the current vfsmount/dentry pair seems to
-have proceeded successfully, but the next step is problematic. This
-can happen if the next name cannot be found in the dcache, if
-permission checking or name revalidation couldn't be achieved while
-the `rcu_read_lock()` is held (which forbids sleeping), if an
-automount point is found, or in a couple of cases involving symlinks.
-It is also called from `complete_walk()` when the lookup has reached
-the final component, or the very end of the path, depending on which
-particular flavor of lookup is used.
-
-Other reasons for dropping out of RCU-walk that do not trigger a call
-to `unlazy_walk()` are when some inconsistency is found that cannot be
-handled immediately, such as `mount_lock` or one of the `d_seq`
-seqlocks reporting a change. In these cases the relevant function
-will return `-ECHILD` which will percolate up until it triggers a new
-attempt from the top using REF-walk.
-
-For those cases where `unlazy_walk()` is an option, it essentially
-takes a reference on each of the pointers that it holds (vfsmount,
-dentry, and possibly some symbolic links) and then verifies that the
-relevant seqlocks have not been changed. If there have been changes,
-it, too, aborts with `-ECHILD`, otherwise the transition to REF-walk
-has been a success and the lookup process continues.
-
-Taking a reference on those pointers is not quite as simple as just
-incrementing a counter. That works to take a second reference if you
-already have one (often indirectly through another object), but it
-isn't sufficient if you don't actually have a counted reference at
-all. For `dentry->d_lockref`, it is safe to increment the reference
-counter to get a reference unless it has been explicitly marked as
-"dead" which involves setting the counter to `-128`.
-`lockref_get_not_dead()` achieves this.
-
-For `mnt->mnt_count` it is safe to take a reference as long as
-`mount_lock` is then used to validate the reference. If that
-validation fails, it may *not* be safe to just drop that reference in
-the standard way of calling `mnt_put()` - an unmount may have
-progressed too far. So the code in `legitimize_mnt()`, when it
-finds that the reference it got might not be safe, checks the
-`MNT_SYNC_UMOUNT` flag to determine if a simple `mnt_put()` is
-correct, or if it should just decrement the count and pretend none of
-this ever happened.
-
-Taking care in filesystems
----------------------------
-
-RCU-walk depends almost entirely on cached information and often will
-not call into the filesystem at all. However there are two places,
-besides the already-mentioned component-name comparison, where the
-file system might be included in RCU-walk, and it must know to be
-careful.
-
-If the filesystem has non-standard permission-checking requirements -
-such as a networked filesystem which may need to check with the server
-- the `i_op->permission` interface might be called during RCU-walk.
-In this case an extra "`MAY_NOT_BLOCK`" flag is passed so that it
-knows not to sleep, but to return `-ECHILD` if it cannot complete
-promptly. `i_op->permission` is given the inode pointer, not the
-dentry, so it doesn't need to worry about further consistency checks.
-However if it accesses any other filesystem data structures, it must
-ensure they are safe to be accessed with only the `rcu_read_lock()`
-held. This typically means they must be freed using `kfree_rcu()` or
-similar.
-
-[`READ_ONCE()`]: https://lwn.net/Articles/624126/
-
-If the filesystem may need to revalidate dcache entries, then
-`d_op->d_revalidate` may be called in RCU-walk too. This interface
-*is* passed the dentry but does not have access to the `inode` or the
-`seq` number from the `nameidata`, so it needs to be extra careful
-when accessing fields in the dentry. This "extra care" typically
-involves using [`READ_ONCE()`] to access fields, and verifying the
-result is not NULL before using it. This pattern can be seen in
-`nfs_lookup_revalidate()`.
-
-A pair of patterns
-------------------
-
-In various places in the details of REF-walk and RCU-walk, and also in
-the big picture, there are a couple of related patterns that are worth
-being aware of.
-
-The first is "try quickly and check, if that fails try slowly". We
-can see that in the high-level approach of first trying RCU-walk and
-then trying REF-walk, and in places where `unlazy_walk()` is used to
-switch to REF-walk for the rest of the path. We also saw it earlier
-in `dget_parent()` when following a "`..`" link. It tries a quick way
-to get a reference, then falls back to taking locks if needed.
-
-The second pattern is "try quickly and check, if that fails try
-again - repeatedly". This is seen with the use of `rename_lock` and
-`mount_lock` in REF-walk. RCU-walk doesn't make use of this pattern -
-if anything goes wrong it is much safer to just abort and try a more
-sedate approach.
-
-The emphasis here is "try quickly and check". It should probably be
-"try quickly _and carefully,_ then check". The fact that checking is
-needed is a reminder that the system is dynamic and only a limited
-number of things are safe at all. The most likely cause of errors in
-this whole process is assuming something is safe when in reality it
-isn't. Careful consideration of what exactly guarantees the safety of
-each access is sometimes necessary.
-
-A walk among the symlinks
-=========================
-
-There are several basic issues that we will examine to understand the
-handling of symbolic links: the symlink stack, together with cache
-lifetimes, will help us understand the overall recursive handling of
-symlinks and lead to the special care needed for the final component.
-Then a consideration of access-time updates and summary of the various
-flags controlling lookup will finish the story.
-
-The symlink stack
------------------
-
-There are only two sorts of filesystem objects that can usefully
-appear in a path prior to the final component: directories and symlinks.
-Handling directories is quite straightforward: the new directory
-simply becomes the starting point at which to interpret the next
-component on the path. Handling symbolic links requires a bit more
-work.
-
-Conceptually, symbolic links could be handled by editing the path. If
-a component name refers to a symbolic link, then that component is
-replaced by the body of the link and, if that body starts with a '/',
-then all preceding parts of the path are discarded. This is what the
-"`readlink -f`" command does, though it also edits out "`.`" and
-"`..`" components.
-
-Directly editing the path string is not really necessary when looking
-up a path, and discarding early components is pointless as they aren't
-looked at anyway. Keeping track of all remaining components is
-important, but they can of course be kept separately; there is no need
-to concatenate them. As one symlink may easily refer to another,
-which in turn can refer to a third, we may need to keep the remaining
-components of several paths, each to be processed when the preceding
-ones are completed. These path remnants are kept on a stack of
-limited size.
-
-There are two reasons for placing limits on how many symlinks can
-occur in a single path lookup. The most obvious is to avoid loops.
-If a symlink referred to itself either directly or through
-intermediaries, then following the symlink can never complete
-successfully - the error `ELOOP` must be returned. Loops can be
-detected without imposing limits, but limits are the simplest solution
-and, given the second reason for restriction, quite sufficient.
-
-[outlined recently]: http://thread.gmane.org/gmane.linux.kernel/1934390/focus=1934550
-
-The second reason was [outlined recently] by Linus:
-
-> Because it's a latency and DoS issue too. We need to react well to
-> true loops, but also to "very deep" non-loops. It's not about memory
-> use, it's about users triggering unreasonable CPU resources.
-
-Linux imposes a limit on the length of any pathname: `PATH_MAX`, which
-is 4096. There are a number of reasons for this limit; not letting the
-kernel spend too much time on just one path is one of them. With
-symbolic links you can effectively generate much longer paths so some
-sort of limit is needed for the same reason. Linux imposes a limit of
-at most 40 symlinks in any one path lookup. It previously imposed a
-further limit of eight on the maximum depth of recursion, but that was
-raised to 40 when a separate stack was implemented, so there is now
-just the one limit.
-
-The `nameidata` structure that we met in an earlier article contains a
-small stack that can be used to store the remaining part of up to two
-symlinks. In many cases this will be sufficient. If it isn't, a
-separate stack is allocated with room for 40 symlinks. Pathname
-lookup will never exceed that stack as, once the 40th symlink is
-detected, an error is returned.
-
-It might seem that the name remnants are all that needs to be stored on
-this stack, but we need a bit more. To see that, we need to move on to
-cache lifetimes.
-
-Storage and lifetime of cached symlinks
----------------------------------------
-
-Like other filesystem resources, such as inodes and directory
-entries, symlinks are cached by Linux to avoid repeated costly access
-to external storage. It is particularly important for RCU-walk to be
-able to find and temporarily hold onto these cached entries, so that
-it doesn't need to drop down into REF-walk.
-
-[object-oriented design pattern]: https://lwn.net/Articles/446317/
-
-While each filesystem is free to make its own choice, symlinks are
-typically stored in one of two places. Short symlinks are often
-stored directly in the inode. When a filesystem allocates a `struct
-inode` it typically allocates extra space to store private data (a
-common [object-oriented design pattern] in the kernel). This will
-sometimes include space for a symlink. The other common location is
-in the page cache, which normally stores the content of files. The
-pathname in a symlink can be seen as the content of that symlink and
-can easily be stored in the page cache just like file content.
-
-When neither of these is suitable, the next most likely scenario is
-that the filesystem will allocate some temporary memory and copy or
-construct the symlink content into that memory whenever it is needed.
-
-When the symlink is stored in the inode, it has the same lifetime as
-the inode which, itself, is protected by RCU or by a counted reference
-on the dentry. This means that the mechanisms that pathname lookup
-uses to access the dcache and icache (inode cache) safely are quite
-sufficient for accessing some cached symlinks safely. In these cases,
-the `i_link` pointer in the inode is set to point to wherever the
-symlink is stored and it can be accessed directly whenever needed.
-
-When the symlink is stored in the page cache or elsewhere, the
-situation is not so straightforward. A reference on a dentry or even
-on an inode does not imply any reference on cached pages of that
-inode, and even an `rcu_read_lock()` is not sufficient to ensure that
-a page will not disappear. So for these symlinks the pathname lookup
-code needs to ask the filesystem to provide a stable reference and,
-significantly, needs to release that reference when it is finished
-with it.
-
-Taking a reference to a cache page is often possible even in RCU-walk
-mode. It does require making changes to memory, which is best avoided,
-but that isn't necessarily a big cost and it is better than dropping
-out of RCU-walk mode completely. Even filesystems that allocate
-space to copy the symlink into can use `GFP_ATOMIC` to often successfully
-allocate memory without the need to drop out of RCU-walk. If a
-filesystem cannot successfully get a reference in RCU-walk mode, it
-must return `-ECHILD` and `unlazy_walk()` will be called to return to
-REF-walk mode in which the filesystem is allowed to sleep.
-
-The place for all this to happen is the `i_op->follow_link()` inode
-method. In the present mainline code this is never actually called in
-RCU-walk mode as the rewrite is not quite complete. It is likely that
-in a future release this method will be passed an `inode` pointer when
-called in RCU-walk mode so it both (1) knows to be careful, and (2) has the
-validated pointer. Much like the `i_op->permission()` method we
-looked at previously, `->follow_link()` would need to be careful that
-all the data structures it references are safe to be accessed while
-holding no counted reference, only the RCU lock. Though getting a
-reference with `->follow_link()` is not yet done in RCU-walk mode, the
-code is ready to release the reference when that does happen.
-
-This need to drop the reference to a symlink adds significant
-complexity. It requires a reference to the inode so that the
-`i_op->put_link()` inode operation can be called. In REF-walk, that
-reference is kept implicitly through a reference to the dentry, so
-keeping the `struct path` of the symlink is easiest. For RCU-walk,
-the pointer to the inode is kept separately. To allow switching from
-RCU-walk back to REF-walk in the middle of processing nested symlinks
-we also need the seq number for the dentry so we can confirm that
-switching back was safe.
-
-Finally, when providing a reference to a symlink, the filesystem also
-provides an opaque "cookie" that must be passed to `->put_link()` so that it
-knows what to free. This might be the allocated memory area, or a
-pointer to the `struct page` in the page cache, or something else
-completely. Only the filesystem knows what it is.
-
-In order for the reference to each symlink to be dropped when the walk completes,
-whether in RCU-walk or REF-walk, the symlink stack needs to contain,
-along with the path remnants:
-
-- the `struct path` to provide a reference to the inode in REF-walk
-- the `struct inode *` to provide a reference to the inode in RCU-walk
-- the `seq` to allow the path to be safely switched from RCU-walk to REF-walk
-- the `cookie` that tells `->put_path()` what to put.
-
-This means that each entry in the symlink stack needs to hold five
-pointers and an integer instead of just one pointer (the path
-remnant). On a 64-bit system, this is about 40 bytes per entry;
-with 40 entries it adds up to 1600 bytes total, which is less than
-half a page. So it might seem like a lot, but is by no means
-excessive.
-
-Note that, in a given stack frame, the path remnant (`name`) is not
-part of the symlink that the other fields refer to. It is the remnant
-to be followed once that symlink has been fully parsed.
-
-Following the symlink
----------------------
-
-The main loop in `link_path_walk()` iterates seamlessly over all
-components in the path and all of the non-final symlinks. As symlinks
-are processed, the `name` pointer is adjusted to point to a new
-symlink, or is restored from the stack, so that much of the loop
-doesn't need to notice. Getting this `name` variable on and off the
-stack is very straightforward; pushing and popping the references is
-a little more complex.
-
-When a symlink is found, `walk_component()` returns the value `1`
-(`0` is returned for any other sort of success, and a negative number
-is, as usual, an error indicator). This causes `get_link()` to be
-called; it then gets the link from the filesystem. Providing that
-operation is successful, the old path `name` is placed on the stack,
-and the new value is used as the `name` for a while. When the end of
-the path is found (i.e. `*name` is `'\0'`) the old `name` is restored
-off the stack and path walking continues.
-
-Pushing and popping the reference pointers (inode, cookie, etc.) is more
-complex in part because of the desire to handle tail recursion. When
-the last component of a symlink itself points to a symlink, we
-want to pop the symlink-just-completed off the stack before pushing
-the symlink-just-found to avoid leaving empty path remnants that would
-just get in the way.
-
-It is most convenient to push the new symlink references onto the
-stack in `walk_component()` immediately when the symlink is found;
-`walk_component()` is also the last piece of code that needs to look at the
-old symlink as it walks that last component. So it is quite
-convenient for `walk_component()` to release the old symlink and pop
-the references just before pushing the reference information for the
-new symlink. It is guided in this by two flags; `WALK_GET`, which
-gives it permission to follow a symlink if it finds one, and
-`WALK_PUT`, which tells it to release the current symlink after it has been
-followed. `WALK_PUT` is tested first, leading to a call to
-`put_link()`. `WALK_GET` is tested subsequently (by
-`should_follow_link()`) leading to a call to `pick_link()` which sets
-up the stack frame.
-
-### Symlinks with no final component ###
-
-A pair of special-case symlinks deserve a little further explanation.
-Both result in a new `struct path` (with mount and dentry) being set
-up in the `nameidata`, and result in `get_link()` returning `NULL`.
-
-The more obvious case is a symlink to "`/`". All symlinks starting
-with "`/`" are detected in `get_link()` which resets the `nameidata`
-to point to the effective filesystem root. If the symlink only
-contains "`/`" then there is nothing more to do, no components at all,
-so `NULL` is returned to indicate that the symlink can be released and
-the stack frame discarded.
-
-The other case involves things in `/proc` that look like symlinks but
-aren't really.
-
-> $ ls -l /proc/self/fd/1
-> lrwx------ 1 neilb neilb 64 Jun 13 10:19 /proc/self/fd/1 -> /dev/pts/4
-
-Every open file descriptor in any process is represented in `/proc` by
-something that looks like a symlink. It is really a reference to the
-target file, not just the name of it. When you `readlink` these
-objects you get a name that might refer to the same file - unless it
-has been unlinked or mounted over. When `walk_component()` follows
-one of these, the `->follow_link()` method in "procfs" doesn't return
-a string name, but instead calls `nd_jump_link()` which updates the
-`nameidata` in place to point to that target. `->follow_link()` then
-returns `NULL`. Again there is no final component and `get_link()`
-reports this by leaving the `last_type` field of `nameidata` as
-`LAST_BIND`.
-
-Following the symlink in the final component
---------------------------------------------
-
-All this leads to `link_path_walk()` walking down every component, and
-following all symbolic links it finds, until it reaches the final
-component. This is just returned in the `last` field of `nameidata`.
-For some callers, this is all they need; they want to create that
-`last` name if it doesn't exist or give an error if it does. Other
-callers will want to follow a symlink if one is found, and possibly
-apply special handling to the last component of that symlink, rather
-than just the last component of the original file name. These callers
-potentially need to call `link_path_walk()` again and again on
-successive symlinks until one is found that doesn't point to another
-symlink.
-
-This case is handled by the relevant caller of `link_path_walk()`, such as
-`path_lookupat()` using a loop that calls `link_path_walk()`, and then
-handles the final component. If the final component is a symlink
-that needs to be followed, then `trailing_symlink()` is called to set
-things up properly and the loop repeats, calling `link_path_walk()`
-again. This could loop as many as 40 times if the last component of
-each symlink is another symlink.
-
-The various functions that examine the final component and possibly
-report that it is a symlink are `lookup_last()`, `mountpoint_last()`
-and `do_last()`, each of which use the same convention as
-`walk_component()` of returning `1` if a symlink was found that needs
-to be followed.
-
-Of these, `do_last()` is the most interesting as it is used for
-opening a file. Part of `do_last()` runs with `i_mutex` held and this
-part is in a separate function: `lookup_open()`.
-
-Explaining `do_last()` completely is beyond the scope of this article,
-but a few highlights should help those interested in exploring the
-code.
-
-1. Rather than just finding the target file, `do_last()` needs to open
- it. If the file was found in the dcache, then `vfs_open()` is used for
- this. If not, then `lookup_open()` will either call `atomic_open()` (if
- the filesystem provides it) to combine the final lookup with the open, or
- will perform the separate `lookup_real()` and `vfs_create()` steps
- directly. In the later case the actual "open" of this newly found or
- created file will be performed by `vfs_open()`, just as if the name
- were found in the dcache.
-
-2. `vfs_open()` can fail with `-EOPENSTALE` if the cached information
- wasn't quite current enough. Rather than restarting the lookup from
- the top with `LOOKUP_REVAL` set, `lookup_open()` is called instead,
- giving the filesystem a chance to resolve small inconsistencies.
- If that doesn't work, only then is the lookup restarted from the top.
-
-3. An open with O_CREAT **does** follow a symlink in the final component,
- unlike other creation system calls (like `mkdir`). So the sequence:
-
- > ln -s bar /tmp/foo
- > echo hello > /tmp/foo
-
- will create a file called `/tmp/bar`. This is not permitted if
- `O_EXCL` is set but otherwise is handled for an O_CREAT open much
- like for a non-creating open: `should_follow_link()` returns `1`, and
- so does `do_last()` so that `trailing_symlink()` gets called and the
- open process continues on the symlink that was found.
-
-Updating the access time
-------------------------
-
-We previously said of RCU-walk that it would "take no locks, increment
-no counts, leave no footprints." We have since seen that some
-"footprints" can be needed when handling symlinks as a counted
-reference (or even a memory allocation) may be needed. But these
-footprints are best kept to a minimum.
-
-One other place where walking down a symlink can involve leaving
-footprints in a way that doesn't affect directories is in updating access times.
-In Unix (and Linux) every filesystem object has a "last accessed
-time", or "`atime`". Passing through a directory to access a file
-within is not considered to be an access for the purposes of
-`atime`; only listing the contents of a directory can update its `atime`.
-Symlinks are different it seems. Both reading a symlink (with `readlink()`)
-and looking up a symlink on the way to some other destination can
-update the atime on that symlink.
-
-[clearest statement]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_08
-
-It is not clear why this is the case; POSIX has little to say on the
-subject. The [clearest statement] is that, if a particular implementation
-updates a timestamp in a place not specified by POSIX, this must be
-documented "except that any changes caused by pathname resolution need
-not be documented". This seems to imply that POSIX doesn't really
-care about access-time updates during pathname lookup.
-
-[Linux 1.3.87]: https://git.kernel.org/cgit/linux/kernel/git/history/history.git/diff/fs/ext2/symlink.c?id=f806c6db77b8eaa6e00dcfb6b567706feae8dbb8
-
-An examination of history shows that prior to [Linux 1.3.87], the ext2
-filesystem, at least, didn't update atime when following a link.
-Unfortunately we have no record of why that behavior was changed.
-
-In any case, access time must now be updated and that operation can be
-quite complex. Trying to stay in RCU-walk while doing it is best
-avoided. Fortunately it is often permitted to skip the `atime`
-update. Because `atime` updates cause performance problems in various
-areas, Linux supports the `relatime` mount option, which generally
-limits the updates of `atime` to once per day on files that aren't
-being changed (and symlinks never change once created). Even without
-`relatime`, many filesystems record `atime` with a one-second
-granularity, so only one update per second is required.
-
-It is easy to test if an `atime` update is needed while in RCU-walk
-mode and, if it isn't, the update can be skipped and RCU-walk mode
-continues. Only when an `atime` update is actually required does the
-path walk drop down to REF-walk. All of this is handled in the
-`get_link()` function.
-
-A few flags
------------
-
-A suitable way to wrap up this tour of pathname walking is to list
-the various flags that can be stored in the `nameidata` to guide the
-lookup process. Many of these are only meaningful on the final
-component, others reflect the current state of the pathname lookup.
-And then there is `LOOKUP_EMPTY`, which doesn't fit conceptually with
-the others. If this is not set, an empty pathname causes an error
-very early on. If it is set, empty pathnames are not considered to be
-an error.
-
-### Global state flags ###
-
-We have already met two global state flags: `LOOKUP_RCU` and
-`LOOKUP_REVAL`. These select between one of three overall approaches
-to lookup: RCU-walk, REF-walk, and REF-walk with forced revalidation.
-
-`LOOKUP_PARENT` indicates that the final component hasn't been reached
-yet. This is primarily used to tell the audit subsystem the full
-context of a particular access being audited.
-
-`LOOKUP_ROOT` indicates that the `root` field in the `nameidata` was
-provided by the caller, so it shouldn't be released when it is no
-longer needed.
-
-`LOOKUP_JUMPED` means that the current dentry was chosen not because
-it had the right name but for some other reason. This happens when
-following "`..`", following a symlink to `/`, crossing a mount point
-or accessing a "`/proc/$PID/fd/$FD`" symlink. In this case the
-filesystem has not been asked to revalidate the name (with
-`d_revalidate()`). In such cases the inode may still need to be
-revalidated, so `d_op->d_weak_revalidate()` is called if
-`LOOKUP_JUMPED` is set when the look completes - which may be at the
-final component or, when creating, unlinking, or renaming, at the penultimate component.
-
-### Final-component flags ###
-
-Some of these flags are only set when the final component is being
-considered. Others are only checked for when considering that final
-component.
-
-`LOOKUP_AUTOMOUNT` ensures that, if the final component is an automount
-point, then the mount is triggered. Some operations would trigger it
-anyway, but operations like `stat()` deliberately don't. `statfs()`
-needs to trigger the mount but otherwise behaves a lot like `stat()`, so
-it sets `LOOKUP_AUTOMOUNT`, as does "`quotactl()`" and the handling of
-"`mount --bind`".
-
-`LOOKUP_FOLLOW` has a similar function to `LOOKUP_AUTOMOUNT` but for
-symlinks. Some system calls set or clear it implicitly, while
-others have API flags such as `AT_SYMLINK_FOLLOW` and
-`UMOUNT_NOFOLLOW` to control it. Its effect is similar to
-`WALK_GET` that we already met, but it is used in a different way.
-
-`LOOKUP_DIRECTORY` insists that the final component is a directory.
-Various callers set this and it is also set when the final component
-is found to be followed by a slash.
-
-Finally `LOOKUP_OPEN`, `LOOKUP_CREATE`, `LOOKUP_EXCL`, and
-`LOOKUP_RENAME_TARGET` are not used directly by the VFS but are made
-available to the filesystem and particularly the `->d_revalidate()`
-method. A filesystem can choose not to bother revalidating too hard
-if it knows that it will be asked to open or create the file soon.
-These flags were previously useful for `->lookup()` too but with the
-introduction of `->atomic_open()` they are less relevant there.
-
-End of the road
----------------
-
-Despite its complexity, all this pathname lookup code appears to be
-in good shape - various parts are certainly easier to understand now
-than even a couple of releases ago. But that doesn't mean it is
-"finished". As already mentioned, RCU-walk currently only follows
-symlinks that are stored in the inode so, while it handles many ext4
-symlinks, it doesn't help with NFS, XFS, or Btrfs. That support
-is not likely to be long delayed.
diff --git a/Documentation/filesystems/path-lookup.rst b/Documentation/filesystems/path-lookup.rst
new file mode 100644
index 0000000..434a07b
--- /dev/null
+++ b/Documentation/filesystems/path-lookup.rst
@@ -0,0 +1,1361 @@
+===============
+Pathname lookup
+===============
+
+This write-up is based on three articles published at lwn.net:
+
+- <https://lwn.net/Articles/649115/> Pathname lookup in Linux
+- <https://lwn.net/Articles/649729/> RCU-walk: faster pathname lookup in Linux
+- <https://lwn.net/Articles/650786/> A walk among the symlinks
+
+Written by Neil Brown with help from Al Viro and Jon Corbet.
+It has subsequently been updated to reflect changes in the kernel
+including:
+
+- per-directory parallel name lookup.
+
+Introduction to pathname lookup
+===============================
+
+The most obvious aspect of pathname lookup, which very little
+exploration is needed to discover, is that it is complex. There are
+many rules, special cases, and implementation alternatives that all
+combine to confuse the unwary reader. Computer science has long been
+acquainted with such complexity and has tools to help manage it. One
+tool that we will make extensive use of is "divide and conquer". For
+the early parts of the analysis we will divide off symlinks - leaving
+them until the final part. Well before we get to symlinks we have
+another major division based on the VFS's approach to locking which
+will allow us to review "REF-walk" and "RCU-walk" separately. But we
+are getting ahead of ourselves. There are some important low level
+distinctions we need to clarify first.
+
+There are two sorts of ...
+--------------------------
+
+.. _openat: http://man7.org/linux/man-pages/man2/openat.2.html
+
+Pathnames (sometimes "file names"), used to identify objects in the
+filesystem, will be familiar to most readers. They contain two sorts
+of elements: "slashes" that are sequences of one or more "``/``"
+characters, and "components" that are sequences of one or more
+non-"``/``" characters. These form two kinds of paths. Those that
+start with slashes are "absolute" and start from the filesystem root.
+The others are "relative" and start from the current directory, or
+from some other location specified by a file descriptor given to a
+"``XXXat``" system call such as `openat() <openat_>`_.
+
+.. _execveat: http://man7.org/linux/man-pages/man2/execveat.2.html
+
+It is tempting to describe the second kind as starting with a
+component, but that isn't always accurate: a pathname can lack both
+slashes and components, it can be empty, in other words. This is
+generally forbidden in POSIX, but some of those "xxx``at``" system calls
+in Linux permit it when the ``AT_EMPTY_PATH`` flag is given. For
+example, if you have an open file descriptor on an executable file you
+can execute it by calling `execveat() <execveat_>`_ passing
+the file descriptor, an empty path, and the ``AT_EMPTY_PATH`` flag.
+
+These paths can be divided into two sections: the final component and
+everything else. The "everything else" is the easy bit. In all cases
+it must identify a directory that already exists, otherwise an error
+such as ``ENOENT`` or ``ENOTDIR`` will be reported.
+
+The final component is not so simple. Not only do different system
+calls interpret it quite differently (e.g. some create it, some do
+not), but it might not even exist: neither the empty pathname nor the
+pathname that is just slashes have a final component. If it does
+exist, it could be "``.``" or "``..``" which are handled quite differently
+from other components.
+
+.. _POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_12
+
+If a pathname ends with a slash, such as "``/tmp/foo/``" it might be
+tempting to consider that to have an empty final component. In many
+ways that would lead to correct results, but not always. In
+particular, ``mkdir()`` and ``rmdir()`` each create or remove a directory named
+by the final component, and they are required to work with pathnames
+ending in "``/``". According to POSIX_
+
+ A pathname that contains at least one non- <slash> character and
+ that ends with one or more trailing <slash> characters shall not
+ be resolved successfully unless the last pathname component before
+ the trailing <slash> characters names an existing directory or a
+ directory entry that is to be created for a directory immediately
+ after the pathname is resolved.
+
+The Linux pathname walking code (mostly in ``fs/namei.c``) deals with
+all of these issues: breaking the path into components, handling the
+"everything else" quite separately from the final component, and
+checking that the trailing slash is not used where it isn't
+permitted. It also addresses the important issue of concurrent
+access.
+
+While one process is looking up a pathname, another might be making
+changes that affect that lookup. One fairly extreme case is that if
+"a/b" were renamed to "a/c/b" while another process were looking up
+"a/b/..", that process might successfully resolve on "a/c".
+Most races are much more subtle, and a big part of the task of
+pathname lookup is to prevent them from having damaging effects. Many
+of the possible races are seen most clearly in the context of the
+"dcache" and an understanding of that is central to understanding
+pathname lookup.
+
+More than just a cache
+----------------------
+
+The "dcache" caches information about names in each filesystem to
+make them quickly available for lookup. Each entry (known as a
+"dentry") contains three significant fields: a component name, a
+pointer to a parent dentry, and a pointer to the "inode" which
+contains further information about the object in that parent with
+the given name. The inode pointer can be ``NULL`` indicating that the
+name doesn't exist in the parent. While there can be linkage in the
+dentry of a directory to the dentries of the children, that linkage is
+not used for pathname lookup, and so will not be considered here.
+
+The dcache has a number of uses apart from accelerating lookup. One
+that will be particularly relevant is that it is closely integrated
+with the mount table that records which filesystem is mounted where.
+What the mount table actually stores is which dentry is mounted on top
+of which other dentry.
+
+When considering the dcache, we have another of our "two types"
+distinctions: there are two types of filesystems.
+
+Some filesystems ensure that the information in the dcache is always
+completely accurate (though not necessarily complete). This can allow
+the VFS to determine if a particular file does or doesn't exist
+without checking with the filesystem, and means that the VFS can
+protect the filesystem against certain races and other problems.
+These are typically "local" filesystems such as ext3, XFS, and Btrfs.
+
+Other filesystems don't provide that guarantee because they cannot.
+These are typically filesystems that are shared across a network,
+whether remote filesystems like NFS and 9P, or cluster filesystems
+like ocfs2 or cephfs. These filesystems allow the VFS to revalidate
+cached information, and must provide their own protection against
+awkward races. The VFS can detect these filesystems by the
+``DCACHE_OP_REVALIDATE`` flag being set in the dentry.
+
+REF-walk: simple concurrency management with refcounts and spinlocks
+--------------------------------------------------------------------
+
+With all of those divisions carefully classified, we can now start
+looking at the actual process of walking along a path. In particular
+we will start with the handling of the "everything else" part of a
+pathname, and focus on the "REF-walk" approach to concurrency
+management. This code is found in the ``link_path_walk()`` function, if
+you ignore all the places that only run when "``LOOKUP_RCU``"
+(indicating the use of RCU-walk) is set.
+
+.. _Meet the Lockers: https://lwn.net/Articles/453685/
+
+REF-walk is fairly heavy-handed with locks and reference counts. Not
+as heavy-handed as in the old "big kernel lock" days, but certainly not
+afraid of taking a lock when one is needed. It uses a variety of
+different concurrency controls. A background understanding of the
+various primitives is assumed, or can be gleaned from elsewhere such
+as in `Meet the Lockers`_.
+
+The locking mechanisms used by REF-walk include:
+
+dentry->d_lockref
+~~~~~~~~~~~~~~~~~
+
+This uses the lockref primitive to provide both a spinlock and a
+reference count. The special-sauce of this primitive is that the
+conceptual sequence "lock; inc_ref; unlock;" can often be performed
+with a single atomic memory operation.
+
+Holding a reference on a dentry ensures that the dentry won't suddenly
+be freed and used for something else, so the values in various fields
+will behave as expected. It also protects the ``->d_inode`` reference
+to the inode to some extent.
+
+The association between a dentry and its inode is fairly permanent.
+For example, when a file is renamed, the dentry and inode move
+together to the new location. When a file is created the dentry will
+initially be negative (i.e. ``d_inode`` is ``NULL``), and will be assigned
+to the new inode as part of the act of creation.
+
+When a file is deleted, this can be reflected in the cache either by
+setting ``d_inode`` to ``NULL``, or by removing it from the hash table
+(described shortly) used to look up the name in the parent directory.
+If the dentry is still in use the second option is used as it is
+perfectly legal to keep using an open file after it has been deleted
+and having the dentry around helps. If the dentry is not otherwise in
+use (i.e. if the refcount in ``d_lockref`` is one), only then will
+``d_inode`` be set to ``NULL``. Doing it this way is more efficient for a
+very common case.
+
+So as long as a counted reference is held to a dentry, a non-``NULL`` ``->d_inode``
+value will never be changed.
+
+dentry->d_lock
+~~~~~~~~~~~~~~
+
+``d_lock`` is a synonym for the spinlock that is part of ``d_lockref`` above.
+For our purposes, holding this lock protects against the dentry being
+renamed or unlinked. In particular, its parent (``d_parent``), and its
+name (``d_name``) cannot be changed, and it cannot be removed from the
+dentry hash table.
+
+When looking for a name in a directory, REF-walk takes ``d_lock`` on
+each candidate dentry that it finds in the hash table and then checks
+that the parent and name are correct. So it doesn't lock the parent
+while searching in the cache; it only locks children.
+
+When looking for the parent for a given name (to handle "``..``"),
+REF-walk can take ``d_lock`` to get a stable reference to ``d_parent``,
+but it first tries a more lightweight approach. As seen in
+``dget_parent()``, if a reference can be claimed on the parent, and if
+subsequently ``d_parent`` can be seen to have not changed, then there is
+no need to actually take the lock on the child.
+
+rename_lock
+~~~~~~~~~~~
+
+Looking up a given name in a given directory involves computing a hash
+from the two values (the name and the dentry of the directory),
+accessing that slot in a hash table, and searching the linked list
+that is found there.
+
+When a dentry is renamed, the name and the parent dentry can both
+change so the hash will almost certainly change too. This would move the
+dentry to a different chain in the hash table. If a filename search
+happened to be looking at a dentry that was moved in this way,
+it might end up continuing the search down the wrong chain,
+and so miss out on part of the correct chain.
+
+The name-lookup process (``d_lookup()``) does _not_ try to prevent this
+from happening, but only to detect when it happens.
+``rename_lock`` is a seqlock that is updated whenever any dentry is
+renamed. If ``d_lookup`` finds that a rename happened while it
+unsuccessfully scanned a chain in the hash table, it simply tries
+again.
+
+inode->i_rwsem
+~~~~~~~~~~~~~~
+
+``i_rwsem`` is a read/write semaphore that serializes all changes to a particular
+directory. This ensures that, for example, an ``unlink()`` and a ``rename()``
+cannot both happen at the same time. It also keeps the directory
+stable while the filesystem is asked to look up a name that is not
+currently in the dcache or, optionally, when the list of entries in a
+directory is being retrieved with ``readdir()``.
+
+This has a complementary role to that of ``d_lock``: ``i_rwsem`` on a
+directory protects all of the names in that directory, while ``d_lock``
+on a name protects just one name in a directory. Most changes to the
+dcache hold ``i_rwsem`` on the relevant directory inode and briefly take
+``d_lock`` on one or more the dentries while the change happens. One
+exception is when idle dentries are removed from the dcache due to
+memory pressure. This uses ``d_lock``, but ``i_rwsem`` plays no role.
+
+The semaphore affects pathname lookup in two distinct ways. Firstly it
+prevents changes during lookup of a name in a directory. ``walk_component()`` uses
+``lookup_fast()`` first which, in turn, checks to see if the name is in the cache,
+using only ``d_lock`` locking. If the name isn't found, then ``walk_component()``
+falls back to ``lookup_slow()`` which takes a shared lock on ``i_rwsem``, checks again that
+the name isn't in the cache, and then calls in to the filesystem to get a
+definitive answer. A new dentry will be added to the cache regardless of
+the result.
+
+Secondly, when pathname lookup reaches the final component, it will
+sometimes need to take an exclusive lock on ``i_rwsem`` before performing the last lookup so
+that the required exclusion can be achieved. How path lookup chooses
+to take, or not take, ``i_rwsem`` is one of the
+issues addressed in a subsequent section.
+
+If two threads attempt to look up the same name at the same time - a
+name that is not yet in the dcache - the shared lock on ``i_rwsem`` will
+not prevent them both adding new dentries with the same name. As this
+would result in confusion an extra level of interlocking is used,
+based around a secondary hash table (``in_lookup_hashtable``) and a
+per-dentry flag bit (``DCACHE_PAR_LOOKUP``).
+
+To add a new dentry to the cache while only holding a shared lock on
+``i_rwsem``, a thread must call ``d_alloc_parallel()``. This allocates a
+dentry, stores the required name and parent in it, checks if there
+is already a matching dentry in the primary or secondary hash
+tables, and if not, stores the newly allocated dentry in the secondary
+hash table, with ``DCACHE_PAR_LOOKUP`` set.
+
+If a matching dentry was found in the primary hash table then that is
+returned and the caller can know that it lost a race with some other
+thread adding the entry. If no matching dentry is found in either
+cache, the newly allocated dentry is returned and the caller can
+detect this from the presence of ``DCACHE_PAR_LOOKUP``. In this case it
+knows that it has won any race and now is responsible for asking the
+filesystem to perform the lookup and find the matching inode. When
+the lookup is complete, it must call ``d_lookup_done()`` which clears
+the flag and does some other house keeping, including removing the
+dentry from the secondary hash table - it will normally have been
+added to the primary hash table already. Note that a ``struct
+waitqueue_head`` is passed to ``d_alloc_parallel()``, and
+``d_lookup_done()`` must be called while this ``waitqueue_head`` is still
+in scope.
+
+If a matching dentry is found in the secondary hash table,
+``d_alloc_parallel()`` has a little more work to do. It first waits for
+``DCACHE_PAR_LOOKUP`` to be cleared, using a wait_queue that was passed
+to the instance of ``d_alloc_parallel()`` that won the race and that
+will be woken by the call to ``d_lookup_done()``. It then checks to see
+if the dentry has now been added to the primary hash table. If it
+has, the dentry is returned and the caller just sees that it lost any
+race. If it hasn't been added to the primary hash table, the most
+likely explanation is that some other dentry was added instead using
+``d_splice_alias()``. In any case, ``d_alloc_parallel()`` repeats all the
+look ups from the start and will normally return something from the
+primary hash table.
+
+mnt->mnt_count
+~~~~~~~~~~~~~~
+
+``mnt_count`` is a per-CPU reference counter on "``mount``" structures.
+Per-CPU here means that incrementing the count is cheap as it only
+uses CPU-local memory, but checking if the count is zero is expensive as
+it needs to check with every CPU. Taking a ``mnt_count`` reference
+prevents the mount structure from disappearing as the result of regular
+unmount operations, but does not prevent a "lazy" unmount. So holding
+``mnt_count`` doesn't ensure that the mount remains in the namespace and,
+in particular, doesn't stabilize the link to the mounted-on dentry. It
+does, however, ensure that the ``mount`` data structure remains coherent,
+and it provides a reference to the root dentry of the mounted
+filesystem. So a reference through ``->mnt_count`` provides a stable
+reference to the mounted dentry, but not the mounted-on dentry.
+
+mount_lock
+~~~~~~~~~~
+
+``mount_lock`` is a global seqlock, a bit like ``rename_lock``. It can be used to
+check if any change has been made to any mount points.
+
+While walking down the tree (away from the root) this lock is used when
+crossing a mount point to check that the crossing was safe. That is,
+the value in the seqlock is read, then the code finds the mount that
+is mounted on the current directory, if there is one, and increments
+the ``mnt_count``. Finally the value in ``mount_lock`` is checked against
+the old value. If there is no change, then the crossing was safe. If there
+was a change, the ``mnt_count`` is decremented and the whole process is
+retried.
+
+When walking up the tree (towards the root) by following a ".." link,
+a little more care is needed. In this case the seqlock (which
+contains both a counter and a spinlock) is fully locked to prevent
+any changes to any mount points while stepping up. This locking is
+needed to stabilize the link to the mounted-on dentry, which the
+refcount on the mount itself doesn't ensure.
+
+RCU
+~~~
+
+Finally the global (but extremely lightweight) RCU read lock is held
+from time to time to ensure certain data structures don't get freed
+unexpectedly.
+
+In particular it is held while scanning chains in the dcache hash
+table, and the mount point hash table.
+
+Bringing it together with ``struct nameidata``
+----------------------------------------------
+
+.. _First edition Unix: http://minnie.tuhs.org/cgi-bin/utree.pl?file=V1/u2.s
+
+Throughout the process of walking a path, the current status is stored
+in a ``struct nameidata``, "namei" being the traditional name - dating
+all the way back to `First Edition Unix`_ - of the function that
+converts a "name" to an "inode". ``struct nameidata`` contains (among
+other fields):
+
+``struct path path``
+~~~~~~~~~~~~~~~~~~~~
+
+A ``path`` contains a ``struct vfsmount`` (which is
+embedded in a ``struct mount``) and a ``struct dentry``. Together these
+record the current status of the walk. They start out referring to the
+starting point (the current working directory, the root directory, or some other
+directory identified by a file descriptor), and are updated on each
+step. A reference through ``d_lockref`` and ``mnt_count`` is always
+held.
+
+``struct qstr last``
+~~~~~~~~~~~~~~~~~~~~
+
+This is a string together with a length (i.e. _not_ ``nul`` terminated)
+that is the "next" component in the pathname.
+
+``int last_type``
+~~~~~~~~~~~~~~~~~
+
+This is one of ``LAST_NORM``, ``LAST_ROOT``, ``LAST_DOT``, ``LAST_DOTDOT``, or
+``LAST_BIND``. The ``last`` field is only valid if the type is
+``LAST_NORM``. ``LAST_BIND`` is used when following a symlink and no
+components of the symlink have been processed yet. Others should be
+fairly self-explanatory.
+
+``struct path root``
+~~~~~~~~~~~~~~~~~~~~
+
+This is used to hold a reference to the effective root of the
+filesystem. Often that reference won't be needed, so this field is
+only assigned the first time it is used, or when a non-standard root
+is requested. Keeping a reference in the ``nameidata`` ensures that
+only one root is in effect for the entire path walk, even if it races
+with a ``chroot()`` system call.
+
+The root is needed when either of two conditions holds: (1) either the
+pathname or a symbolic link starts with a "'/'", or (2) a "``..``"
+component is being handled, since "``..``" from the root must always stay
+at the root. The value used is usually the current root directory of
+the calling process. An alternate root can be provided as when
+``sysctl()`` calls ``file_open_root()``, and when NFSv4 or Btrfs call
+``mount_subtree()``. In each case a pathname is being looked up in a very
+specific part of the filesystem, and the lookup must not be allowed to
+escape that subtree. It works a bit like a local ``chroot()``.
+
+Ignoring the handling of symbolic links, we can now describe the
+"``link_path_walk()``" function, which handles the lookup of everything
+except the final component as:
+
+ Given a path (``name``) and a nameidata structure (``nd``), check that the
+ current directory has execute permission and then advance ``name``
+ over one component while updating ``last_type`` and ``last``. If that
+ was the final component, then return, otherwise call
+ ``walk_component()`` and repeat from the top.
+
+``walk_component()`` is even easier. If the component is ``LAST_DOTS``,
+it calls ``handle_dots()`` which does the necessary locking as already
+described. If it finds a ``LAST_NORM`` component it first calls
+"``lookup_fast()``" which only looks in the dcache, but will ask the
+filesystem to revalidate the result if it is that sort of filesystem.
+If that doesn't get a good result, it calls "``lookup_slow()``" which
+takes ``i_rwsem``, rechecks the cache, and then asks the filesystem
+to find a definitive answer. Each of these will call
+``follow_managed()`` (as described below) to handle any mount points.
+
+In the absence of symbolic links, ``walk_component()`` creates a new
+``struct path`` containing a counted reference to the new dentry and a
+reference to the new ``vfsmount`` which is only counted if it is
+different from the previous ``vfsmount``. It then calls
+``path_to_nameidata()`` to install the new ``struct path`` in the
+``struct nameidata`` and drop the unneeded references.
+
+This "hand-over-hand" sequencing of getting a reference to the new
+dentry before dropping the reference to the previous dentry may
+seem obvious, but is worth pointing out so that we will recognize its
+analogue in the "RCU-walk" version.
+
+Handling the final component
+----------------------------
+
+``link_path_walk()`` only walks as far as setting ``nd->last`` and
+``nd->last_type`` to refer to the final component of the path. It does
+not call ``walk_component()`` that last time. Handling that final
+component remains for the caller to sort out. Those callers are
+``path_lookupat()``, ``path_parentat()``, ``path_mountpoint()`` and
+``path_openat()`` each of which handles the differing requirements of
+different system calls.
+
+``path_parentat()`` is clearly the simplest - it just wraps a little bit
+of housekeeping around ``link_path_walk()`` and returns the parent
+directory and final component to the caller. The caller will be either
+aiming to create a name (via ``filename_create()``) or remove or rename
+a name (in which case ``user_path_parent()`` is used). They will use
+``i_rwsem`` to exclude other changes while they validate and then
+perform their operation.
+
+``path_lookupat()`` is nearly as simple - it is used when an existing
+object is wanted such as by ``stat()`` or ``chmod()``. It essentially just
+calls ``walk_component()`` on the final component through a call to
+``lookup_last()``. ``path_lookupat()`` returns just the final dentry.
+
+``path_mountpoint()`` handles the special case of unmounting which must
+not try to revalidate the mounted filesystem. It effectively
+contains, through a call to ``mountpoint_last()``, an alternate
+implementation of ``lookup_slow()`` which skips that step. This is
+important when unmounting a filesystem that is inaccessible, such as
+one provided by a dead NFS server.
+
+Finally ``path_openat()`` is used for the ``open()`` system call; it
+contains, in support functions starting with "``do_last()``", all the
+complexity needed to handle the different subtleties of O_CREAT (with
+or without O_EXCL), final "``/``" characters, and trailing symbolic
+links. We will revisit this in the final part of this series, which
+focuses on those symbolic links. "``do_last()``" will sometimes, but
+not always, take ``i_rwsem``, depending on what it finds.
+
+Each of these, or the functions which call them, need to be alert to
+the possibility that the final component is not ``LAST_NORM``. If the
+goal of the lookup is to create something, then any value for
+``last_type`` other than ``LAST_NORM`` will result in an error. For
+example if ``path_parentat()`` reports ``LAST_DOTDOT``, then the caller
+won't try to create that name. They also check for trailing slashes
+by testing ``last.name[last.len]``. If there is any character beyond
+the final component, it must be a trailing slash.
+
+Revalidation and automounts
+---------------------------
+
+Apart from symbolic links, there are only two parts of the "REF-walk"
+process not yet covered. One is the handling of stale cache entries
+and the other is automounts.
+
+On filesystems that require it, the lookup routines will call the
+``->d_revalidate()`` dentry method to ensure that the cached information
+is current. This will often confirm validity or update a few details
+from a server. In some cases it may find that there has been change
+further up the path and that something that was thought to be valid
+previously isn't really. When this happens the lookup of the whole
+path is aborted and retried with the "``LOOKUP_REVAL``" flag set. This
+forces revalidation to be more thorough. We will see more details of
+this retry process in the next article.
+
+Automount points are locations in the filesystem where an attempt to
+lookup a name can trigger changes to how that lookup should be
+handled, in particular by mounting a filesystem there. These are
+covered in greater detail in autofs.txt in the Linux documentation
+tree, but a few notes specifically related to path lookup are in order
+here.
+
+The Linux VFS has a concept of "managed" dentries which is reflected
+in function names such as "``follow_managed()``". There are three
+potentially interesting things about these dentries corresponding
+to three different flags that might be set in ``dentry->d_flags``:
+
+``DCACHE_MANAGE_TRANSIT``
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If this flag has been set, then the filesystem has requested that the
+``d_manage()`` dentry operation be called before handling any possible
+mount point. This can perform two particular services:
+
+It can block to avoid races. If an automount point is being
+unmounted, the ``d_manage()`` function will usually wait for that
+process to complete before letting the new lookup proceed and possibly
+trigger a new automount.
+
+It can selectively allow only some processes to transit through a
+mount point. When a server process is managing automounts, it may
+need to access a directory without triggering normal automount
+processing. That server process can identify itself to the ``autofs``
+filesystem, which will then give it a special pass through
+``d_manage()`` by returning ``-EISDIR``.
+
+``DCACHE_MOUNTED``
+~~~~~~~~~~~~~~~~~~
+
+This flag is set on every dentry that is mounted on. As Linux
+supports multiple filesystem namespaces, it is possible that the
+dentry may not be mounted on in *this* namespace, just in some
+other. So this flag is seen as a hint, not a promise.
+
+If this flag is set, and ``d_manage()`` didn't return ``-EISDIR``,
+``lookup_mnt()`` is called to examine the mount hash table (honoring the
+``mount_lock`` described earlier) and possibly return a new ``vfsmount``
+and a new ``dentry`` (both with counted references).
+
+``DCACHE_NEED_AUTOMOUNT``
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If ``d_manage()`` allowed us to get this far, and ``lookup_mnt()`` didn't
+find a mount point, then this flag causes the ``d_automount()`` dentry
+operation to be called.
+
+The ``d_automount()`` operation can be arbitrarily complex and may
+communicate with server processes etc. but it should ultimately either
+report that there was an error, that there was nothing to mount, or
+should provide an updated ``struct path`` with new ``dentry`` and ``vfsmount``.
+
+In the latter case, ``finish_automount()`` will be called to safely
+install the new mount point into the mount table.
+
+There is no new locking of import here and it is important that no
+locks (only counted references) are held over this processing due to
+the very real possibility of extended delays.
+This will become more important next time when we examine RCU-walk
+which is particularly sensitive to delays.
+
+RCU-walk - faster pathname lookup in Linux
+==========================================
+
+RCU-walk is another algorithm for performing pathname lookup in Linux.
+It is in many ways similar to REF-walk and the two share quite a bit
+of code. The significant difference in RCU-walk is how it allows for
+the possibility of concurrent access.
+
+We noted that REF-walk is complex because there are numerous details
+and special cases. RCU-walk reduces this complexity by simply
+refusing to handle a number of cases -- it instead falls back to
+REF-walk. The difficulty with RCU-walk comes from a different
+direction: unfamiliarity. The locking rules when depending on RCU are
+quite different from traditional locking, so we will spend a little extra
+time when we come to those.
+
+Clear demarcation of roles
+--------------------------
+
+The easiest way to manage concurrency is to forcibly stop any other
+thread from changing the data structures that a given thread is
+looking at. In cases where no other thread would even think of
+changing the data and lots of different threads want to read at the
+same time, this can be very costly. Even when using locks that permit
+multiple concurrent readers, the simple act of updating the count of
+the number of current readers can impose an unwanted cost. So the
+goal when reading a shared data structure that no other process is
+changing is to avoid writing anything to memory at all. Take no
+locks, increment no counts, leave no footprints.
+
+The REF-walk mechanism already described certainly doesn't follow this
+principle, but then it is really designed to work when there may well
+be other threads modifying the data. RCU-walk, in contrast, is
+designed for the common situation where there are lots of frequent
+readers and only occasional writers. This may not be common in all
+parts of the filesystem tree, but in many parts it will be. For the
+other parts it is important that RCU-walk can quickly fall back to
+using REF-walk.
+
+Pathname lookup always starts in RCU-walk mode but only remains there
+as long as what it is looking for is in the cache and is stable. It
+dances lightly down the cached filesystem image, leaving no footprints
+and carefully watching where it is, to be sure it doesn't trip. If it
+notices that something has changed or is changing, or if something
+isn't in the cache, then it tries to stop gracefully and switch to
+REF-walk.
+
+This stopping requires getting a counted reference on the current
+``vfsmount`` and ``dentry``, and ensuring that these are still valid -
+that a path walk with REF-walk would have found the same entries.
+This is an invariant that RCU-walk must guarantee. It can only make
+decisions, such as selecting the next step, that are decisions which
+REF-walk could also have made if it were walking down the tree at the
+same time. If the graceful stop succeeds, the rest of the path is
+processed with the reliable, if slightly sluggish, REF-walk. If
+RCU-walk finds it cannot stop gracefully, it simply gives up and
+restarts from the top with REF-walk.
+
+This pattern of "try RCU-walk, if that fails try REF-walk" can be
+clearly seen in functions like ``filename_lookup()``,
+``filename_parentat()``, ``filename_mountpoint()``,
+``do_filp_open()``, and ``do_file_open_root()``. These five
+correspond roughly to the four ``path_``* functions we met earlier,
+each of which calls ``link_path_walk()``. The ``path_*`` functions are
+called using different mode flags until a mode is found which works.
+They are first called with ``LOOKUP_RCU`` set to request "RCU-walk". If
+that fails with the error ``ECHILD`` they are called again with no
+special flag to request "REF-walk". If either of those report the
+error ``ESTALE`` a final attempt is made with ``LOOKUP_REVAL`` set (and no
+``LOOKUP_RCU``) to ensure that entries found in the cache are forcibly
+revalidated - normally entries are only revalidated if the filesystem
+determines that they are too old to trust.
+
+The ``LOOKUP_RCU`` attempt may drop that flag internally and switch to
+REF-walk, but will never then try to switch back to RCU-walk. Places
+that trip up RCU-walk are much more likely to be near the leaves and
+so it is very unlikely that there will be much, if any, benefit from
+switching back.
+
+RCU and seqlocks: fast and light
+--------------------------------
+
+RCU is, unsurprisingly, critical to RCU-walk mode. The
+``rcu_read_lock()`` is held for the entire time that RCU-walk is walking
+down a path. The particular guarantee it provides is that the key
+data structures - dentries, inodes, super_blocks, and mounts - will
+not be freed while the lock is held. They might be unlinked or
+invalidated in one way or another, but the memory will not be
+repurposed so values in various fields will still be meaningful. This
+is the only guarantee that RCU provides; everything else is done using
+seqlocks.
+
+As we saw above, REF-walk holds a counted reference to the current
+dentry and the current vfsmount, and does not release those references
+before taking references to the "next" dentry or vfsmount. It also
+sometimes takes the ``d_lock`` spinlock. These references and locks are
+taken to prevent certain changes from happening. RCU-walk must not
+take those references or locks and so cannot prevent such changes.
+Instead, it checks to see if a change has been made, and aborts or
+retries if it has.
+
+To preserve the invariant mentioned above (that RCU-walk may only make
+decisions that REF-walk could have made), it must make the checks at
+or near the same places that REF-walk holds the references. So, when
+REF-walk increments a reference count or takes a spinlock, RCU-walk
+samples the status of a seqlock using ``read_seqcount_begin()`` or a
+similar function. When REF-walk decrements the count or drops the
+lock, RCU-walk checks if the sampled status is still valid using
+``read_seqcount_retry()`` or similar.
+
+However, there is a little bit more to seqlocks than that. If
+RCU-walk accesses two different fields in a seqlock-protected
+structure, or accesses the same field twice, there is no a priori
+guarantee of any consistency between those accesses. When consistency
+is needed - which it usually is - RCU-walk must take a copy and then
+use ``read_seqcount_retry()`` to validate that copy.
+
+``read_seqcount_retry()`` not only checks the sequence number, but also
+imposes a memory barrier so that no memory-read instruction from
+*before* the call can be delayed until *after* the call, either by the
+CPU or by the compiler. A simple example of this can be seen in
+``slow_dentry_cmp()`` which, for filesystems which do not use simple
+byte-wise name equality, calls into the filesystem to compare a name
+against a dentry. The length and name pointer are copied into local
+variables, then ``read_seqcount_retry()`` is called to confirm the two
+are consistent, and only then is ``->d_compare()`` called. When
+standard filename comparison is used, ``dentry_cmp()`` is called
+instead. Notably it does _not_ use ``read_seqcount_retry()``, but
+instead has a large comment explaining why the consistency guarantee
+isn't necessary. A subsequent ``read_seqcount_retry()`` will be
+sufficient to catch any problem that could occur at this point.
+
+With that little refresher on seqlocks out of the way we can look at
+the bigger picture of how RCU-walk uses seqlocks.
+
+``mount_lock`` and ``nd->m_seq``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We already met the ``mount_lock`` seqlock when REF-walk used it to
+ensure that crossing a mount point is performed safely. RCU-walk uses
+it for that too, but for quite a bit more.
+
+Instead of taking a counted reference to each ``vfsmount`` as it
+descends the tree, RCU-walk samples the state of ``mount_lock`` at the
+start of the walk and stores this initial sequence number in the
+``struct nameidata`` in the ``m_seq`` field. This one lock and one
+sequence number are used to validate all accesses to all ``vfsmounts``,
+and all mount point crossings. As changes to the mount table are
+relatively rare, it is reasonable to fall back on REF-walk any time
+that any "mount" or "unmount" happens.
+
+``m_seq`` is checked (using ``read_seqretry()``) at the end of an RCU-walk
+sequence, whether switching to REF-walk for the rest of the path or
+when the end of the path is reached. It is also checked when stepping
+down over a mount point (in ``__follow_mount_rcu()``) or up (in
+``follow_dotdot_rcu()``). If it is ever found to have changed, the
+whole RCU-walk sequence is aborted and the path is processed again by
+REF-walk.
+
+If RCU-walk finds that ``mount_lock`` hasn't changed then it can be sure
+that, had REF-walk taken counted references on each vfsmount, the
+results would have been the same. This ensures the invariant holds,
+at least for vfsmount structures.
+
+``dentry->d_seq`` and ``nd->seq``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In place of taking a count or lock on ``d_reflock``, RCU-walk samples
+the per-dentry ``d_seq`` seqlock, and stores the sequence number in the
+``seq`` field of the nameidata structure, so ``nd->seq`` should always be
+the current sequence number of ``nd->dentry``. This number needs to be
+revalidated after copying, and before using, the name, parent, or
+inode of the dentry.
+
+The handling of the name we have already looked at, and the parent is
+only accessed in ``follow_dotdot_rcu()`` which fairly trivially follows
+the required pattern, though it does so for three different cases.
+
+When not at a mount point, ``d_parent`` is followed and its ``d_seq`` is
+collected. When we are at a mount point, we instead follow the
+``mnt->mnt_mountpoint`` link to get a new dentry and collect its
+``d_seq``. Then, after finally finding a ``d_parent`` to follow, we must
+check if we have landed on a mount point and, if so, must find that
+mount point and follow the ``mnt->mnt_root`` link. This would imply a
+somewhat unusual, but certainly possible, circumstance where the
+starting point of the path lookup was in part of the filesystem that
+was mounted on, and so not visible from the root.
+
+The inode pointer, stored in ``->d_inode``, is a little more
+interesting. The inode will always need to be accessed at least
+twice, once to determine if it is NULL and once to verify access
+permissions. Symlink handling requires a validated inode pointer too.
+Rather than revalidating on each access, a copy is made on the first
+access and it is stored in the ``inode`` field of ``nameidata`` from where
+it can be safely accessed without further validation.
+
+``lookup_fast()`` is the only lookup routine that is used in RCU-mode,
+``lookup_slow()`` being too slow and requiring locks. It is in
+``lookup_fast()`` that we find the important "hand over hand" tracking
+of the current dentry.
+
+The current ``dentry`` and current ``seq`` number are passed to
+``__d_lookup_rcu()`` which, on success, returns a new ``dentry`` and a
+new ``seq`` number. ``lookup_fast()`` then copies the inode pointer and
+revalidates the new ``seq`` number. It then validates the old ``dentry``
+with the old ``seq`` number one last time and only then continues. This
+process of getting the ``seq`` number of the new dentry and then
+checking the ``seq`` number of the old exactly mirrors the process of
+getting a counted reference to the new dentry before dropping that for
+the old dentry which we saw in REF-walk.
+
+No ``inode->i_rwsem`` or even ``rename_lock``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A semaphore is a fairly heavyweight lock that can only be taken when it is
+permissible to sleep. As ``rcu_read_lock()`` forbids sleeping,
+``inode->i_rwsem`` plays no role in RCU-walk. If some other thread does
+take ``i_rwsem`` and modifies the directory in a way that RCU-walk needs
+to notice, the result will be either that RCU-walk fails to find the
+dentry that it is looking for, or it will find a dentry which
+``read_seqretry()`` won't validate. In either case it will drop down to
+REF-walk mode which can take whatever locks are needed.
+
+Though ``rename_lock`` could be used by RCU-walk as it doesn't require
+any sleeping, RCU-walk doesn't bother. REF-walk uses ``rename_lock`` to
+protect against the possibility of hash chains in the dcache changing
+while they are being searched. This can result in failing to find
+something that actually is there. When RCU-walk fails to find
+something in the dentry cache, whether it is really there or not, it
+already drops down to REF-walk and tries again with appropriate
+locking. This neatly handles all cases, so adding extra checks on
+rename_lock would bring no significant value.
+
+``unlazy walk()`` and ``complete_walk()``
+-----------------------------------------
+
+That "dropping down to REF-walk" typically involves a call to
+``unlazy_walk()``, so named because "RCU-walk" is also sometimes
+referred to as "lazy walk". ``unlazy_walk()`` is called when
+following the path down to the current vfsmount/dentry pair seems to
+have proceeded successfully, but the next step is problematic. This
+can happen if the next name cannot be found in the dcache, if
+permission checking or name revalidation couldn't be achieved while
+the ``rcu_read_lock()`` is held (which forbids sleeping), if an
+automount point is found, or in a couple of cases involving symlinks.
+It is also called from ``complete_walk()`` when the lookup has reached
+the final component, or the very end of the path, depending on which
+particular flavor of lookup is used.
+
+Other reasons for dropping out of RCU-walk that do not trigger a call
+to ``unlazy_walk()`` are when some inconsistency is found that cannot be
+handled immediately, such as ``mount_lock`` or one of the ``d_seq``
+seqlocks reporting a change. In these cases the relevant function
+will return ``-ECHILD`` which will percolate up until it triggers a new
+attempt from the top using REF-walk.
+
+For those cases where ``unlazy_walk()`` is an option, it essentially
+takes a reference on each of the pointers that it holds (vfsmount,
+dentry, and possibly some symbolic links) and then verifies that the
+relevant seqlocks have not been changed. If there have been changes,
+it, too, aborts with ``-ECHILD``, otherwise the transition to REF-walk
+has been a success and the lookup process continues.
+
+Taking a reference on those pointers is not quite as simple as just
+incrementing a counter. That works to take a second reference if you
+already have one (often indirectly through another object), but it
+isn't sufficient if you don't actually have a counted reference at
+all. For ``dentry->d_lockref``, it is safe to increment the reference
+counter to get a reference unless it has been explicitly marked as
+"dead" which involves setting the counter to ``-128``.
+``lockref_get_not_dead()`` achieves this.
+
+For ``mnt->mnt_count`` it is safe to take a reference as long as
+``mount_lock`` is then used to validate the reference. If that
+validation fails, it may *not* be safe to just drop that reference in
+the standard way of calling ``mnt_put()`` - an unmount may have
+progressed too far. So the code in ``legitimize_mnt()``, when it
+finds that the reference it got might not be safe, checks the
+``MNT_SYNC_UMOUNT`` flag to determine if a simple ``mnt_put()`` is
+correct, or if it should just decrement the count and pretend none of
+this ever happened.
+
+Taking care in filesystems
+--------------------------
+
+RCU-walk depends almost entirely on cached information and often will
+not call into the filesystem at all. However there are two places,
+besides the already-mentioned component-name comparison, where the
+file system might be included in RCU-walk, and it must know to be
+careful.
+
+If the filesystem has non-standard permission-checking requirements -
+such as a networked filesystem which may need to check with the server
+- the ``i_op->permission`` interface might be called during RCU-walk.
+In this case an extra "``MAY_NOT_BLOCK``" flag is passed so that it
+knows not to sleep, but to return ``-ECHILD`` if it cannot complete
+promptly. ``i_op->permission`` is given the inode pointer, not the
+dentry, so it doesn't need to worry about further consistency checks.
+However if it accesses any other filesystem data structures, it must
+ensure they are safe to be accessed with only the ``rcu_read_lock()``
+held. This typically means they must be freed using ``kfree_rcu()`` or
+similar.
+
+.. _READ_ONCE: https://lwn.net/Articles/624126/
+
+If the filesystem may need to revalidate dcache entries, then
+``d_op->d_revalidate`` may be called in RCU-walk too. This interface
+*is* passed the dentry but does not have access to the ``inode`` or the
+``seq`` number from the ``nameidata``, so it needs to be extra careful
+when accessing fields in the dentry. This "extra care" typically
+involves using `READ_ONCE() <READ_ONCE_>`_ to access fields, and verifying the
+result is not NULL before using it. This pattern can be seen in
+``nfs_lookup_revalidate()``.
+
+A pair of patterns
+------------------
+
+In various places in the details of REF-walk and RCU-walk, and also in
+the big picture, there are a couple of related patterns that are worth
+being aware of.
+
+The first is "try quickly and check, if that fails try slowly". We
+can see that in the high-level approach of first trying RCU-walk and
+then trying REF-walk, and in places where ``unlazy_walk()`` is used to
+switch to REF-walk for the rest of the path. We also saw it earlier
+in ``dget_parent()`` when following a "``..``" link. It tries a quick way
+to get a reference, then falls back to taking locks if needed.
+
+The second pattern is "try quickly and check, if that fails try
+again - repeatedly". This is seen with the use of ``rename_lock`` and
+``mount_lock`` in REF-walk. RCU-walk doesn't make use of this pattern -
+if anything goes wrong it is much safer to just abort and try a more
+sedate approach.
+
+The emphasis here is "try quickly and check". It should probably be
+"try quickly _and carefully,_ then check". The fact that checking is
+needed is a reminder that the system is dynamic and only a limited
+number of things are safe at all. The most likely cause of errors in
+this whole process is assuming something is safe when in reality it
+isn't. Careful consideration of what exactly guarantees the safety of
+each access is sometimes necessary.
+
+A walk among the symlinks
+=========================
+
+There are several basic issues that we will examine to understand the
+handling of symbolic links: the symlink stack, together with cache
+lifetimes, will help us understand the overall recursive handling of
+symlinks and lead to the special care needed for the final component.
+Then a consideration of access-time updates and summary of the various
+flags controlling lookup will finish the story.
+
+The symlink stack
+-----------------
+
+There are only two sorts of filesystem objects that can usefully
+appear in a path prior to the final component: directories and symlinks.
+Handling directories is quite straightforward: the new directory
+simply becomes the starting point at which to interpret the next
+component on the path. Handling symbolic links requires a bit more
+work.
+
+Conceptually, symbolic links could be handled by editing the path. If
+a component name refers to a symbolic link, then that component is
+replaced by the body of the link and, if that body starts with a '/',
+then all preceding parts of the path are discarded. This is what the
+"``readlink -f``" command does, though it also edits out "``.``" and
+"``..``" components.
+
+Directly editing the path string is not really necessary when looking
+up a path, and discarding early components is pointless as they aren't
+looked at anyway. Keeping track of all remaining components is
+important, but they can of course be kept separately; there is no need
+to concatenate them. As one symlink may easily refer to another,
+which in turn can refer to a third, we may need to keep the remaining
+components of several paths, each to be processed when the preceding
+ones are completed. These path remnants are kept on a stack of
+limited size.
+
+There are two reasons for placing limits on how many symlinks can
+occur in a single path lookup. The most obvious is to avoid loops.
+If a symlink referred to itself either directly or through
+intermediaries, then following the symlink can never complete
+successfully - the error ``ELOOP`` must be returned. Loops can be
+detected without imposing limits, but limits are the simplest solution
+and, given the second reason for restriction, quite sufficient.
+
+.. _outlined recently: http://thread.gmane.org/gmane.linux.kernel/1934390/focus=1934550
+
+The second reason was `outlined recently`_ by Linus:
+
+ Because it's a latency and DoS issue too. We need to react well to
+ true loops, but also to "very deep" non-loops. It's not about memory
+ use, it's about users triggering unreasonable CPU resources.
+
+Linux imposes a limit on the length of any pathname: ``PATH_MAX``, which
+is 4096. There are a number of reasons for this limit; not letting the
+kernel spend too much time on just one path is one of them. With
+symbolic links you can effectively generate much longer paths so some
+sort of limit is needed for the same reason. Linux imposes a limit of
+at most 40 symlinks in any one path lookup. It previously imposed a
+further limit of eight on the maximum depth of recursion, but that was
+raised to 40 when a separate stack was implemented, so there is now
+just the one limit.
+
+The ``nameidata`` structure that we met in an earlier article contains a
+small stack that can be used to store the remaining part of up to two
+symlinks. In many cases this will be sufficient. If it isn't, a
+separate stack is allocated with room for 40 symlinks. Pathname
+lookup will never exceed that stack as, once the 40th symlink is
+detected, an error is returned.
+
+It might seem that the name remnants are all that needs to be stored on
+this stack, but we need a bit more. To see that, we need to move on to
+cache lifetimes.
+
+Storage and lifetime of cached symlinks
+---------------------------------------
+
+Like other filesystem resources, such as inodes and directory
+entries, symlinks are cached by Linux to avoid repeated costly access
+to external storage. It is particularly important for RCU-walk to be
+able to find and temporarily hold onto these cached entries, so that
+it doesn't need to drop down into REF-walk.
+
+.. _object-oriented design pattern: https://lwn.net/Articles/446317/
+
+While each filesystem is free to make its own choice, symlinks are
+typically stored in one of two places. Short symlinks are often
+stored directly in the inode. When a filesystem allocates a ``struct
+inode`` it typically allocates extra space to store private data (a
+common `object-oriented design pattern`_ in the kernel). This will
+sometimes include space for a symlink. The other common location is
+in the page cache, which normally stores the content of files. The
+pathname in a symlink can be seen as the content of that symlink and
+can easily be stored in the page cache just like file content.
+
+When neither of these is suitable, the next most likely scenario is
+that the filesystem will allocate some temporary memory and copy or
+construct the symlink content into that memory whenever it is needed.
+
+When the symlink is stored in the inode, it has the same lifetime as
+the inode which, itself, is protected by RCU or by a counted reference
+on the dentry. This means that the mechanisms that pathname lookup
+uses to access the dcache and icache (inode cache) safely are quite
+sufficient for accessing some cached symlinks safely. In these cases,
+the ``i_link`` pointer in the inode is set to point to wherever the
+symlink is stored and it can be accessed directly whenever needed.
+
+When the symlink is stored in the page cache or elsewhere, the
+situation is not so straightforward. A reference on a dentry or even
+on an inode does not imply any reference on cached pages of that
+inode, and even an ``rcu_read_lock()`` is not sufficient to ensure that
+a page will not disappear. So for these symlinks the pathname lookup
+code needs to ask the filesystem to provide a stable reference and,
+significantly, needs to release that reference when it is finished
+with it.
+
+Taking a reference to a cache page is often possible even in RCU-walk
+mode. It does require making changes to memory, which is best avoided,
+but that isn't necessarily a big cost and it is better than dropping
+out of RCU-walk mode completely. Even filesystems that allocate
+space to copy the symlink into can use ``GFP_ATOMIC`` to often successfully
+allocate memory without the need to drop out of RCU-walk. If a
+filesystem cannot successfully get a reference in RCU-walk mode, it
+must return ``-ECHILD`` and ``unlazy_walk()`` will be called to return to
+REF-walk mode in which the filesystem is allowed to sleep.
+
+The place for all this to happen is the ``i_op->follow_link()`` inode
+method. In the present mainline code this is never actually called in
+RCU-walk mode as the rewrite is not quite complete. It is likely that
+in a future release this method will be passed an ``inode`` pointer when
+called in RCU-walk mode so it both (1) knows to be careful, and (2) has the
+validated pointer. Much like the ``i_op->permission()`` method we
+looked at previously, ``->follow_link()`` would need to be careful that
+all the data structures it references are safe to be accessed while
+holding no counted reference, only the RCU lock. Though getting a
+reference with ``->follow_link()`` is not yet done in RCU-walk mode, the
+code is ready to release the reference when that does happen.
+
+This need to drop the reference to a symlink adds significant
+complexity. It requires a reference to the inode so that the
+``i_op->put_link()`` inode operation can be called. In REF-walk, that
+reference is kept implicitly through a reference to the dentry, so
+keeping the ``struct path`` of the symlink is easiest. For RCU-walk,
+the pointer to the inode is kept separately. To allow switching from
+RCU-walk back to REF-walk in the middle of processing nested symlinks
+we also need the seq number for the dentry so we can confirm that
+switching back was safe.
+
+Finally, when providing a reference to a symlink, the filesystem also
+provides an opaque "cookie" that must be passed to ``->put_link()`` so that it
+knows what to free. This might be the allocated memory area, or a
+pointer to the ``struct page`` in the page cache, or something else
+completely. Only the filesystem knows what it is.
+
+In order for the reference to each symlink to be dropped when the walk completes,
+whether in RCU-walk or REF-walk, the symlink stack needs to contain,
+along with the path remnants:
+
+- the ``struct path`` to provide a reference to the inode in REF-walk
+- the ``struct inode *`` to provide a reference to the inode in RCU-walk
+- the ``seq`` to allow the path to be safely switched from RCU-walk to REF-walk
+- the ``cookie`` that tells ``->put_path()`` what to put.
+
+This means that each entry in the symlink stack needs to hold five
+pointers and an integer instead of just one pointer (the path
+remnant). On a 64-bit system, this is about 40 bytes per entry;
+with 40 entries it adds up to 1600 bytes total, which is less than
+half a page. So it might seem like a lot, but is by no means
+excessive.
+
+Note that, in a given stack frame, the path remnant (``name``) is not
+part of the symlink that the other fields refer to. It is the remnant
+to be followed once that symlink has been fully parsed.
+
+Following the symlink
+---------------------
+
+The main loop in ``link_path_walk()`` iterates seamlessly over all
+components in the path and all of the non-final symlinks. As symlinks
+are processed, the ``name`` pointer is adjusted to point to a new
+symlink, or is restored from the stack, so that much of the loop
+doesn't need to notice. Getting this ``name`` variable on and off the
+stack is very straightforward; pushing and popping the references is
+a little more complex.
+
+When a symlink is found, ``walk_component()`` returns the value ``1``
+(``0`` is returned for any other sort of success, and a negative number
+is, as usual, an error indicator). This causes ``get_link()`` to be
+called; it then gets the link from the filesystem. Providing that
+operation is successful, the old path ``name`` is placed on the stack,
+and the new value is used as the ``name`` for a while. When the end of
+the path is found (i.e. ``*name`` is ``'\0'``) the old ``name`` is restored
+off the stack and path walking continues.
+
+Pushing and popping the reference pointers (inode, cookie, etc.) is more
+complex in part because of the desire to handle tail recursion. When
+the last component of a symlink itself points to a symlink, we
+want to pop the symlink-just-completed off the stack before pushing
+the symlink-just-found to avoid leaving empty path remnants that would
+just get in the way.
+
+It is most convenient to push the new symlink references onto the
+stack in ``walk_component()`` immediately when the symlink is found;
+``walk_component()`` is also the last piece of code that needs to look at the
+old symlink as it walks that last component. So it is quite
+convenient for ``walk_component()`` to release the old symlink and pop
+the references just before pushing the reference information for the
+new symlink. It is guided in this by two flags; ``WALK_GET``, which
+gives it permission to follow a symlink if it finds one, and
+``WALK_PUT``, which tells it to release the current symlink after it has been
+followed. ``WALK_PUT`` is tested first, leading to a call to
+``put_link()``. ``WALK_GET`` is tested subsequently (by
+``should_follow_link()``) leading to a call to ``pick_link()`` which sets
+up the stack frame.
+
+Symlinks with no final component
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A pair of special-case symlinks deserve a little further explanation.
+Both result in a new ``struct path`` (with mount and dentry) being set
+up in the ``nameidata``, and result in ``get_link()`` returning ``NULL``.
+
+The more obvious case is a symlink to "``/``". All symlinks starting
+with "``/``" are detected in ``get_link()`` which resets the ``nameidata``
+to point to the effective filesystem root. If the symlink only
+contains "``/``" then there is nothing more to do, no components at all,
+so ``NULL`` is returned to indicate that the symlink can be released and
+the stack frame discarded.
+
+The other case involves things in ``/proc`` that look like symlinks but
+aren't really::
+
+ $ ls -l /proc/self/fd/1
+ lrwx------ 1 neilb neilb 64 Jun 13 10:19 /proc/self/fd/1 -> /dev/pts/4
+
+Every open file descriptor in any process is represented in ``/proc`` by
+something that looks like a symlink. It is really a reference to the
+target file, not just the name of it. When you ``readlink`` these
+objects you get a name that might refer to the same file - unless it
+has been unlinked or mounted over. When ``walk_component()`` follows
+one of these, the ``->follow_link()`` method in "procfs" doesn't return
+a string name, but instead calls ``nd_jump_link()`` which updates the
+``nameidata`` in place to point to that target. ``->follow_link()`` then
+returns ``NULL``. Again there is no final component and ``get_link()``
+reports this by leaving the ``last_type`` field of ``nameidata`` as
+``LAST_BIND``.
+
+Following the symlink in the final component
+--------------------------------------------
+
+All this leads to ``link_path_walk()`` walking down every component, and
+following all symbolic links it finds, until it reaches the final
+component. This is just returned in the ``last`` field of ``nameidata``.
+For some callers, this is all they need; they want to create that
+``last`` name if it doesn't exist or give an error if it does. Other
+callers will want to follow a symlink if one is found, and possibly
+apply special handling to the last component of that symlink, rather
+than just the last component of the original file name. These callers
+potentially need to call ``link_path_walk()`` again and again on
+successive symlinks until one is found that doesn't point to another
+symlink.
+
+This case is handled by the relevant caller of ``link_path_walk()``, such as
+``path_lookupat()`` using a loop that calls ``link_path_walk()``, and then
+handles the final component. If the final component is a symlink
+that needs to be followed, then ``trailing_symlink()`` is called to set
+things up properly and the loop repeats, calling ``link_path_walk()``
+again. This could loop as many as 40 times if the last component of
+each symlink is another symlink.
+
+The various functions that examine the final component and possibly
+report that it is a symlink are ``lookup_last()``, ``mountpoint_last()``
+and ``do_last()``, each of which use the same convention as
+``walk_component()`` of returning ``1`` if a symlink was found that needs
+to be followed.
+
+Of these, ``do_last()`` is the most interesting as it is used for
+opening a file. Part of ``do_last()`` runs with ``i_rwsem`` held and this
+part is in a separate function: ``lookup_open()``.
+
+Explaining ``do_last()`` completely is beyond the scope of this article,
+but a few highlights should help those interested in exploring the
+code.
+
+1. Rather than just finding the target file, ``do_last()`` needs to open
+ it. If the file was found in the dcache, then ``vfs_open()`` is used for
+ this. If not, then ``lookup_open()`` will either call ``atomic_open()`` (if
+ the filesystem provides it) to combine the final lookup with the open, or
+ will perform the separate ``lookup_real()`` and ``vfs_create()`` steps
+ directly. In the later case the actual "open" of this newly found or
+ created file will be performed by ``vfs_open()``, just as if the name
+ were found in the dcache.
+
+2. ``vfs_open()`` can fail with ``-EOPENSTALE`` if the cached information
+ wasn't quite current enough. Rather than restarting the lookup from
+ the top with ``LOOKUP_REVAL`` set, ``lookup_open()`` is called instead,
+ giving the filesystem a chance to resolve small inconsistencies.
+ If that doesn't work, only then is the lookup restarted from the top.
+
+3. An open with O_CREAT **does** follow a symlink in the final component,
+ unlike other creation system calls (like ``mkdir``). So the sequence::
+
+ ln -s bar /tmp/foo
+ echo hello > /tmp/foo
+
+ will create a file called ``/tmp/bar``. This is not permitted if
+ ``O_EXCL`` is set but otherwise is handled for an O_CREAT open much
+ like for a non-creating open: ``should_follow_link()`` returns ``1``, and
+ so does ``do_last()`` so that ``trailing_symlink()`` gets called and the
+ open process continues on the symlink that was found.
+
+Updating the access time
+------------------------
+
+We previously said of RCU-walk that it would "take no locks, increment
+no counts, leave no footprints." We have since seen that some
+"footprints" can be needed when handling symlinks as a counted
+reference (or even a memory allocation) may be needed. But these
+footprints are best kept to a minimum.
+
+One other place where walking down a symlink can involve leaving
+footprints in a way that doesn't affect directories is in updating access times.
+In Unix (and Linux) every filesystem object has a "last accessed
+time", or "``atime``". Passing through a directory to access a file
+within is not considered to be an access for the purposes of
+``atime``; only listing the contents of a directory can update its ``atime``.
+Symlinks are different it seems. Both reading a symlink (with ``readlink()``)
+and looking up a symlink on the way to some other destination can
+update the atime on that symlink.
+
+.. _clearest statement: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_08
+
+It is not clear why this is the case; POSIX has little to say on the
+subject. The `clearest statement`_ is that, if a particular implementation
+updates a timestamp in a place not specified by POSIX, this must be
+documented "except that any changes caused by pathname resolution need
+not be documented". This seems to imply that POSIX doesn't really
+care about access-time updates during pathname lookup.
+
+.. _Linux 1.3.87: https://git.kernel.org/cgit/linux/kernel/git/history/history.git/diff/fs/ext2/symlink.c?id=f806c6db77b8eaa6e00dcfb6b567706feae8dbb8
+
+An examination of history shows that prior to `Linux 1.3.87`_, the ext2
+filesystem, at least, didn't update atime when following a link.
+Unfortunately we have no record of why that behavior was changed.
+
+In any case, access time must now be updated and that operation can be
+quite complex. Trying to stay in RCU-walk while doing it is best
+avoided. Fortunately it is often permitted to skip the ``atime``
+update. Because ``atime`` updates cause performance problems in various
+areas, Linux supports the ``relatime`` mount option, which generally
+limits the updates of ``atime`` to once per day on files that aren't
+being changed (and symlinks never change once created). Even without
+``relatime``, many filesystems record ``atime`` with a one-second
+granularity, so only one update per second is required.
+
+It is easy to test if an ``atime`` update is needed while in RCU-walk
+mode and, if it isn't, the update can be skipped and RCU-walk mode
+continues. Only when an ``atime`` update is actually required does the
+path walk drop down to REF-walk. All of this is handled in the
+``get_link()`` function.
+
+A few flags
+-----------
+
+A suitable way to wrap up this tour of pathname walking is to list
+the various flags that can be stored in the ``nameidata`` to guide the
+lookup process. Many of these are only meaningful on the final
+component, others reflect the current state of the pathname lookup.
+And then there is ``LOOKUP_EMPTY``, which doesn't fit conceptually with
+the others. If this is not set, an empty pathname causes an error
+very early on. If it is set, empty pathnames are not considered to be
+an error.
+
+Global state flags
+~~~~~~~~~~~~~~~~~~
+
+We have already met two global state flags: ``LOOKUP_RCU`` and
+``LOOKUP_REVAL``. These select between one of three overall approaches
+to lookup: RCU-walk, REF-walk, and REF-walk with forced revalidation.
+
+``LOOKUP_PARENT`` indicates that the final component hasn't been reached
+yet. This is primarily used to tell the audit subsystem the full
+context of a particular access being audited.
+
+``LOOKUP_ROOT`` indicates that the ``root`` field in the ``nameidata`` was
+provided by the caller, so it shouldn't be released when it is no
+longer needed.
+
+``LOOKUP_JUMPED`` means that the current dentry was chosen not because
+it had the right name but for some other reason. This happens when
+following "``..``", following a symlink to ``/``, crossing a mount point
+or accessing a "``/proc/$PID/fd/$FD``" symlink. In this case the
+filesystem has not been asked to revalidate the name (with
+``d_revalidate()``). In such cases the inode may still need to be
+revalidated, so ``d_op->d_weak_revalidate()`` is called if
+``LOOKUP_JUMPED`` is set when the look completes - which may be at the
+final component or, when creating, unlinking, or renaming, at the penultimate component.
+
+Final-component flags
+~~~~~~~~~~~~~~~~~~~~~
+
+Some of these flags are only set when the final component is being
+considered. Others are only checked for when considering that final
+component.
+
+``LOOKUP_AUTOMOUNT`` ensures that, if the final component is an automount
+point, then the mount is triggered. Some operations would trigger it
+anyway, but operations like ``stat()`` deliberately don't. ``statfs()``
+needs to trigger the mount but otherwise behaves a lot like ``stat()``, so
+it sets ``LOOKUP_AUTOMOUNT``, as does "``quotactl()``" and the handling of
+"``mount --bind``".
+
+``LOOKUP_FOLLOW`` has a similar function to ``LOOKUP_AUTOMOUNT`` but for
+symlinks. Some system calls set or clear it implicitly, while
+others have API flags such as ``AT_SYMLINK_FOLLOW`` and
+``UMOUNT_NOFOLLOW`` to control it. Its effect is similar to
+``WALK_GET`` that we already met, but it is used in a different way.
+
+``LOOKUP_DIRECTORY`` insists that the final component is a directory.
+Various callers set this and it is also set when the final component
+is found to be followed by a slash.
+
+Finally ``LOOKUP_OPEN``, ``LOOKUP_CREATE``, ``LOOKUP_EXCL``, and
+``LOOKUP_RENAME_TARGET`` are not used directly by the VFS but are made
+available to the filesystem and particularly the ``->d_revalidate()``
+method. A filesystem can choose not to bother revalidating too hard
+if it knows that it will be asked to open or create the file soon.
+These flags were previously useful for ``->lookup()`` too but with the
+introduction of ``->atomic_open()`` they are less relevant there.
+
+End of the road
+---------------
+
+Despite its complexity, all this pathname lookup code appears to be
+in good shape - various parts are certainly easier to understand now
+than even a couple of releases ago. But that doesn't mean it is
+"finished". As already mentioned, RCU-walk currently only follows
+symlinks that are stored in the inode so, while it handles many ext4
+symlinks, it doesn't help with NFS, XFS, or Btrfs. That support
+is not likely to be long delayed.
diff --git a/Documentation/filesystems/pohmelfs/design_notes.txt b/Documentation/filesystems/pohmelfs/design_notes.txt
deleted file mode 100644
index 106d17f..0000000
--- a/Documentation/filesystems/pohmelfs/design_notes.txt
+++ /dev/null
@@ -1,72 +0,0 @@
-POHMELFS: Parallel Optimized Host Message Exchange Layered File System.
-
- Evgeniy Polyakov <zbr@ioremap.net>
-
-Homepage: http://www.ioremap.net/projects/pohmelfs
-
-POHMELFS first began as a network filesystem with coherent local data and
-metadata caches but is now evolving into a parallel distributed filesystem.
-
-Main features of this FS include:
- * Locally coherent cache for data and metadata with (potentially) byte-range locks.
- Since all Linux filesystems lock the whole inode during writing, algorithm
- is very simple and does not use byte-ranges, although they are sent in
- locking messages.
- * Completely async processing of all events except creation of hard and symbolic
- links, and rename events.
- Object creation and data reading and writing are processed asynchronously.
- * Flexible object architecture optimized for network processing.
- Ability to create long paths to objects and remove arbitrarily huge
- directories with a single network command.
- (like removing the whole kernel tree via a single network command).
- * Very high performance.
- * Fast and scalable multithreaded userspace server. Being in userspace it works
- with any underlying filesystem and still is much faster than async in-kernel NFS one.
- * Client is able to switch between different servers (if one goes down, client
- automatically reconnects to second and so on).
- * Transactions support. Full failover for all operations.
- Resending transactions to different servers on timeout or error.
- * Read request (data read, directory listing, lookup requests) balancing between multiple servers.
- * Write requests are replicated to multiple servers and completed only when all of them are acked.
- * Ability to add and/or remove servers from the working set at run-time.
- * Strong authentication and possible data encryption in network channel.
- * Extended attributes support.
-
-POHMELFS is based on transactions, which are potentially long-standing objects that live
-in the client's memory. Each transaction contains all the information needed to process a given
-command (or set of commands, which is frequently used during data writing: single transactions
-can contain creation and data writing commands). Transactions are committed by all the servers
-to which they are sent and, in case of failures, are eventually resent or dropped with an error.
-For example, reading will return an error if no servers are available.
-
-POHMELFS uses a asynchronous approach to data processing. Courtesy of transactions, it is
-possible to detach replies from requests and, if the command requires data to be received, the
-caller sleeps waiting for it. Thus, it is possible to issue multiple read commands to different
-servers and async threads will pick up replies in parallel, find appropriate transactions in the
-system and put the data where it belongs (like the page or inode cache).
-
-The main feature of POHMELFS is writeback data and the metadata cache.
-Only a few non-performance critical operations use the write-through cache and
-are synchronous: hard and symbolic link creation, and object rename. Creation,
-removal of objects and data writing are asynchronous and are sent to
-the server during system writeback. Only one writer at a time is allowed for any
-given inode, which is guarded by an appropriate locking protocol.
-Because of this feature, POHMELFS is extremely fast at metadata intensive
-workloads and can fully utilize the bandwidth to the servers when doing bulk
-data transfers.
-
-POHMELFS clients operate with a working set of servers and are capable of balancing read-only
-operations (like lookups or directory listings) between them according to IO priorities.
-Administrators can add or remove servers from the set at run-time via special commands (described
-in Documentation/filesystems/pohmelfs/info.txt file). Writes are replicated to all servers, which
-are connected with write permission turned on. IO priority and permissions can be changed in
-run-time.
-
-POHMELFS is capable of full data channel encryption and/or strong crypto hashing.
-One can select any kernel supported cipher, encryption mode, hash type and operation mode
-(hmac or digest). It is also possible to use both or neither (default). Crypto configuration
-is checked during mount time and, if the server does not support it, appropriate capabilities
-will be disabled or mount will fail (if 'crypto_fail_unsupported' mount option is specified).
-Crypto performance heavily depends on the number of crypto threads, which asynchronously perform
-crypto operations and send the resulting data to server or submit it up the stack. This number
-can be controlled via a mount option.
diff --git a/Documentation/filesystems/pohmelfs/info.txt b/Documentation/filesystems/pohmelfs/info.txt
deleted file mode 100644
index db2e413..0000000
--- a/Documentation/filesystems/pohmelfs/info.txt
+++ /dev/null
@@ -1,99 +0,0 @@
-POHMELFS usage information.
-
-Mount options.
-All but index, number of crypto threads and maximum IO size can changed via remount.
-
-idx=%u
- Each mountpoint is associated with a special index via this option.
- Administrator can add or remove servers from the given index, so all mounts,
- which were attached to it, are updated.
- Default it is 0.
-
-trans_scan_timeout=%u
- This timeout, expressed in milliseconds, specifies time to scan transaction
- trees looking for stale requests, which have to be resent, or if number of
- retries exceed specified limit, dropped with error.
- Default is 5 seconds.
-
-drop_scan_timeout=%u
- Internal timeout, expressed in milliseconds, which specifies how frequently
- inodes marked to be dropped are freed. It also specifies how frequently
- the system checks that servers have to be added or removed from current working set.
- Default is 1 second.
-
-wait_on_page_timeout=%u
- Number of milliseconds to wait for reply from remote server for data reading command.
- If this timeout is exceeded, reading returns an error.
- Default is 5 seconds.
-
-trans_retries=%u
- This is the number of times that a transaction will be resent to a server that did
- not answer for the last @trans_scan_timeout milliseconds.
- When the number of resends exceeds this limit, the transaction is completed with error.
- Default is 5 resends.
-
-crypto_thread_num=%u
- Number of crypto processing threads. Threads are used both for RX and TX traffic.
- Default is 2, or no threads if crypto operations are not supported.
-
-trans_max_pages=%u
- Maximum number of pages in a single transaction. This parameter also controls
- the number of pages, allocated for crypto processing (each crypto thread has
- pool of pages, the number of which is equal to 'trans_max_pages'.
- Default is 100 pages.
-
-crypto_fail_unsupported
- If specified, mount will fail if the server does not support requested crypto operations.
- By default mount will disable non-matching crypto operations.
-
-mcache_timeout=%u
- Maximum number of milliseconds to wait for the mcache objects to be processed.
- Mcache includes locks (given lock should be granted by server), attributes (they should be
- fully received in the given timeframe).
- Default is 5 seconds.
-
-Usage examples.
-
-Add server server1.net:1025 into the working set with index $idx
-with appropriate hash algorithm and key file and cipher algorithm, mode and key file:
-$cfg A add -a server1.net -p 1025 -i $idx -K $hash_key -k $cipher_key
-
-Mount filesystem with given index $idx to /mnt mountpoint.
-Client will connect to all servers specified in the working set via previous command:
-mount -t pohmel -o idx=$idx q /mnt
-
-Change permissions to read-only (-I 1 option, '-I 2' - write-only, 3 - rw):
-$cfg A modify -a server1.net -p 1025 -i $idx -I 1
-
-Change IO priority to 123 (node with the highest priority gets read requests).
-$cfg A modify -a server1.net -p 1025 -i $idx -P 123
-
-One can check currect status of all connections in the mountstats file:
-# cat /proc/$PID/mountstats
-...
-device none mounted on /mnt with fstype pohmel
-idx addr(:port) socket_type protocol active priority permissions
-0 server1.net:1026 1 6 1 250 1
-0 server2.net:1025 1 6 1 123 3
-
-Server installation.
-
-Creating a server, which listens at port 1025 and 0.0.0.0 address.
-Working root directory (note, that server chroots there, so you have to have appropriate permissions)
-is set to /mnt, server will negotiate hash/cipher with client, in case client requested it, there
-are appropriate key files.
-Number of working threads is set to 10.
-
-# ./fserver -a 0.0.0.0 -p 1025 -r /mnt -w 10 -K hash_key -k cipher_key
-
- -A 6 - listen on ipv6 address. Default: Disabled.
- -r root - path to root directory. Default: /tmp.
- -a addr - listen address. Default: 0.0.0.0.
- -p port - listen port. Default: 1025.
- -w workers - number of workers per connected client. Default: 1.
- -K file - hash key size. Default: none.
- -k file - cipher key size. Default: none.
- -h - this help.
-
-Number of worker threads specifies how many workers will be created for each client.
-Bulk single-client transafers usually are better handled with smaller number (like 1-3).
diff --git a/Documentation/filesystems/pohmelfs/network_protocol.txt b/Documentation/filesystems/pohmelfs/network_protocol.txt
deleted file mode 100644
index c680b4b..0000000
--- a/Documentation/filesystems/pohmelfs/network_protocol.txt
+++ /dev/null
@@ -1,227 +0,0 @@
-POHMELFS network protocol.
-
-Basic structure used in network communication is following command:
-
-struct netfs_cmd
-{
- __u16 cmd; /* Command number */
- __u16 csize; /* Attached crypto information size */
- __u16 cpad; /* Attached padding size */
- __u16 ext; /* External flags */
- __u32 size; /* Size of the attached data */
- __u32 trans; /* Transaction id */
- __u64 id; /* Object ID to operate on. Used for feedback.*/
- __u64 start; /* Start of the object. */
- __u64 iv; /* IV sequence */
- __u8 data[0];
-};
-
-Commands can be embedded into transaction command (which in turn has own command),
-so one can extend protocol as needed without breaking backward compatibility as long
-as old commands are supported. All string lengths include tail 0 byte.
-
-All commands are transferred over the network in big-endian. CPU endianness is used at the end peers.
-
-@cmd - command number, which specifies command to be processed. Following
- commands are used currently:
-
- NETFS_READDIR = 1, /* Read directory for given inode number */
- NETFS_READ_PAGE, /* Read data page from the server */
- NETFS_WRITE_PAGE, /* Write data page to the server */
- NETFS_CREATE, /* Create directory entry */
- NETFS_REMOVE, /* Remove directory entry */
- NETFS_LOOKUP, /* Lookup single object */
- NETFS_LINK, /* Create a link */
- NETFS_TRANS, /* Transaction */
- NETFS_OPEN, /* Open intent */
- NETFS_INODE_INFO, /* Metadata cache coherency synchronization message */
- NETFS_PAGE_CACHE, /* Page cache invalidation message */
- NETFS_READ_PAGES, /* Read multiple contiguous pages in one go */
- NETFS_RENAME, /* Rename object */
- NETFS_CAPABILITIES, /* Capabilities of the client, for example supported crypto */
- NETFS_LOCK, /* Distributed lock message */
- NETFS_XATTR_SET, /* Set extended attribute */
- NETFS_XATTR_GET, /* Get extended attribute */
-
-@ext - external flags. Used by different commands to specify some extra arguments
- like partial size of the embedded objects or creation flags.
-
-@size - size of the attached data. For NETFS_READ_PAGE and NETFS_READ_PAGES no data is attached,
- but size of the requested data is incorporated here. It does not include size of the command
- header (struct netfs_cmd) itself.
-
-@id - id of the object this command operates on. Each command can use it for own purpose.
-
-@start - start of the object this command operates on. Each command can use it for own purpose.
-
-@csize, @cpad - size and padding size of the (attached if needed) crypto information.
-
-Command specifications.
-
-@NETFS_READDIR
-This command is used to sync content of the remote dir to the client.
-
-@ext - length of the path to object.
-@size - the same.
-@id - local inode number of the directory to read.
-@start - zero.
-
-
-@NETFS_READ_PAGE
-This command is used to read data from remote server.
-Data size does not exceed local page cache size.
-
-@id - inode number.
-@start - first byte offset.
-@size - number of bytes to read plus length of the path to object.
-@ext - object path length.
-
-
-@NETFS_CREATE
-Used to create object.
-It does not require that all directories on top of the object were
-already created, it will create them automatically. Each object has
-associated @netfs_path_entry data structure, which contains creation
-mode (permissions and type) and length of the name as long as name itself.
-
-@start - 0
-@size - size of the all data structures needed to create a path
-@id - local inode number
-@ext - 0
-
-
-@NETFS_REMOVE
-Used to remove object.
-
-@ext - length of the path to object.
-@size - the same.
-@id - local inode number.
-@start - zero.
-
-
-@NETFS_LOOKUP
-Lookup information about object on server.
-
-@ext - length of the path to object.
-@size - the same.
-@id - local inode number of the directory to look object in.
-@start - local inode number of the object to look at.
-
-
-@NETFS_LINK
-Create hard of symlink.
-Command is sent as "object_path|target_path".
-
-@size - size of the above string.
-@id - parent local inode number.
-@start - 1 for symlink, 0 for hardlink.
-@ext - size of the "object_path" above.
-
-
-@NETFS_TRANS
-Transaction header.
-
-@size - incorporates all embedded command sizes including theirs header sizes.
-@start - transaction generation number - unique id used to find transaction.
-@ext - transaction flags. Unused at the moment.
-@id - 0.
-
-
-@NETFS_OPEN
-Open intent for given transaction.
-
-@id - local inode number.
-@start - 0.
-@size - path length to the object.
-@ext - open flags (O_RDWR and so on).
-
-
-@NETFS_INODE_INFO
-Metadata update command.
-It is sent to servers when attributes of the object are changed and received
-when data or metadata were updated. It operates with the following structure:
-
-struct netfs_inode_info
-{
- unsigned int mode;
- unsigned int nlink;
- unsigned int uid;
- unsigned int gid;
- unsigned int blocksize;
- unsigned int padding;
- __u64 ino;
- __u64 blocks;
- __u64 rdev;
- __u64 size;
- __u64 version;
-};
-
-It effectively mirrors stat(2) returned data.
-
-
-@ext - path length to the object.
-@size - the same plus size of the netfs_inode_info structure.
-@id - local inode number.
-@start - 0.
-
-
-@NETFS_PAGE_CACHE
-Command is only received by clients. It contains information about
-page to be marked as not up-to-date.
-
-@id - client's inode number.
-@start - last byte of the page to be invalidated. If it is not equal to
- current inode size, it will be vmtruncated().
-@size - 0
-@ext - 0
-
-
-@NETFS_READ_PAGES
-Used to read multiple contiguous pages in one go.
-
-@start - first byte of the contiguous region to read.
-@size - contains of two fields: lower 8 bits are used to represent page cache shift
- used by client, another 3 bytes are used to get number of pages.
-@id - local inode number.
-@ext - path length to the object.
-
-
-@NETFS_RENAME
-Used to rename object.
-Attached data is formed into following string: "old_path|new_path".
-
-@id - local inode number.
-@start - parent inode number.
-@size - length of the above string.
-@ext - length of the old path part.
-
-
-@NETFS_CAPABILITIES
-Used to exchange crypto capabilities with server.
-If crypto capabilities are not supported by server, then client will disable it
-or fail (if 'crypto_fail_unsupported' mount options was specified).
-
-@id - superblock index. Used to specify crypto information for group of servers.
-@size - size of the attached capabilities structure.
-@start - 0.
-@size - 0.
-@scsize - 0.
-
-@NETFS_LOCK
-Used to send lock request/release messages. Although it sends byte range request
-and is capable of flushing pages based on that, it is not used, since all Linux
-filesystems lock the whole inode.
-
-@id - lock generation number.
-@start - start of the locked range.
-@size - size of the locked range.
-@ext - lock type: read/write. Not used actually. 15'th bit is used to determine,
- if it is lock request (1) or release (0).
-
-@NETFS_XATTR_SET
-@NETFS_XATTR_GET
-Used to set/get extended attributes for given inode.
-@id - attribute generation number or xattr setting type
-@start - size of the attribute (request or attached)
-@size - name length, path len and data size for given attribute
-@ext - path length for given object
diff --git a/Documentation/filesystems/porting b/Documentation/filesystems/porting
deleted file mode 100644
index 7b7b845..0000000
--- a/Documentation/filesystems/porting
+++ /dev/null
@@ -1,624 +0,0 @@
-Changes since 2.5.0:
-
----
-[recommended]
-
-New helpers: sb_bread(), sb_getblk(), sb_find_get_block(), set_bh(),
- sb_set_blocksize() and sb_min_blocksize().
-
-Use them.
-
-(sb_find_get_block() replaces 2.4's get_hash_table())
-
----
-[recommended]
-
-New methods: ->alloc_inode() and ->destroy_inode().
-
-Remove inode->u.foo_inode_i
-Declare
- struct foo_inode_info {
- /* fs-private stuff */
- struct inode vfs_inode;
- };
- static inline struct foo_inode_info *FOO_I(struct inode *inode)
- {
- return list_entry(inode, struct foo_inode_info, vfs_inode);
- }
-
-Use FOO_I(inode) instead of &inode->u.foo_inode_i;
-
-Add foo_alloc_inode() and foo_destroy_inode() - the former should allocate
-foo_inode_info and return the address of ->vfs_inode, the latter should free
-FOO_I(inode) (see in-tree filesystems for examples).
-
-Make them ->alloc_inode and ->destroy_inode in your super_operations.
-
-Keep in mind that now you need explicit initialization of private data
-typically between calling iget_locked() and unlocking the inode.
-
-At some point that will become mandatory.
-
----
-[mandatory]
-
-Change of file_system_type method (->read_super to ->get_sb)
-
-->read_super() is no more. Ditto for DECLARE_FSTYPE and DECLARE_FSTYPE_DEV.
-
-Turn your foo_read_super() into a function that would return 0 in case of
-success and negative number in case of error (-EINVAL unless you have more
-informative error value to report). Call it foo_fill_super(). Now declare
-
-int foo_get_sb(struct file_system_type *fs_type,
- int flags, const char *dev_name, void *data, struct vfsmount *mnt)
-{
- return get_sb_bdev(fs_type, flags, dev_name, data, foo_fill_super,
- mnt);
-}
-
-(or similar with s/bdev/nodev/ or s/bdev/single/, depending on the kind of
-filesystem).
-
-Replace DECLARE_FSTYPE... with explicit initializer and have ->get_sb set as
-foo_get_sb.
-
----
-[mandatory]
-
-Locking change: ->s_vfs_rename_sem is taken only by cross-directory renames.
-Most likely there is no need to change anything, but if you relied on
-global exclusion between renames for some internal purpose - you need to
-change your internal locking. Otherwise exclusion warranties remain the
-same (i.e. parents and victim are locked, etc.).
-
----
-[informational]
-
-Now we have the exclusion between ->lookup() and directory removal (by
-->rmdir() and ->rename()). If you used to need that exclusion and do
-it by internal locking (most of filesystems couldn't care less) - you
-can relax your locking.
-
----
-[mandatory]
-
-->lookup(), ->truncate(), ->create(), ->unlink(), ->mknod(), ->mkdir(),
-->rmdir(), ->link(), ->lseek(), ->symlink(), ->rename()
-and ->readdir() are called without BKL now. Grab it on entry, drop upon return
-- that will guarantee the same locking you used to have. If your method or its
-parts do not need BKL - better yet, now you can shift lock_kernel() and
-unlock_kernel() so that they would protect exactly what needs to be
-protected.
-
----
-[mandatory]
-
-BKL is also moved from around sb operations. BKL should have been shifted into
-individual fs sb_op functions. If you don't need it, remove it.
-
----
-[informational]
-
-check for ->link() target not being a directory is done by callers. Feel
-free to drop it...
-
----
-[informational]
-
-->link() callers hold ->i_mutex on the object we are linking to. Some of your
-problems might be over...
-
----
-[mandatory]
-
-new file_system_type method - kill_sb(superblock). If you are converting
-an existing filesystem, set it according to ->fs_flags:
- FS_REQUIRES_DEV - kill_block_super
- FS_LITTER - kill_litter_super
- neither - kill_anon_super
-FS_LITTER is gone - just remove it from fs_flags.
-
----
-[mandatory]
-
- FS_SINGLE is gone (actually, that had happened back when ->get_sb()
-went in - and hadn't been documented ;-/). Just remove it from fs_flags
-(and see ->get_sb() entry for other actions).
-
----
-[mandatory]
-
-->setattr() is called without BKL now. Caller _always_ holds ->i_mutex, so
-watch for ->i_mutex-grabbing code that might be used by your ->setattr().
-Callers of notify_change() need ->i_mutex now.
-
----
-[recommended]
-
-New super_block field "struct export_operations *s_export_op" for
-explicit support for exporting, e.g. via NFS. The structure is fully
-documented at its declaration in include/linux/fs.h, and in
-Documentation/filesystems/nfs/Exporting.
-
-Briefly it allows for the definition of decode_fh and encode_fh operations
-to encode and decode filehandles, and allows the filesystem to use
-a standard helper function for decode_fh, and provide file-system specific
-support for this helper, particularly get_parent.
-
-It is planned that this will be required for exporting once the code
-settles down a bit.
-
-[mandatory]
-
-s_export_op is now required for exporting a filesystem.
-isofs, ext2, ext3, resierfs, fat
-can be used as examples of very different filesystems.
-
----
-[mandatory]
-
-iget4() and the read_inode2 callback have been superseded by iget5_locked()
-which has the following prototype,
-
- struct inode *iget5_locked(struct super_block *sb, unsigned long ino,
- int (*test)(struct inode *, void *),
- int (*set)(struct inode *, void *),
- void *data);
-
-'test' is an additional function that can be used when the inode
-number is not sufficient to identify the actual file object. 'set'
-should be a non-blocking function that initializes those parts of a
-newly created inode to allow the test function to succeed. 'data' is
-passed as an opaque value to both test and set functions.
-
-When the inode has been created by iget5_locked(), it will be returned with the
-I_NEW flag set and will still be locked. The filesystem then needs to finalize
-the initialization. Once the inode is initialized it must be unlocked by
-calling unlock_new_inode().
-
-The filesystem is responsible for setting (and possibly testing) i_ino
-when appropriate. There is also a simpler iget_locked function that
-just takes the superblock and inode number as arguments and does the
-test and set for you.
-
-e.g.
- inode = iget_locked(sb, ino);
- if (inode->i_state & I_NEW) {
- err = read_inode_from_disk(inode);
- if (err < 0) {
- iget_failed(inode);
- return err;
- }
- unlock_new_inode(inode);
- }
-
-Note that if the process of setting up a new inode fails, then iget_failed()
-should be called on the inode to render it dead, and an appropriate error
-should be passed back to the caller.
-
----
-[recommended]
-
-->getattr() finally getting used. See instances in nfs, minix, etc.
-
----
-[mandatory]
-
-->revalidate() is gone. If your filesystem had it - provide ->getattr()
-and let it call whatever you had as ->revlidate() + (for symlinks that
-had ->revalidate()) add calls in ->follow_link()/->readlink().
-
----
-[mandatory]
-
-->d_parent changes are not protected by BKL anymore. Read access is safe
-if at least one of the following is true:
- * filesystem has no cross-directory rename()
- * we know that parent had been locked (e.g. we are looking at
-->d_parent of ->lookup() argument).
- * we are called from ->rename().
- * the child's ->d_lock is held
-Audit your code and add locking if needed. Notice that any place that is
-not protected by the conditions above is risky even in the old tree - you
-had been relying on BKL and that's prone to screwups. Old tree had quite
-a few holes of that kind - unprotected access to ->d_parent leading to
-anything from oops to silent memory corruption.
-
----
-[mandatory]
-
- FS_NOMOUNT is gone. If you use it - just set SB_NOUSER in flags
-(see rootfs for one kind of solution and bdev/socket/pipe for another).
-
----
-[recommended]
-
- Use bdev_read_only(bdev) instead of is_read_only(kdev). The latter
-is still alive, but only because of the mess in drivers/s390/block/dasd.c.
-As soon as it gets fixed is_read_only() will die.
-
----
-[mandatory]
-
-->permission() is called without BKL now. Grab it on entry, drop upon
-return - that will guarantee the same locking you used to have. If
-your method or its parts do not need BKL - better yet, now you can
-shift lock_kernel() and unlock_kernel() so that they would protect
-exactly what needs to be protected.
-
----
-[mandatory]
-
-->statfs() is now called without BKL held. BKL should have been
-shifted into individual fs sb_op functions where it's not clear that
-it's safe to remove it. If you don't need it, remove it.
-
----
-[mandatory]
-
- is_read_only() is gone; use bdev_read_only() instead.
-
----
-[mandatory]
-
- destroy_buffers() is gone; use invalidate_bdev().
-
----
-[mandatory]
-
- fsync_dev() is gone; use fsync_bdev(). NOTE: lvm breakage is
-deliberate; as soon as struct block_device * is propagated in a reasonable
-way by that code fixing will become trivial; until then nothing can be
-done.
-
-[mandatory]
-
- block truncatation on error exit from ->write_begin, and ->direct_IO
-moved from generic methods (block_write_begin, cont_write_begin,
-nobh_write_begin, blockdev_direct_IO*) to callers. Take a look at
-ext2_write_failed and callers for an example.
-
-[mandatory]
-
- ->truncate is gone. The whole truncate sequence needs to be
-implemented in ->setattr, which is now mandatory for filesystems
-implementing on-disk size changes. Start with a copy of the old inode_setattr
-and vmtruncate, and the reorder the vmtruncate + foofs_vmtruncate sequence to
-be in order of zeroing blocks using block_truncate_page or similar helpers,
-size update and on finally on-disk truncation which should not fail.
-setattr_prepare (which used to be inode_change_ok) now includes the size checks
-for ATTR_SIZE and must be called in the beginning of ->setattr unconditionally.
-
-[mandatory]
-
- ->clear_inode() and ->delete_inode() are gone; ->evict_inode() should
-be used instead. It gets called whenever the inode is evicted, whether it has
-remaining links or not. Caller does *not* evict the pagecache or inode-associated
-metadata buffers; the method has to use truncate_inode_pages_final() to get rid
-of those. Caller makes sure async writeback cannot be running for the inode while
-(or after) ->evict_inode() is called.
-
- ->drop_inode() returns int now; it's called on final iput() with
-inode->i_lock held and it returns true if filesystems wants the inode to be
-dropped. As before, generic_drop_inode() is still the default and it's been
-updated appropriately. generic_delete_inode() is also alive and it consists
-simply of return 1. Note that all actual eviction work is done by caller after
-->drop_inode() returns.
-
- As before, clear_inode() must be called exactly once on each call of
-->evict_inode() (as it used to be for each call of ->delete_inode()). Unlike
-before, if you are using inode-associated metadata buffers (i.e.
-mark_buffer_dirty_inode()), it's your responsibility to call
-invalidate_inode_buffers() before clear_inode().
-
- NOTE: checking i_nlink in the beginning of ->write_inode() and bailing out
-if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput()
-may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly
-free the on-disk inode, you may end up doing that while ->write_inode() is writing
-to it.
-
----
-[mandatory]
-
- .d_delete() now only advises the dcache as to whether or not to cache
-unreferenced dentries, and is now only called when the dentry refcount goes to
-0. Even on 0 refcount transition, it must be able to tolerate being called 0,
-1, or more times (eg. constant, idempotent).
-
----
-[mandatory]
-
- .d_compare() calling convention and locking rules are significantly
-changed. Read updated documentation in Documentation/filesystems/vfs.txt (and
-look at examples of other filesystems) for guidance.
-
----
-[mandatory]
-
- .d_hash() calling convention and locking rules are significantly
-changed. Read updated documentation in Documentation/filesystems/vfs.txt (and
-look at examples of other filesystems) for guidance.
-
----
-[mandatory]
- dcache_lock is gone, replaced by fine grained locks. See fs/dcache.c
-for details of what locks to replace dcache_lock with in order to protect
-particular things. Most of the time, a filesystem only needs ->d_lock, which
-protects *all* the dcache state of a given dentry.
-
---
-[mandatory]
-
- Filesystems must RCU-free their inodes, if they can have been accessed
-via rcu-walk path walk (basically, if the file can have had a path name in the
-vfs namespace).
-
- Even though i_dentry and i_rcu share storage in a union, we will
-initialize the former in inode_init_always(), so just leave it alone in
-the callback. It used to be necessary to clean it there, but not anymore
-(starting at 3.2).
-
---
-[recommended]
- vfs now tries to do path walking in "rcu-walk mode", which avoids
-atomic operations and scalability hazards on dentries and inodes (see
-Documentation/filesystems/path-lookup.txt). d_hash and d_compare changes
-(above) are examples of the changes required to support this. For more complex
-filesystem callbacks, the vfs drops out of rcu-walk mode before the fs call, so
-no changes are required to the filesystem. However, this is costly and loses
-the benefits of rcu-walk mode. We will begin to add filesystem callbacks that
-are rcu-walk aware, shown below. Filesystems should take advantage of this
-where possible.
-
---
-[mandatory]
- d_revalidate is a callback that is made on every path element (if
-the filesystem provides it), which requires dropping out of rcu-walk mode. This
-may now be called in rcu-walk mode (nd->flags & LOOKUP_RCU). -ECHILD should be
-returned if the filesystem cannot handle rcu-walk. See
-Documentation/filesystems/vfs.txt for more details.
-
- permission is an inode permission check that is called on many or all
-directory inodes on the way down a path walk (to check for exec permission). It
-must now be rcu-walk aware (mask & MAY_NOT_BLOCK). See
-Documentation/filesystems/vfs.txt for more details.
-
---
-[mandatory]
- In ->fallocate() you must check the mode option passed in. If your
-filesystem does not support hole punching (deallocating space in the middle of a
-file) you must return -EOPNOTSUPP if FALLOC_FL_PUNCH_HOLE is set in mode.
-Currently you can only have FALLOC_FL_PUNCH_HOLE with FALLOC_FL_KEEP_SIZE set,
-so the i_size should not change when hole punching, even when puching the end of
-a file off.
-
---
-[mandatory]
- ->get_sb() is gone. Switch to use of ->mount(). Typically it's just
-a matter of switching from calling get_sb_... to mount_... and changing the
-function type. If you were doing it manually, just switch from setting ->mnt_root
-to some pointer to returning that pointer. On errors return ERR_PTR(...).
-
---
-[mandatory]
- ->permission() and generic_permission()have lost flags
-argument; instead of passing IPERM_FLAG_RCU we add MAY_NOT_BLOCK into mask.
- generic_permission() has also lost the check_acl argument; ACL checking
-has been taken to VFS and filesystems need to provide a non-NULL ->i_op->get_acl
-to read an ACL from disk.
-
---
-[mandatory]
- If you implement your own ->llseek() you must handle SEEK_HOLE and
-SEEK_DATA. You can hanle this by returning -EINVAL, but it would be nicer to
-support it in some way. The generic handler assumes that the entire file is
-data and there is a virtual hole at the end of the file. So if the provided
-offset is less than i_size and SEEK_DATA is specified, return the same offset.
-If the above is true for the offset and you are given SEEK_HOLE, return the end
-of the file. If the offset is i_size or greater return -ENXIO in either case.
-
-[mandatory]
- If you have your own ->fsync() you must make sure to call
-filemap_write_and_wait_range() so that all dirty pages are synced out properly.
-You must also keep in mind that ->fsync() is not called with i_mutex held
-anymore, so if you require i_mutex locking you must make sure to take it and
-release it yourself.
-
---
-[mandatory]
- d_alloc_root() is gone, along with a lot of bugs caused by code
-misusing it. Replacement: d_make_root(inode). The difference is,
-d_make_root() drops the reference to inode if dentry allocation fails.
-
---
-[mandatory]
- The witch is dead! Well, 2/3 of it, anyway. ->d_revalidate() and
-->lookup() do *not* take struct nameidata anymore; just the flags.
---
-[mandatory]
- ->create() doesn't take struct nameidata *; unlike the previous
-two, it gets "is it an O_EXCL or equivalent?" boolean argument. Note that
-local filesystems can ignore tha argument - they are guaranteed that the
-object doesn't exist. It's remote/distributed ones that might care...
---
-[mandatory]
- FS_REVAL_DOT is gone; if you used to have it, add ->d_weak_revalidate()
-in your dentry operations instead.
---
-[mandatory]
- vfs_readdir() is gone; switch to iterate_dir() instead
---
-[mandatory]
- ->readdir() is gone now; switch to ->iterate()
-[mandatory]
- vfs_follow_link has been removed. Filesystems must use nd_set_link
- from ->follow_link for normal symlinks, or nd_jump_link for magic
- /proc/<pid> style links.
---
-[mandatory]
- iget5_locked()/ilookup5()/ilookup5_nowait() test() callback used to be
- called with both ->i_lock and inode_hash_lock held; the former is *not*
- taken anymore, so verify that your callbacks do not rely on it (none
- of the in-tree instances did). inode_hash_lock is still held,
- of course, so they are still serialized wrt removal from inode hash,
- as well as wrt set() callback of iget5_locked().
---
-[mandatory]
- d_materialise_unique() is gone; d_splice_alias() does everything you
- need now. Remember that they have opposite orders of arguments ;-/
---
-[mandatory]
- f_dentry is gone; use f_path.dentry, or, better yet, see if you can avoid
- it entirely.
---
-[mandatory]
- never call ->read() and ->write() directly; use __vfs_{read,write} or
- wrappers; instead of checking for ->write or ->read being NULL, look for
- FMODE_CAN_{WRITE,READ} in file->f_mode.
---
-[mandatory]
- do _not_ use new_sync_{read,write} for ->read/->write; leave it NULL
- instead.
---
-[mandatory]
- ->aio_read/->aio_write are gone. Use ->read_iter/->write_iter.
----
-[recommended]
- for embedded ("fast") symlinks just set inode->i_link to wherever the
- symlink body is and use simple_follow_link() as ->follow_link().
---
-[mandatory]
- calling conventions for ->follow_link() have changed. Instead of returning
- cookie and using nd_set_link() to store the body to traverse, we return
- the body to traverse and store the cookie using explicit void ** argument.
- nameidata isn't passed at all - nd_jump_link() doesn't need it and
- nd_[gs]et_link() is gone.
---
-[mandatory]
- calling conventions for ->put_link() have changed. It gets inode instead of
- dentry, it does not get nameidata at all and it gets called only when cookie
- is non-NULL. Note that link body isn't available anymore, so if you need it,
- store it as cookie.
---
-[mandatory]
- any symlink that might use page_follow_link_light/page_put_link() must
- have inode_nohighmem(inode) called before anything might start playing with
- its pagecache. No highmem pages should end up in the pagecache of such
- symlinks. That includes any preseeding that might be done during symlink
- creation. __page_symlink() will honour the mapping gfp flags, so once
- you've done inode_nohighmem() it's safe to use, but if you allocate and
- insert the page manually, make sure to use the right gfp flags.
---
-[mandatory]
- ->follow_link() is replaced with ->get_link(); same API, except that
- * ->get_link() gets inode as a separate argument
- * ->get_link() may be called in RCU mode - in that case NULL
- dentry is passed
---
-[mandatory]
- ->get_link() gets struct delayed_call *done now, and should do
- set_delayed_call() where it used to set *cookie.
- ->put_link() is gone - just give the destructor to set_delayed_call()
- in ->get_link().
---
-[mandatory]
- ->getxattr() and xattr_handler.get() get dentry and inode passed separately.
- dentry might be yet to be attached to inode, so do _not_ use its ->d_inode
- in the instances. Rationale: !@#!@# security_d_instantiate() needs to be
- called before we attach dentry to inode.
---
-[mandatory]
- symlinks are no longer the only inodes that do *not* have i_bdev/i_cdev/
- i_pipe/i_link union zeroed out at inode eviction. As the result, you can't
- assume that non-NULL value in ->i_nlink at ->destroy_inode() implies that
- it's a symlink. Checking ->i_mode is really needed now. In-tree we had
- to fix shmem_destroy_callback() that used to take that kind of shortcut;
- watch out, since that shortcut is no longer valid.
---
-[mandatory]
- ->i_mutex is replaced with ->i_rwsem now. inode_lock() et.al. work as
- they used to - they just take it exclusive. However, ->lookup() may be
- called with parent locked shared. Its instances must not
- * use d_instantiate) and d_rehash() separately - use d_add() or
- d_splice_alias() instead.
- * use d_rehash() alone - call d_add(new_dentry, NULL) instead.
- * in the unlikely case when (read-only) access to filesystem
- data structures needs exclusion for some reason, arrange it
- yourself. None of the in-tree filesystems needed that.
- * rely on ->d_parent and ->d_name not changing after dentry has
- been fed to d_add() or d_splice_alias(). Again, none of the
- in-tree instances relied upon that.
- We are guaranteed that lookups of the same name in the same directory
- will not happen in parallel ("same" in the sense of your ->d_compare()).
- Lookups on different names in the same directory can and do happen in
- parallel now.
---
-[recommended]
- ->iterate_shared() is added; it's a parallel variant of ->iterate().
- Exclusion on struct file level is still provided (as well as that
- between it and lseek on the same struct file), but if your directory
- has been opened several times, you can get these called in parallel.
- Exclusion between that method and all directory-modifying ones is
- still provided, of course.
-
- Often enough ->iterate() can serve as ->iterate_shared() without any
- changes - it is a read-only operation, after all. If you have any
- per-inode or per-dentry in-core data structures modified by ->iterate(),
- you might need something to serialize the access to them. If you
- do dcache pre-seeding, you'll need to switch to d_alloc_parallel() for
- that; look for in-tree examples.
-
- Old method is only used if the new one is absent; eventually it will
- be removed. Switch while you still can; the old one won't stay.
---
-[mandatory]
- ->atomic_open() calls without O_CREAT may happen in parallel.
---
-[mandatory]
- ->setxattr() and xattr_handler.set() get dentry and inode passed separately.
- dentry might be yet to be attached to inode, so do _not_ use its ->d_inode
- in the instances. Rationale: !@#!@# security_d_instantiate() needs to be
- called before we attach dentry to inode and !@#!@##!@$!$#!@#$!@$!@$ smack
- ->d_instantiate() uses not just ->getxattr() but ->setxattr() as well.
---
-[mandatory]
- ->d_compare() doesn't get parent as a separate argument anymore. If you
- used it for finding the struct super_block involved, dentry->d_sb will
- work just as well; if it's something more complicated, use dentry->d_parent.
- Just be careful not to assume that fetching it more than once will yield
- the same value - in RCU mode it could change under you.
---
-[mandatory]
- ->rename() has an added flags argument. Any flags not handled by the
- filesystem should result in EINVAL being returned.
---
-[recommended]
- ->readlink is optional for symlinks. Don't set, unless filesystem needs
- to fake something for readlink(2).
---
-[mandatory]
- ->getattr() is now passed a struct path rather than a vfsmount and
- dentry separately, and it now has request_mask and query_flags arguments
- to specify the fields and sync type requested by statx. Filesystems not
- supporting any statx-specific features may ignore the new arguments.
---
-[mandatory]
- ->atomic_open() calling conventions have changed. Gone is int *opened,
- along with FILE_OPENED/FILE_CREATED. In place of those we have
- FMODE_OPENED/FMODE_CREATED, set in file->f_mode. Additionally, return
- value for 'called finish_no_open(), open it yourself' case has become
- 0, not 1. Since finish_no_open() itself is returning 0 now, that part
- does not need any changes in ->atomic_open() instances.
---
-[mandatory]
- alloc_file() has become static now; two wrappers are to be used instead.
- alloc_file_pseudo(inode, vfsmount, name, flags, ops) is for the cases
- when dentry needs to be created; that's the majority of old alloc_file()
- users. Calling conventions: on success a reference to new struct file
- is returned and callers reference to inode is subsumed by that. On
- failure, ERR_PTR() is returned and no caller's references are affected,
- so the caller needs to drop the inode reference it held.
- alloc_file_clone(file, flags, ops) does not affect any caller's references.
- On success you get a new struct file sharing the mount/dentry with the
- original, on failure - ERR_PTR().
diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst
new file mode 100644
index 0000000..f185060
--- /dev/null
+++ b/Documentation/filesystems/porting.rst
@@ -0,0 +1,852 @@
+====================
+Changes since 2.5.0:
+====================
+
+---
+
+**recommended**
+
+New helpers: sb_bread(), sb_getblk(), sb_find_get_block(), set_bh(),
+sb_set_blocksize() and sb_min_blocksize().
+
+Use them.
+
+(sb_find_get_block() replaces 2.4's get_hash_table())
+
+---
+
+**recommended**
+
+New methods: ->alloc_inode() and ->destroy_inode().
+
+Remove inode->u.foo_inode_i
+
+Declare::
+
+ struct foo_inode_info {
+ /* fs-private stuff */
+ struct inode vfs_inode;
+ };
+ static inline struct foo_inode_info *FOO_I(struct inode *inode)
+ {
+ return list_entry(inode, struct foo_inode_info, vfs_inode);
+ }
+
+Use FOO_I(inode) instead of &inode->u.foo_inode_i;
+
+Add foo_alloc_inode() and foo_destroy_inode() - the former should allocate
+foo_inode_info and return the address of ->vfs_inode, the latter should free
+FOO_I(inode) (see in-tree filesystems for examples).
+
+Make them ->alloc_inode and ->destroy_inode in your super_operations.
+
+Keep in mind that now you need explicit initialization of private data
+typically between calling iget_locked() and unlocking the inode.
+
+At some point that will become mandatory.
+
+---
+
+**mandatory**
+
+Change of file_system_type method (->read_super to ->get_sb)
+
+->read_super() is no more. Ditto for DECLARE_FSTYPE and DECLARE_FSTYPE_DEV.
+
+Turn your foo_read_super() into a function that would return 0 in case of
+success and negative number in case of error (-EINVAL unless you have more
+informative error value to report). Call it foo_fill_super(). Now declare::
+
+ int foo_get_sb(struct file_system_type *fs_type,
+ int flags, const char *dev_name, void *data, struct vfsmount *mnt)
+ {
+ return get_sb_bdev(fs_type, flags, dev_name, data, foo_fill_super,
+ mnt);
+ }
+
+(or similar with s/bdev/nodev/ or s/bdev/single/, depending on the kind of
+filesystem).
+
+Replace DECLARE_FSTYPE... with explicit initializer and have ->get_sb set as
+foo_get_sb.
+
+---
+
+**mandatory**
+
+Locking change: ->s_vfs_rename_sem is taken only by cross-directory renames.
+Most likely there is no need to change anything, but if you relied on
+global exclusion between renames for some internal purpose - you need to
+change your internal locking. Otherwise exclusion warranties remain the
+same (i.e. parents and victim are locked, etc.).
+
+---
+
+**informational**
+
+Now we have the exclusion between ->lookup() and directory removal (by
+->rmdir() and ->rename()). If you used to need that exclusion and do
+it by internal locking (most of filesystems couldn't care less) - you
+can relax your locking.
+
+---
+
+**mandatory**
+
+->lookup(), ->truncate(), ->create(), ->unlink(), ->mknod(), ->mkdir(),
+->rmdir(), ->link(), ->lseek(), ->symlink(), ->rename()
+and ->readdir() are called without BKL now. Grab it on entry, drop upon return
+- that will guarantee the same locking you used to have. If your method or its
+parts do not need BKL - better yet, now you can shift lock_kernel() and
+unlock_kernel() so that they would protect exactly what needs to be
+protected.
+
+---
+
+**mandatory**
+
+BKL is also moved from around sb operations. BKL should have been shifted into
+individual fs sb_op functions. If you don't need it, remove it.
+
+---
+
+**informational**
+
+check for ->link() target not being a directory is done by callers. Feel
+free to drop it...
+
+---
+
+**informational**
+
+->link() callers hold ->i_mutex on the object we are linking to. Some of your
+problems might be over...
+
+---
+
+**mandatory**
+
+new file_system_type method - kill_sb(superblock). If you are converting
+an existing filesystem, set it according to ->fs_flags::
+
+ FS_REQUIRES_DEV - kill_block_super
+ FS_LITTER - kill_litter_super
+ neither - kill_anon_super
+
+FS_LITTER is gone - just remove it from fs_flags.
+
+---
+
+**mandatory**
+
+FS_SINGLE is gone (actually, that had happened back when ->get_sb()
+went in - and hadn't been documented ;-/). Just remove it from fs_flags
+(and see ->get_sb() entry for other actions).
+
+---
+
+**mandatory**
+
+->setattr() is called without BKL now. Caller _always_ holds ->i_mutex, so
+watch for ->i_mutex-grabbing code that might be used by your ->setattr().
+Callers of notify_change() need ->i_mutex now.
+
+---
+
+**recommended**
+
+New super_block field ``struct export_operations *s_export_op`` for
+explicit support for exporting, e.g. via NFS. The structure is fully
+documented at its declaration in include/linux/fs.h, and in
+Documentation/filesystems/nfs/exporting.rst.
+
+Briefly it allows for the definition of decode_fh and encode_fh operations
+to encode and decode filehandles, and allows the filesystem to use
+a standard helper function for decode_fh, and provide file-system specific
+support for this helper, particularly get_parent.
+
+It is planned that this will be required for exporting once the code
+settles down a bit.
+
+**mandatory**
+
+s_export_op is now required for exporting a filesystem.
+isofs, ext2, ext3, resierfs, fat
+can be used as examples of very different filesystems.
+
+---
+
+**mandatory**
+
+iget4() and the read_inode2 callback have been superseded by iget5_locked()
+which has the following prototype::
+
+ struct inode *iget5_locked(struct super_block *sb, unsigned long ino,
+ int (*test)(struct inode *, void *),
+ int (*set)(struct inode *, void *),
+ void *data);
+
+'test' is an additional function that can be used when the inode
+number is not sufficient to identify the actual file object. 'set'
+should be a non-blocking function that initializes those parts of a
+newly created inode to allow the test function to succeed. 'data' is
+passed as an opaque value to both test and set functions.
+
+When the inode has been created by iget5_locked(), it will be returned with the
+I_NEW flag set and will still be locked. The filesystem then needs to finalize
+the initialization. Once the inode is initialized it must be unlocked by
+calling unlock_new_inode().
+
+The filesystem is responsible for setting (and possibly testing) i_ino
+when appropriate. There is also a simpler iget_locked function that
+just takes the superblock and inode number as arguments and does the
+test and set for you.
+
+e.g.::
+
+ inode = iget_locked(sb, ino);
+ if (inode->i_state & I_NEW) {
+ err = read_inode_from_disk(inode);
+ if (err < 0) {
+ iget_failed(inode);
+ return err;
+ }
+ unlock_new_inode(inode);
+ }
+
+Note that if the process of setting up a new inode fails, then iget_failed()
+should be called on the inode to render it dead, and an appropriate error
+should be passed back to the caller.
+
+---
+
+**recommended**
+
+->getattr() finally getting used. See instances in nfs, minix, etc.
+
+---
+
+**mandatory**
+
+->revalidate() is gone. If your filesystem had it - provide ->getattr()
+and let it call whatever you had as ->revlidate() + (for symlinks that
+had ->revalidate()) add calls in ->follow_link()/->readlink().
+
+---
+
+**mandatory**
+
+->d_parent changes are not protected by BKL anymore. Read access is safe
+if at least one of the following is true:
+
+ * filesystem has no cross-directory rename()
+ * we know that parent had been locked (e.g. we are looking at
+ ->d_parent of ->lookup() argument).
+ * we are called from ->rename().
+ * the child's ->d_lock is held
+
+Audit your code and add locking if needed. Notice that any place that is
+not protected by the conditions above is risky even in the old tree - you
+had been relying on BKL and that's prone to screwups. Old tree had quite
+a few holes of that kind - unprotected access to ->d_parent leading to
+anything from oops to silent memory corruption.
+
+---
+
+**mandatory**
+
+FS_NOMOUNT is gone. If you use it - just set SB_NOUSER in flags
+(see rootfs for one kind of solution and bdev/socket/pipe for another).
+
+---
+
+**recommended**
+
+Use bdev_read_only(bdev) instead of is_read_only(kdev). The latter
+is still alive, but only because of the mess in drivers/s390/block/dasd.c.
+As soon as it gets fixed is_read_only() will die.
+
+---
+
+**mandatory**
+
+->permission() is called without BKL now. Grab it on entry, drop upon
+return - that will guarantee the same locking you used to have. If
+your method or its parts do not need BKL - better yet, now you can
+shift lock_kernel() and unlock_kernel() so that they would protect
+exactly what needs to be protected.
+
+---
+
+**mandatory**
+
+->statfs() is now called without BKL held. BKL should have been
+shifted into individual fs sb_op functions where it's not clear that
+it's safe to remove it. If you don't need it, remove it.
+
+---
+
+**mandatory**
+
+is_read_only() is gone; use bdev_read_only() instead.
+
+---
+
+**mandatory**
+
+destroy_buffers() is gone; use invalidate_bdev().
+
+---
+
+**mandatory**
+
+fsync_dev() is gone; use fsync_bdev(). NOTE: lvm breakage is
+deliberate; as soon as struct block_device * is propagated in a reasonable
+way by that code fixing will become trivial; until then nothing can be
+done.
+
+**mandatory**
+
+block truncatation on error exit from ->write_begin, and ->direct_IO
+moved from generic methods (block_write_begin, cont_write_begin,
+nobh_write_begin, blockdev_direct_IO*) to callers. Take a look at
+ext2_write_failed and callers for an example.
+
+**mandatory**
+
+->truncate is gone. The whole truncate sequence needs to be
+implemented in ->setattr, which is now mandatory for filesystems
+implementing on-disk size changes. Start with a copy of the old inode_setattr
+and vmtruncate, and the reorder the vmtruncate + foofs_vmtruncate sequence to
+be in order of zeroing blocks using block_truncate_page or similar helpers,
+size update and on finally on-disk truncation which should not fail.
+setattr_prepare (which used to be inode_change_ok) now includes the size checks
+for ATTR_SIZE and must be called in the beginning of ->setattr unconditionally.
+
+**mandatory**
+
+->clear_inode() and ->delete_inode() are gone; ->evict_inode() should
+be used instead. It gets called whenever the inode is evicted, whether it has
+remaining links or not. Caller does *not* evict the pagecache or inode-associated
+metadata buffers; the method has to use truncate_inode_pages_final() to get rid
+of those. Caller makes sure async writeback cannot be running for the inode while
+(or after) ->evict_inode() is called.
+
+->drop_inode() returns int now; it's called on final iput() with
+inode->i_lock held and it returns true if filesystems wants the inode to be
+dropped. As before, generic_drop_inode() is still the default and it's been
+updated appropriately. generic_delete_inode() is also alive and it consists
+simply of return 1. Note that all actual eviction work is done by caller after
+->drop_inode() returns.
+
+As before, clear_inode() must be called exactly once on each call of
+->evict_inode() (as it used to be for each call of ->delete_inode()). Unlike
+before, if you are using inode-associated metadata buffers (i.e.
+mark_buffer_dirty_inode()), it's your responsibility to call
+invalidate_inode_buffers() before clear_inode().
+
+NOTE: checking i_nlink in the beginning of ->write_inode() and bailing out
+if it's zero is not *and* *never* *had* *been* enough. Final unlink() and iput()
+may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly
+free the on-disk inode, you may end up doing that while ->write_inode() is writing
+to it.
+
+---
+
+**mandatory**
+
+.d_delete() now only advises the dcache as to whether or not to cache
+unreferenced dentries, and is now only called when the dentry refcount goes to
+0. Even on 0 refcount transition, it must be able to tolerate being called 0,
+1, or more times (eg. constant, idempotent).
+
+---
+
+**mandatory**
+
+.d_compare() calling convention and locking rules are significantly
+changed. Read updated documentation in Documentation/filesystems/vfs.rst (and
+look at examples of other filesystems) for guidance.
+
+---
+
+**mandatory**
+
+.d_hash() calling convention and locking rules are significantly
+changed. Read updated documentation in Documentation/filesystems/vfs.rst (and
+look at examples of other filesystems) for guidance.
+
+---
+
+**mandatory**
+
+dcache_lock is gone, replaced by fine grained locks. See fs/dcache.c
+for details of what locks to replace dcache_lock with in order to protect
+particular things. Most of the time, a filesystem only needs ->d_lock, which
+protects *all* the dcache state of a given dentry.
+
+---
+
+**mandatory**
+
+Filesystems must RCU-free their inodes, if they can have been accessed
+via rcu-walk path walk (basically, if the file can have had a path name in the
+vfs namespace).
+
+Even though i_dentry and i_rcu share storage in a union, we will
+initialize the former in inode_init_always(), so just leave it alone in
+the callback. It used to be necessary to clean it there, but not anymore
+(starting at 3.2).
+
+---
+
+**recommended**
+
+vfs now tries to do path walking in "rcu-walk mode", which avoids
+atomic operations and scalability hazards on dentries and inodes (see
+Documentation/filesystems/path-lookup.txt). d_hash and d_compare changes
+(above) are examples of the changes required to support this. For more complex
+filesystem callbacks, the vfs drops out of rcu-walk mode before the fs call, so
+no changes are required to the filesystem. However, this is costly and loses
+the benefits of rcu-walk mode. We will begin to add filesystem callbacks that
+are rcu-walk aware, shown below. Filesystems should take advantage of this
+where possible.
+
+---
+
+**mandatory**
+
+d_revalidate is a callback that is made on every path element (if
+the filesystem provides it), which requires dropping out of rcu-walk mode. This
+may now be called in rcu-walk mode (nd->flags & LOOKUP_RCU). -ECHILD should be
+returned if the filesystem cannot handle rcu-walk. See
+Documentation/filesystems/vfs.rst for more details.
+
+permission is an inode permission check that is called on many or all
+directory inodes on the way down a path walk (to check for exec permission). It
+must now be rcu-walk aware (mask & MAY_NOT_BLOCK). See
+Documentation/filesystems/vfs.rst for more details.
+
+---
+
+**mandatory**
+
+In ->fallocate() you must check the mode option passed in. If your
+filesystem does not support hole punching (deallocating space in the middle of a
+file) you must return -EOPNOTSUPP if FALLOC_FL_PUNCH_HOLE is set in mode.
+Currently you can only have FALLOC_FL_PUNCH_HOLE with FALLOC_FL_KEEP_SIZE set,
+so the i_size should not change when hole punching, even when puching the end of
+a file off.
+
+---
+
+**mandatory**
+
+->get_sb() is gone. Switch to use of ->mount(). Typically it's just
+a matter of switching from calling ``get_sb_``... to ``mount_``... and changing
+the function type. If you were doing it manually, just switch from setting
+->mnt_root to some pointer to returning that pointer. On errors return
+ERR_PTR(...).
+
+---
+
+**mandatory**
+
+->permission() and generic_permission()have lost flags
+argument; instead of passing IPERM_FLAG_RCU we add MAY_NOT_BLOCK into mask.
+
+generic_permission() has also lost the check_acl argument; ACL checking
+has been taken to VFS and filesystems need to provide a non-NULL ->i_op->get_acl
+to read an ACL from disk.
+
+---
+
+**mandatory**
+
+If you implement your own ->llseek() you must handle SEEK_HOLE and
+SEEK_DATA. You can hanle this by returning -EINVAL, but it would be nicer to
+support it in some way. The generic handler assumes that the entire file is
+data and there is a virtual hole at the end of the file. So if the provided
+offset is less than i_size and SEEK_DATA is specified, return the same offset.
+If the above is true for the offset and you are given SEEK_HOLE, return the end
+of the file. If the offset is i_size or greater return -ENXIO in either case.
+
+**mandatory**
+
+If you have your own ->fsync() you must make sure to call
+filemap_write_and_wait_range() so that all dirty pages are synced out properly.
+You must also keep in mind that ->fsync() is not called with i_mutex held
+anymore, so if you require i_mutex locking you must make sure to take it and
+release it yourself.
+
+---
+
+**mandatory**
+
+d_alloc_root() is gone, along with a lot of bugs caused by code
+misusing it. Replacement: d_make_root(inode). On success d_make_root(inode)
+allocates and returns a new dentry instantiated with the passed in inode.
+On failure NULL is returned and the passed in inode is dropped so the reference
+to inode is consumed in all cases and failure handling need not do any cleanup
+for the inode. If d_make_root(inode) is passed a NULL inode it returns NULL
+and also requires no further error handling. Typical usage is::
+
+ inode = foofs_new_inode(....);
+ s->s_root = d_make_root(inode);
+ if (!s->s_root)
+ /* Nothing needed for the inode cleanup */
+ return -ENOMEM;
+ ...
+
+---
+
+**mandatory**
+
+The witch is dead! Well, 2/3 of it, anyway. ->d_revalidate() and
+->lookup() do *not* take struct nameidata anymore; just the flags.
+
+---
+
+**mandatory**
+
+->create() doesn't take ``struct nameidata *``; unlike the previous
+two, it gets "is it an O_EXCL or equivalent?" boolean argument. Note that
+local filesystems can ignore tha argument - they are guaranteed that the
+object doesn't exist. It's remote/distributed ones that might care...
+
+---
+
+**mandatory**
+
+FS_REVAL_DOT is gone; if you used to have it, add ->d_weak_revalidate()
+in your dentry operations instead.
+
+---
+
+**mandatory**
+
+vfs_readdir() is gone; switch to iterate_dir() instead
+
+---
+
+**mandatory**
+
+->readdir() is gone now; switch to ->iterate()
+
+**mandatory**
+
+vfs_follow_link has been removed. Filesystems must use nd_set_link
+from ->follow_link for normal symlinks, or nd_jump_link for magic
+/proc/<pid> style links.
+
+---
+
+**mandatory**
+
+iget5_locked()/ilookup5()/ilookup5_nowait() test() callback used to be
+called with both ->i_lock and inode_hash_lock held; the former is *not*
+taken anymore, so verify that your callbacks do not rely on it (none
+of the in-tree instances did). inode_hash_lock is still held,
+of course, so they are still serialized wrt removal from inode hash,
+as well as wrt set() callback of iget5_locked().
+
+---
+
+**mandatory**
+
+d_materialise_unique() is gone; d_splice_alias() does everything you
+need now. Remember that they have opposite orders of arguments ;-/
+
+---
+
+**mandatory**
+
+f_dentry is gone; use f_path.dentry, or, better yet, see if you can avoid
+it entirely.
+
+---
+
+**mandatory**
+
+never call ->read() and ->write() directly; use __vfs_{read,write} or
+wrappers; instead of checking for ->write or ->read being NULL, look for
+FMODE_CAN_{WRITE,READ} in file->f_mode.
+
+---
+
+**mandatory**
+
+do _not_ use new_sync_{read,write} for ->read/->write; leave it NULL
+instead.
+
+---
+
+**mandatory**
+ ->aio_read/->aio_write are gone. Use ->read_iter/->write_iter.
+
+---
+
+**recommended**
+
+for embedded ("fast") symlinks just set inode->i_link to wherever the
+symlink body is and use simple_follow_link() as ->follow_link().
+
+---
+
+**mandatory**
+
+calling conventions for ->follow_link() have changed. Instead of returning
+cookie and using nd_set_link() to store the body to traverse, we return
+the body to traverse and store the cookie using explicit void ** argument.
+nameidata isn't passed at all - nd_jump_link() doesn't need it and
+nd_[gs]et_link() is gone.
+
+---
+
+**mandatory**
+
+calling conventions for ->put_link() have changed. It gets inode instead of
+dentry, it does not get nameidata at all and it gets called only when cookie
+is non-NULL. Note that link body isn't available anymore, so if you need it,
+store it as cookie.
+
+---
+
+**mandatory**
+
+any symlink that might use page_follow_link_light/page_put_link() must
+have inode_nohighmem(inode) called before anything might start playing with
+its pagecache. No highmem pages should end up in the pagecache of such
+symlinks. That includes any preseeding that might be done during symlink
+creation. __page_symlink() will honour the mapping gfp flags, so once
+you've done inode_nohighmem() it's safe to use, but if you allocate and
+insert the page manually, make sure to use the right gfp flags.
+
+---
+
+**mandatory**
+
+->follow_link() is replaced with ->get_link(); same API, except that
+
+ * ->get_link() gets inode as a separate argument
+ * ->get_link() may be called in RCU mode - in that case NULL
+ dentry is passed
+
+---
+
+**mandatory**
+
+->get_link() gets struct delayed_call ``*done`` now, and should do
+set_delayed_call() where it used to set ``*cookie``.
+
+->put_link() is gone - just give the destructor to set_delayed_call()
+in ->get_link().
+
+---
+
+**mandatory**
+
+->getxattr() and xattr_handler.get() get dentry and inode passed separately.
+dentry might be yet to be attached to inode, so do _not_ use its ->d_inode
+in the instances. Rationale: !@#!@# security_d_instantiate() needs to be
+called before we attach dentry to inode.
+
+---
+
+**mandatory**
+
+symlinks are no longer the only inodes that do *not* have i_bdev/i_cdev/
+i_pipe/i_link union zeroed out at inode eviction. As the result, you can't
+assume that non-NULL value in ->i_nlink at ->destroy_inode() implies that
+it's a symlink. Checking ->i_mode is really needed now. In-tree we had
+to fix shmem_destroy_callback() that used to take that kind of shortcut;
+watch out, since that shortcut is no longer valid.
+
+---
+
+**mandatory**
+
+->i_mutex is replaced with ->i_rwsem now. inode_lock() et.al. work as
+they used to - they just take it exclusive. However, ->lookup() may be
+called with parent locked shared. Its instances must not
+
+ * use d_instantiate) and d_rehash() separately - use d_add() or
+ d_splice_alias() instead.
+ * use d_rehash() alone - call d_add(new_dentry, NULL) instead.
+ * in the unlikely case when (read-only) access to filesystem
+ data structures needs exclusion for some reason, arrange it
+ yourself. None of the in-tree filesystems needed that.
+ * rely on ->d_parent and ->d_name not changing after dentry has
+ been fed to d_add() or d_splice_alias(). Again, none of the
+ in-tree instances relied upon that.
+
+We are guaranteed that lookups of the same name in the same directory
+will not happen in parallel ("same" in the sense of your ->d_compare()).
+Lookups on different names in the same directory can and do happen in
+parallel now.
+
+---
+
+**recommended**
+
+->iterate_shared() is added; it's a parallel variant of ->iterate().
+Exclusion on struct file level is still provided (as well as that
+between it and lseek on the same struct file), but if your directory
+has been opened several times, you can get these called in parallel.
+Exclusion between that method and all directory-modifying ones is
+still provided, of course.
+
+Often enough ->iterate() can serve as ->iterate_shared() without any
+changes - it is a read-only operation, after all. If you have any
+per-inode or per-dentry in-core data structures modified by ->iterate(),
+you might need something to serialize the access to them. If you
+do dcache pre-seeding, you'll need to switch to d_alloc_parallel() for
+that; look for in-tree examples.
+
+Old method is only used if the new one is absent; eventually it will
+be removed. Switch while you still can; the old one won't stay.
+
+---
+
+**mandatory**
+
+->atomic_open() calls without O_CREAT may happen in parallel.
+
+---
+
+**mandatory**
+
+->setxattr() and xattr_handler.set() get dentry and inode passed separately.
+dentry might be yet to be attached to inode, so do _not_ use its ->d_inode
+in the instances. Rationale: !@#!@# security_d_instantiate() needs to be
+called before we attach dentry to inode and !@#!@##!@$!$#!@#$!@$!@$ smack
+->d_instantiate() uses not just ->getxattr() but ->setxattr() as well.
+
+---
+
+**mandatory**
+
+->d_compare() doesn't get parent as a separate argument anymore. If you
+used it for finding the struct super_block involved, dentry->d_sb will
+work just as well; if it's something more complicated, use dentry->d_parent.
+Just be careful not to assume that fetching it more than once will yield
+the same value - in RCU mode it could change under you.
+
+---
+
+**mandatory**
+
+->rename() has an added flags argument. Any flags not handled by the
+filesystem should result in EINVAL being returned.
+
+---
+
+
+**recommended**
+
+->readlink is optional for symlinks. Don't set, unless filesystem needs
+to fake something for readlink(2).
+
+---
+
+**mandatory**
+
+->getattr() is now passed a struct path rather than a vfsmount and
+dentry separately, and it now has request_mask and query_flags arguments
+to specify the fields and sync type requested by statx. Filesystems not
+supporting any statx-specific features may ignore the new arguments.
+
+---
+
+**mandatory**
+
+->atomic_open() calling conventions have changed. Gone is ``int *opened``,
+along with FILE_OPENED/FILE_CREATED. In place of those we have
+FMODE_OPENED/FMODE_CREATED, set in file->f_mode. Additionally, return
+value for 'called finish_no_open(), open it yourself' case has become
+0, not 1. Since finish_no_open() itself is returning 0 now, that part
+does not need any changes in ->atomic_open() instances.
+
+---
+
+**mandatory**
+
+alloc_file() has become static now; two wrappers are to be used instead.
+alloc_file_pseudo(inode, vfsmount, name, flags, ops) is for the cases
+when dentry needs to be created; that's the majority of old alloc_file()
+users. Calling conventions: on success a reference to new struct file
+is returned and callers reference to inode is subsumed by that. On
+failure, ERR_PTR() is returned and no caller's references are affected,
+so the caller needs to drop the inode reference it held.
+alloc_file_clone(file, flags, ops) does not affect any caller's references.
+On success you get a new struct file sharing the mount/dentry with the
+original, on failure - ERR_PTR().
+
+---
+
+**mandatory**
+
+->clone_file_range() and ->dedupe_file_range have been replaced with
+->remap_file_range(). See Documentation/filesystems/vfs.rst for more
+information.
+
+---
+
+**recommended**
+
+->lookup() instances doing an equivalent of::
+
+ if (IS_ERR(inode))
+ return ERR_CAST(inode);
+ return d_splice_alias(inode, dentry);
+
+don't need to bother with the check - d_splice_alias() will do the
+right thing when given ERR_PTR(...) as inode. Moreover, passing NULL
+inode to d_splice_alias() will also do the right thing (equivalent of
+d_add(dentry, NULL); return NULL;), so that kind of special cases
+also doesn't need a separate treatment.
+
+---
+
+**strongly recommended**
+
+take the RCU-delayed parts of ->destroy_inode() into a new method -
+->free_inode(). If ->destroy_inode() becomes empty - all the better,
+just get rid of it. Synchronous work (e.g. the stuff that can't
+be done from an RCU callback, or any WARN_ON() where we want the
+stack trace) *might* be movable to ->evict_inode(); however,
+that goes only for the things that are not needed to balance something
+done by ->alloc_inode(). IOW, if it's cleaning up the stuff that
+might have accumulated over the life of in-core inode, ->evict_inode()
+might be a fit.
+
+Rules for inode destruction:
+
+ * if ->destroy_inode() is non-NULL, it gets called
+ * if ->free_inode() is non-NULL, it gets scheduled by call_rcu()
+ * combination of NULL ->destroy_inode and NULL ->free_inode is
+ treated as NULL/free_inode_nonrcu, to preserve the compatibility.
+
+Note that the callback (be it via ->free_inode() or explicit call_rcu()
+in ->destroy_inode()) is *NOT* ordered wrt superblock destruction;
+as the matter of fact, the superblock and all associated structures
+might be already gone. The filesystem driver is guaranteed to be still
+there, but that's it. Freeing memory in the callback is fine; doing
+more than that is possible, but requires a lot of care and is best
+avoided.
+
+---
+
+**mandatory**
+
+DCACHE_RCUACCESS is gone; having an RCU delay on dentry freeing is the
+default. DCACHE_NORCU opts out, and only d_alloc_pseudo() has any
+business doing so.
+
+---
+
+**mandatory**
+
+d_alloc_pseudo() is internal-only; uses outside of alloc_file_pseudo() are
+very suspect (and won't work in modules). Such uses are very likely to
+be misspelled d_alloc_anon().
diff --git a/Documentation/filesystems/proc.txt b/Documentation/filesystems/proc.txt
index 22b4b00..99ca040 100644
--- a/Documentation/filesystems/proc.txt
+++ b/Documentation/filesystems/proc.txt
@@ -45,6 +45,7 @@
3.9 /proc/<pid>/map_files - Information about memory mapped files
3.10 /proc/<pid>/timerslack_ns - Task timerslack value
3.11 /proc/<pid>/patch_state - Livepatch patch operation state
+ 3.12 /proc/<pid>/arch_status - Task architecture specific information
4 Configuring procfs
4.1 Mount options
@@ -125,6 +126,13 @@
The link self points to the process reading the file system. Each process
subdirectory has the entries listed in Table 1-1.
+Note that an open a file descriptor to /proc/<pid> or to any of its
+contained files or subdirectories does not prevent <pid> being reused
+for some other process in the event that <pid> exits. Operations on
+open /proc/<pid> file descriptors corresponding to dead processes
+never act on any new process that the kernel may, through chance, have
+also assigned the process ID <pid>. Instead, operations on these FDs
+usually fail with ESRCH.
Table 1-1: Process specific entries in /proc
..............................................................................
@@ -146,9 +154,11 @@
symbol the task is blocked in - or "0" if not blocked.
pagemap Page table
stack Report full stack trace, enable via CONFIG_STACKTRACE
- smaps an extension based on maps, showing the memory consumption of
+ smaps An extension based on maps, showing the memory consumption of
each mapping and flags associated with it
- numa_maps an extension based on maps, showing the memory locality and
+ smaps_rollup Accumulated smaps stats for all mappings of the process. This
+ can be derived from smaps, but is faster and more convenient
+ numa_maps An extension based on maps, showing the memory locality and
binding policy as well as mem usage (in pages) of each mapping.
..............................................................................
@@ -182,6 +192,7 @@
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
+ THP_enabled: 1
Threads: 1
SigQ: 0/28578
SigPnd: 0000000000000000
@@ -193,8 +204,10 @@
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
+ CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
+ Speculation_Store_Bypass: thread vulnerable
voluntary_ctxt_switches: 0
nonvoluntary_ctxt_switches: 1
@@ -214,7 +227,7 @@
snapshot of a moment, you can see /proc/<pid>/smaps file and scan page table.
It's slow but very precise.
-Table 1-2: Contents of the status files (as of 4.8)
+Table 1-2: Contents of the status files (as of 4.19)
..............................................................................
Field Content
Name filename of the executable
@@ -256,6 +269,8 @@
HugetlbPages size of hugetlb memory portions
CoreDumping process's memory is currently being dumped
(killing the process may lead to a corrupted core)
+ THP_enabled process is allowed to use THP (returns 0 when
+ PR_SET_THP_DISABLE is set on the process
Threads number of threads
SigQ number of signals queued/max. number for queue
SigPnd bitmap of pending signals for the thread
@@ -267,8 +282,10 @@
CapPrm bitmap of permitted capabilities
CapEff bitmap of effective capabilities
CapBnd bitmap of capabilities bounding set
+ CapAmb bitmap of ambient capabilities
NoNewPrivs no_new_privs, like prctl(PR_GET_NO_NEW_PRIV, ...)
Seccomp seccomp mode, like prctl(PR_GET_SECCOMP, ...)
+ Speculation_Store_Bypass speculative store bypass mitigation status
Cpus_allowed mask of CPUs on which this process may run
Cpus_allowed_list Same as previous, but in "list format"
Mems_allowed mask of memory nodes allowed to this process
@@ -351,7 +368,7 @@
exit_code the thread's exit_code in the form reported by the waitpid system call
..............................................................................
-The /proc/PID/maps file containing the currently mapped memory regions and
+The /proc/PID/maps file contains the currently mapped memory regions and
their access permissions.
The format is:
@@ -402,11 +419,14 @@
or if empty, the mapping is anonymous.
The /proc/PID/smaps is an extension based on maps, showing the memory
-consumption for each of the process's mappings. For each of mappings there
-is a series of lines such as the following:
+consumption for each of the process's mappings. For each mapping (aka Virtual
+Memory Area, or VMA) there is a series of lines such as the following:
08048000-080bc000 r-xp 00000000 03:02 13130 /bin/bash
+
Size: 1084 kB
+KernelPageSize: 4 kB
+MMUPageSize: 4 kB
Rss: 892 kB
Pss: 374 kB
Shared_Clean: 892 kB
@@ -425,13 +445,17 @@
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Locked: 0 kB
+THPeligible: 0
VmFlags: rd ex mr mw me dw
-the first of these lines shows the same information as is displayed for the
-mapping in /proc/PID/maps. The remaining lines show the size of the mapping
-(size), the amount of the mapping that is currently resident in RAM (RSS), the
-process' proportional share of this mapping (PSS), the number of clean and
-dirty private pages in the mapping.
+The first of these lines shows the same information as is displayed for the
+mapping in /proc/PID/maps. Following lines show the size of the mapping
+(size); the size of each page allocated when backing a VMA (KernelPageSize),
+which is usually the same as the size in the page table entries; the page size
+used by the MMU when backing a VMA (in most cases, the same as KernelPageSize);
+the amount of the mapping that is currently resident in RAM (RSS); the
+process' proportional share of this mapping (PSS); and the number of clean and
+dirty shared and private pages in the mapping.
The "proportional set size" (PSS) of a process is the count of pages it has
in memory, where each page is divided by the number of processes sharing it.
@@ -462,6 +486,8 @@
"SwapPss" shows proportional swap share of this mapping. Unlike "Swap", this
does not take into account swapped out page of underlying shmem objects.
"Locked" indicates whether the mapping is locked in memory or not.
+"THPeligible" indicates whether the mapping is eligible for allocating THP
+pages - 1 if true, 0 otherwise. It just shows the current status.
"VmFlags" field deserves a separate description. This member represents the kernel
flags associated with the particular virtual memory area in two letter encoded
@@ -496,7 +522,9 @@
Note that there is no guarantee that every flag and associated mnemonic will
be present in all further kernel releases. Things get changed, the flags may
-be vanished or the reverse -- new added.
+be vanished or the reverse -- new added. Interpretation of their meaning
+might change in future as well. So each consumer of these flags has to
+follow each specific kernel version for the exact semantic.
This file is only present if the CONFIG_MMU kernel configuration option is
enabled.
@@ -512,6 +540,19 @@
2) If there is something at a given vaddr during the entirety of the
life of the smaps/maps walk, there will be some output for it.
+The /proc/PID/smaps_rollup file includes the same fields as /proc/PID/smaps,
+but their values are the sums of the corresponding values for all mappings of
+the process. Additionally, it contains these fields:
+
+Pss_Anon
+Pss_File
+Pss_Shmem
+
+They represent the proportional shares of anonymous, file, and shmem pages, as
+described for smaps above. These fields are omitted in smaps since each
+mapping identifies the type (anon, file, or shmem) of all pages it contains.
+Thus all information in smaps_rollup can be derived from smaps, but at a
+significantly higher cost.
The /proc/PID/clear_refs is used to reset the PG_Referenced and ACCESSED/YOUNG
bits on both physical and virtual pages associated with a process, and the
@@ -858,6 +899,7 @@
AnonPages: 861800 kB
Mapped: 280372 kB
Shmem: 644 kB
+KReclaimable: 168048 kB
Slab: 284364 kB
SReclaimable: 159856 kB
SUnreclaim: 124508 kB
@@ -925,6 +967,9 @@
ShmemHugePages: Memory used by shared memory (shmem) and tmpfs allocated
with huge pages
ShmemPmdMapped: Shared memory mapped into userspace with huge pages
+KReclaimable: Kernel allocations that the kernel will attempt to reclaim
+ under memory pressure. Includes SReclaimable (below), and other
+ direct allocations with a shrinker.
Slab: in-kernel data structures cache
SReclaimable: Part of Slab, that might be reclaimed, such as caches
SUnreclaim: Part of Slab, that cannot be reclaimed on memory pressure
@@ -1455,7 +1500,7 @@
This chapter is heavily based on the documentation included in the pre 2.2
kernels, and became part of it in version 2.2.1 of the Linux kernel.
-Please see: Documentation/sysctl/ directory for descriptions of these
+Please see: Documentation/admin-guide/sysctl/ directory for descriptions of these
entries.
------------------------------------------------------------------------------
@@ -1925,6 +1970,45 @@
patched. If the patch is being disabled, then the task hasn't been
unpatched yet.
+3.12 /proc/<pid>/arch_status - task architecture specific status
+-------------------------------------------------------------------
+When CONFIG_PROC_PID_ARCH_STATUS is enabled, this file displays the
+architecture specific status of the task.
+
+Example
+-------
+ $ cat /proc/6753/arch_status
+ AVX512_elapsed_ms: 8
+
+Description
+-----------
+
+x86 specific entries:
+---------------------
+ AVX512_elapsed_ms:
+ ------------------
+ If AVX512 is supported on the machine, this entry shows the milliseconds
+ elapsed since the last time AVX512 usage was recorded. The recording
+ happens on a best effort basis when a task is scheduled out. This means
+ that the value depends on two factors:
+
+ 1) The time which the task spent on the CPU without being scheduled
+ out. With CPU isolation and a single runnable task this can take
+ several seconds.
+
+ 2) The time since the task was scheduled out last. Depending on the
+ reason for being scheduled out (time slice exhausted, syscall ...)
+ this can be arbitrary long time.
+
+ As a consequence the value cannot be considered precise and authoritative
+ information. The application which uses this information has to be aware
+ of the overall scenario on the system in order to determine whether a
+ task is a real AVX512 user or not. Precise information can be obtained
+ with performance counters.
+
+ A special value of '-1' indicates that no AVX512 usage was recorded, thus
+ the task is unlikely an AVX512 user, but depends on the workload and the
+ scheduling scenario, it also could be a false negative mentioned above.
------------------------------------------------------------------------------
Configuring procfs
diff --git a/Documentation/filesystems/qnx6.txt b/Documentation/filesystems/qnx6.txt
index 4f3d6a8..48ea68f 100644
--- a/Documentation/filesystems/qnx6.txt
+++ b/Documentation/filesystems/qnx6.txt
@@ -87,7 +87,7 @@
For more than 16 blocks an indirect addressing in form of another tree is
used. (scheme is the same as the one used for the superblock root nodes)
-The filesize is stored 64bit. Inode counting starts with 1. (whilst long
+The filesize is stored 64bit. Inode counting starts with 1. (while long
filename inodes start with 0)
Directories
@@ -155,7 +155,7 @@
The requirement for a static, fixed preallocated system area comes from how
qnx6fs deals with writes.
Each superblock got it's own half of the system area. So superblock #1
-always uses blocks from the lower half whilst superblock #2 just writes to
+always uses blocks from the lower half while superblock #2 just writes to
blocks represented by the upper half bitmap system area bits.
Bitmap blocks, Inode blocks and indirect addressing blocks for those two
diff --git a/Documentation/filesystems/ramfs-rootfs-initramfs.txt b/Documentation/filesystems/ramfs-rootfs-initramfs.txt
index 79637d2..97d42cc 100644
--- a/Documentation/filesystems/ramfs-rootfs-initramfs.txt
+++ b/Documentation/filesystems/ramfs-rootfs-initramfs.txt
@@ -105,7 +105,7 @@
- The old initrd file was a gzipped filesystem image (in some file format,
such as ext2, that needed a driver built into the kernel), while the new
initramfs archive is a gzipped cpio archive (like tar only simpler,
- see cpio(1) and Documentation/early-userspace/buffer-format.txt). The
+ see cpio(1) and Documentation/driver-api/early-userspace/buffer-format.rst). The
kernel's cpio extraction code is not only extremely small, it's also
__init text and data that can be discarded during the boot process.
@@ -159,7 +159,7 @@
set permissions or create device nodes in the new archive. (Note that those
two example "file" entries expect to find files named "init.sh" and "busybox" in
a directory called "initramfs", under the linux-2.6.* directory. See
-Documentation/early-userspace/README for more details.)
+Documentation/driver-api/early-userspace/early_userspace_support.rst for more details.)
The kernel does not depend on external cpio tools. If you specify a
directory instead of a configuration file, the kernel's build infrastructure
diff --git a/Documentation/filesystems/splice.rst b/Documentation/filesystems/splice.rst
new file mode 100644
index 0000000..edd8748
--- /dev/null
+++ b/Documentation/filesystems/splice.rst
@@ -0,0 +1,22 @@
+================
+splice and pipes
+================
+
+splice API
+==========
+
+splice is a method for moving blocks of data around inside the kernel,
+without continually transferring them between the kernel and user space.
+
+.. kernel-doc:: fs/splice.c
+
+pipes API
+=========
+
+Pipe interfaces are all for in-kernel (builtin image) use. They are not
+exported for use by modules.
+
+.. kernel-doc:: include/linux/pipe_fs_i.h
+ :internal:
+
+.. kernel-doc:: fs/pipe.c
diff --git a/Documentation/filesystems/spufs.txt b/Documentation/filesystems/spufs.txt
index 1343d11..eb9e3aa 100644
--- a/Documentation/filesystems/spufs.txt
+++ b/Documentation/filesystems/spufs.txt
@@ -452,7 +452,7 @@
ERRORS
- EACCESS
+ EACCES
The current user does not have write access on the spufs mount
point.
diff --git a/Documentation/filesystems/sysfs.txt b/Documentation/filesystems/sysfs.txt
index a1426ca..ddf15b1 100644
--- a/Documentation/filesystems/sysfs.txt
+++ b/Documentation/filesystems/sysfs.txt
@@ -116,6 +116,27 @@
.store = store_foo,
};
+Note as stated in include/linux/kernel.h "OTHER_WRITABLE? Generally
+considered a bad idea." so trying to set a sysfs file writable for
+everyone will fail reverting to RO mode for "Others".
+
+For the common cases sysfs.h provides convenience macros to make
+defining attributes easier as well as making code more concise and
+readable. The above case could be shortened to:
+
+static struct device_attribute dev_attr_foo = __ATTR_RW(foo);
+
+the list of helpers available to define your wrapper function is:
+__ATTR_RO(name): assumes default name_show and mode 0444
+__ATTR_WO(name): assumes a name_store only and is restricted to mode
+ 0200 that is root write access only.
+__ATTR_RO_MODE(name, mode): fore more restrictive RO access currently
+ only use case is the EFI System Resource Table
+ (see drivers/firmware/efi/esrt.c)
+__ATTR_RW(name): assumes default name_show, name_store and setting
+ mode to 0644.
+__ATTR_NULL: which sets the name to NULL and is used as end of list
+ indicator (see: kernel/workqueue.c)
Subsystem-Specific Callbacks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -298,7 +319,7 @@
a stat(2) operation.
More information can driver-model specific features can be found in
-Documentation/driver-model/.
+Documentation/driver-api/driver-model/.
TODO: Finish this section.
@@ -344,7 +365,9 @@
Declaring:
-BUS_ATTR(_name, _mode, _show, _store)
+static BUS_ATTR_RW(name);
+static BUS_ATTR_RO(name);
+static BUS_ATTR_WO(name);
Creation/Removal:
diff --git a/Documentation/filesystems/tmpfs.txt b/Documentation/filesystems/tmpfs.txt
index d06e9a5..5ecbc03 100644
--- a/Documentation/filesystems/tmpfs.txt
+++ b/Documentation/filesystems/tmpfs.txt
@@ -98,7 +98,7 @@
use at file creation time. When a task allocates a file in the file
system, the mount option memory policy will be applied with a NodeList,
if any, modified by the calling task's cpuset constraints
-[See Documentation/cgroup-v1/cpusets.txt] and any optional flags, listed
+[See Documentation/admin-guide/cgroup-v1/cpusets.rst] and any optional flags, listed
below. If the resulting NodeLists is the empty set, the effective memory
policy for the file will revert to "default" policy.
diff --git a/Documentation/filesystems/ubifs-authentication.rst b/Documentation/filesystems/ubifs-authentication.rst
new file mode 100644
index 0000000..6a9584f
--- /dev/null
+++ b/Documentation/filesystems/ubifs-authentication.rst
@@ -0,0 +1,444 @@
+:orphan:
+
+.. UBIFS Authentication
+.. sigma star gmbh
+.. 2018
+
+Introduction
+============
+
+UBIFS utilizes the fscrypt framework to provide confidentiality for file
+contents and file names. This prevents attacks where an attacker is able to
+read contents of the filesystem on a single point in time. A classic example
+is a lost smartphone where the attacker is unable to read personal data stored
+on the device without the filesystem decryption key.
+
+At the current state, UBIFS encryption however does not prevent attacks where
+the attacker is able to modify the filesystem contents and the user uses the
+device afterwards. In such a scenario an attacker can modify filesystem
+contents arbitrarily without the user noticing. One example is to modify a
+binary to perform a malicious action when executed [DMC-CBC-ATTACK]. Since
+most of the filesystem metadata of UBIFS is stored in plain, this makes it
+fairly easy to swap files and replace their contents.
+
+Other full disk encryption systems like dm-crypt cover all filesystem metadata,
+which makes such kinds of attacks more complicated, but not impossible.
+Especially, if the attacker is given access to the device multiple points in
+time. For dm-crypt and other filesystems that build upon the Linux block IO
+layer, the dm-integrity or dm-verity subsystems [DM-INTEGRITY, DM-VERITY]
+can be used to get full data authentication at the block layer.
+These can also be combined with dm-crypt [CRYPTSETUP2].
+
+This document describes an approach to get file contents _and_ full metadata
+authentication for UBIFS. Since UBIFS uses fscrypt for file contents and file
+name encryption, the authentication system could be tied into fscrypt such that
+existing features like key derivation can be utilized. It should however also
+be possible to use UBIFS authentication without using encryption.
+
+
+MTD, UBI & UBIFS
+----------------
+
+On Linux, the MTD (Memory Technology Devices) subsystem provides a uniform
+interface to access raw flash devices. One of the more prominent subsystems that
+work on top of MTD is UBI (Unsorted Block Images). It provides volume management
+for flash devices and is thus somewhat similar to LVM for block devices. In
+addition, it deals with flash-specific wear-leveling and transparent I/O error
+handling. UBI offers logical erase blocks (LEBs) to the layers on top of it
+and maps them transparently to physical erase blocks (PEBs) on the flash.
+
+UBIFS is a filesystem for raw flash which operates on top of UBI. Thus, wear
+leveling and some flash specifics are left to UBI, while UBIFS focuses on
+scalability, performance and recoverability.
+
+::
+
+ +------------+ +*******+ +-----------+ +-----+
+ | | * UBIFS * | UBI-BLOCK | | ... |
+ | JFFS/JFFS2 | +*******+ +-----------+ +-----+
+ | | +-----------------------------+ +-----------+ +-----+
+ | | | UBI | | MTD-BLOCK | | ... |
+ +------------+ +-----------------------------+ +-----------+ +-----+
+ +------------------------------------------------------------------+
+ | MEMORY TECHNOLOGY DEVICES (MTD) |
+ +------------------------------------------------------------------+
+ +-----------------------------+ +--------------------------+ +-----+
+ | NAND DRIVERS | | NOR DRIVERS | | ... |
+ +-----------------------------+ +--------------------------+ +-----+
+
+ Figure 1: Linux kernel subsystems for dealing with raw flash
+
+
+
+Internally, UBIFS maintains multiple data structures which are persisted on
+the flash:
+
+- *Index*: an on-flash B+ tree where the leaf nodes contain filesystem data
+- *Journal*: an additional data structure to collect FS changes before updating
+ the on-flash index and reduce flash wear.
+- *Tree Node Cache (TNC)*: an in-memory B+ tree that reflects the current FS
+ state to avoid frequent flash reads. It is basically the in-memory
+ representation of the index, but contains additional attributes.
+- *LEB property tree (LPT)*: an on-flash B+ tree for free space accounting per
+ UBI LEB.
+
+In the remainder of this section we will cover the on-flash UBIFS data
+structures in more detail. The TNC is of less importance here since it is never
+persisted onto the flash directly. More details on UBIFS can also be found in
+[UBIFS-WP].
+
+
+UBIFS Index & Tree Node Cache
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Basic on-flash UBIFS entities are called *nodes*. UBIFS knows different types
+of nodes. Eg. data nodes (`struct ubifs_data_node`) which store chunks of file
+contents or inode nodes (`struct ubifs_ino_node`) which represent VFS inodes.
+Almost all types of nodes share a common header (`ubifs_ch`) containing basic
+information like node type, node length, a sequence number, etc. (see
+`fs/ubifs/ubifs-media.h`in kernel source). Exceptions are entries of the LPT
+and some less important node types like padding nodes which are used to pad
+unusable content at the end of LEBs.
+
+To avoid re-writing the whole B+ tree on every single change, it is implemented
+as *wandering tree*, where only the changed nodes are re-written and previous
+versions of them are obsoleted without erasing them right away. As a result,
+the index is not stored in a single place on the flash, but *wanders* around
+and there are obsolete parts on the flash as long as the LEB containing them is
+not reused by UBIFS. To find the most recent version of the index, UBIFS stores
+a special node called *master node* into UBI LEB 1 which always points to the
+most recent root node of the UBIFS index. For recoverability, the master node
+is additionally duplicated to LEB 2. Mounting UBIFS is thus a simple read of
+LEB 1 and 2 to get the current master node and from there get the location of
+the most recent on-flash index.
+
+The TNC is the in-memory representation of the on-flash index. It contains some
+additional runtime attributes per node which are not persisted. One of these is
+a dirty-flag which marks nodes that have to be persisted the next time the
+index is written onto the flash. The TNC acts as a write-back cache and all
+modifications of the on-flash index are done through the TNC. Like other caches,
+the TNC does not have to mirror the full index into memory, but reads parts of
+it from flash whenever needed. A *commit* is the UBIFS operation of updating the
+on-flash filesystem structures like the index. On every commit, the TNC nodes
+marked as dirty are written to the flash to update the persisted index.
+
+
+Journal
+~~~~~~~
+
+To avoid wearing out the flash, the index is only persisted (*commited*) when
+certain conditions are met (eg. ``fsync(2)``). The journal is used to record
+any changes (in form of inode nodes, data nodes etc.) between commits
+of the index. During mount, the journal is read from the flash and replayed
+onto the TNC (which will be created on-demand from the on-flash index).
+
+UBIFS reserves a bunch of LEBs just for the journal called *log area*. The
+amount of log area LEBs is configured on filesystem creation (using
+``mkfs.ubifs``) and stored in the superblock node. The log area contains only
+two types of nodes: *reference nodes* and *commit start nodes*. A commit start
+node is written whenever an index commit is performed. Reference nodes are
+written on every journal update. Each reference node points to the position of
+other nodes (inode nodes, data nodes etc.) on the flash that are part of this
+journal entry. These nodes are called *buds* and describe the actual filesystem
+changes including their data.
+
+The log area is maintained as a ring. Whenever the journal is almost full,
+a commit is initiated. This also writes a commit start node so that during
+mount, UBIFS will seek for the most recent commit start node and just replay
+every reference node after that. Every reference node before the commit start
+node will be ignored as they are already part of the on-flash index.
+
+When writing a journal entry, UBIFS first ensures that enough space is
+available to write the reference node and buds part of this entry. Then, the
+reference node is written and afterwards the buds describing the file changes.
+On replay, UBIFS will record every reference node and inspect the location of
+the referenced LEBs to discover the buds. If these are corrupt or missing,
+UBIFS will attempt to recover them by re-reading the LEB. This is however only
+done for the last referenced LEB of the journal. Only this can become corrupt
+because of a power cut. If the recovery fails, UBIFS will not mount. An error
+for every other LEB will directly cause UBIFS to fail the mount operation.
+
+::
+
+ | ---- LOG AREA ---- | ---------- MAIN AREA ------------ |
+
+ -----+------+-----+--------+---- ------+-----+-----+---------------
+ \ | | | | / / | | | \
+ / CS | REF | REF | | \ \ DENT | INO | INO | /
+ \ | | | | / / | | | \
+ ----+------+-----+--------+--- -------+-----+-----+----------------
+ | | ^ ^
+ | | | |
+ +------------------------+ |
+ | |
+ +-------------------------------+
+
+
+ Figure 2: UBIFS flash layout of log area with commit start nodes
+ (CS) and reference nodes (REF) pointing to main area
+ containing their buds
+
+
+LEB Property Tree/Table
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The LEB property tree is used to store per-LEB information. This includes the
+LEB type and amount of free and *dirty* (old, obsolete content) space [1]_ on
+the LEB. The type is important, because UBIFS never mixes index nodes with data
+nodes on a single LEB and thus each LEB has a specific purpose. This again is
+useful for free space calculations. See [UBIFS-WP] for more details.
+
+The LEB property tree again is a B+ tree, but it is much smaller than the
+index. Due to its smaller size it is always written as one chunk on every
+commit. Thus, saving the LPT is an atomic operation.
+
+
+.. [1] Since LEBs can only be appended and never overwritten, there is a
+ difference between free space ie. the remaining space left on the LEB to be
+ written to without erasing it and previously written content that is obsolete
+ but can't be overwritten without erasing the full LEB.
+
+
+UBIFS Authentication
+====================
+
+This chapter introduces UBIFS authentication which enables UBIFS to verify
+the authenticity and integrity of metadata and file contents stored on flash.
+
+
+Threat Model
+------------
+
+UBIFS authentication enables detection of offline data modification. While it
+does not prevent it, it enables (trusted) code to check the integrity and
+authenticity of on-flash file contents and filesystem metadata. This covers
+attacks where file contents are swapped.
+
+UBIFS authentication will not protect against rollback of full flash contents.
+Ie. an attacker can still dump the flash and restore it at a later time without
+detection. It will also not protect against partial rollback of individual
+index commits. That means that an attacker is able to partially undo changes.
+This is possible because UBIFS does not immediately overwrites obsolete
+versions of the index tree or the journal, but instead marks them as obsolete
+and garbage collection erases them at a later time. An attacker can use this by
+erasing parts of the current tree and restoring old versions that are still on
+the flash and have not yet been erased. This is possible, because every commit
+will always write a new version of the index root node and the master node
+without overwriting the previous version. This is further helped by the
+wear-leveling operations of UBI which copies contents from one physical
+eraseblock to another and does not atomically erase the first eraseblock.
+
+UBIFS authentication does not cover attacks where an attacker is able to
+execute code on the device after the authentication key was provided.
+Additional measures like secure boot and trusted boot have to be taken to
+ensure that only trusted code is executed on a device.
+
+
+Authentication
+--------------
+
+To be able to fully trust data read from flash, all UBIFS data structures
+stored on flash are authenticated. That is:
+
+- The index which includes file contents, file metadata like extended
+ attributes, file length etc.
+- The journal which also contains file contents and metadata by recording changes
+ to the filesystem
+- The LPT which stores UBI LEB metadata which UBIFS uses for free space accounting
+
+
+Index Authentication
+~~~~~~~~~~~~~~~~~~~~
+
+Through UBIFS' concept of a wandering tree, it already takes care of only
+updating and persisting changed parts from leaf node up to the root node
+of the full B+ tree. This enables us to augment the index nodes of the tree
+with a hash over each node's child nodes. As a result, the index basically also
+a Merkle tree. Since the leaf nodes of the index contain the actual filesystem
+data, the hashes of their parent index nodes thus cover all the file contents
+and file metadata. When a file changes, the UBIFS index is updated accordingly
+from the leaf nodes up to the root node including the master node. This process
+can be hooked to recompute the hash only for each changed node at the same time.
+Whenever a file is read, UBIFS can verify the hashes from each leaf node up to
+the root node to ensure the node's integrity.
+
+To ensure the authenticity of the whole index, the UBIFS master node stores a
+keyed hash (HMAC) over its own contents and a hash of the root node of the index
+tree. As mentioned above, the master node is always written to the flash whenever
+the index is persisted (ie. on index commit).
+
+Using this approach only UBIFS index nodes and the master node are changed to
+include a hash. All other types of nodes will remain unchanged. This reduces
+the storage overhead which is precious for users of UBIFS (ie. embedded
+devices).
+
+::
+
+ +---------------+
+ | Master Node |
+ | (hash) |
+ +---------------+
+ |
+ v
+ +-------------------+
+ | Index Node #1 |
+ | |
+ | branch0 branchn |
+ | (hash) (hash) |
+ +-------------------+
+ | ... | (fanout: 8)
+ | |
+ +-------+ +------+
+ | |
+ v v
+ +-------------------+ +-------------------+
+ | Index Node #2 | | Index Node #3 |
+ | | | |
+ | branch0 branchn | | branch0 branchn |
+ | (hash) (hash) | | (hash) (hash) |
+ +-------------------+ +-------------------+
+ | ... | ... |
+ v v v
+ +-----------+ +----------+ +-----------+
+ | Data Node | | INO Node | | DENT Node |
+ +-----------+ +----------+ +-----------+
+
+
+ Figure 3: Coverage areas of index node hash and master node HMAC
+
+
+
+The most important part for robustness and power-cut safety is to atomically
+persist the hash and file contents. Here the existing UBIFS logic for how
+changed nodes are persisted is already designed for this purpose such that
+UBIFS can safely recover if a power-cut occurs while persisting. Adding
+hashes to index nodes does not change this since each hash will be persisted
+atomically together with its respective node.
+
+
+Journal Authentication
+~~~~~~~~~~~~~~~~~~~~~~
+
+The journal is authenticated too. Since the journal is continuously written
+it is necessary to also add authentication information frequently to the
+journal so that in case of a powercut not too much data can't be authenticated.
+This is done by creating a continuous hash beginning from the commit start node
+over the previous reference nodes, the current reference node, and the bud
+nodes. From time to time whenever it is suitable authentication nodes are added
+between the bud nodes. This new node type contains a HMAC over the current state
+of the hash chain. That way a journal can be authenticated up to the last
+authentication node. The tail of the journal which may not have a authentication
+node cannot be authenticated and is skipped during journal replay.
+
+We get this picture for journal authentication::
+
+ ,,,,,,,,
+ ,......,...........................................
+ ,. CS , hash1.----. hash2.----.
+ ,. | , . |hmac . |hmac
+ ,. v , . v . v
+ ,.REF#0,-> bud -> bud -> bud.-> auth -> bud -> bud.-> auth ...
+ ,..|...,...........................................
+ , | ,
+ , | ,,,,,,,,,,,,,,,
+ . | hash3,----.
+ , | , |hmac
+ , v , v
+ , REF#1 -> bud -> bud,-> auth ...
+ ,,,|,,,,,,,,,,,,,,,,,,
+ v
+ REF#2 -> ...
+ |
+ V
+ ...
+
+Since the hash also includes the reference nodes an attacker cannot reorder or
+skip any journal heads for replay. An attacker can only remove bud nodes or
+reference nodes from the end of the journal, effectively rewinding the
+filesystem at maximum back to the last commit.
+
+The location of the log area is stored in the master node. Since the master
+node is authenticated with a HMAC as described above, it is not possible to
+tamper with that without detection. The size of the log area is specified when
+the filesystem is created using `mkfs.ubifs` and stored in the superblock node.
+To avoid tampering with this and other values stored there, a HMAC is added to
+the superblock struct. The superblock node is stored in LEB 0 and is only
+modified on feature flag or similar changes, but never on file changes.
+
+
+LPT Authentication
+~~~~~~~~~~~~~~~~~~
+
+The location of the LPT root node on the flash is stored in the UBIFS master
+node. Since the LPT is written and read atomically on every commit, there is
+no need to authenticate individual nodes of the tree. It suffices to
+protect the integrity of the full LPT by a simple hash stored in the master
+node. Since the master node itself is authenticated, the LPTs authenticity can
+be verified by verifying the authenticity of the master node and comparing the
+LTP hash stored there with the hash computed from the read on-flash LPT.
+
+
+Key Management
+--------------
+
+For simplicity, UBIFS authentication uses a single key to compute the HMACs
+of superblock, master, commit start and reference nodes. This key has to be
+available on creation of the filesystem (`mkfs.ubifs`) to authenticate the
+superblock node. Further, it has to be available on mount of the filesystem
+to verify authenticated nodes and generate new HMACs for changes.
+
+UBIFS authentication is intended to operate side-by-side with UBIFS encryption
+(fscrypt) to provide confidentiality and authenticity. Since UBIFS encryption
+has a different approach of encryption policies per directory, there can be
+multiple fscrypt master keys and there might be folders without encryption.
+UBIFS authentication on the other hand has an all-or-nothing approach in the
+sense that it either authenticates everything of the filesystem or nothing.
+Because of this and because UBIFS authentication should also be usable without
+encryption, it does not share the same master key with fscrypt, but manages
+a dedicated authentication key.
+
+The API for providing the authentication key has yet to be defined, but the
+key can eg. be provided by userspace through a keyring similar to the way it
+is currently done in fscrypt. It should however be noted that the current
+fscrypt approach has shown its flaws and the userspace API will eventually
+change [FSCRYPT-POLICY2].
+
+Nevertheless, it will be possible for a user to provide a single passphrase
+or key in userspace that covers UBIFS authentication and encryption. This can
+be solved by the corresponding userspace tools which derive a second key for
+authentication in addition to the derived fscrypt master key used for
+encryption.
+
+To be able to check if the proper key is available on mount, the UBIFS
+superblock node will additionally store a hash of the authentication key. This
+approach is similar to the approach proposed for fscrypt encryption policy v2
+[FSCRYPT-POLICY2].
+
+
+Future Extensions
+=================
+
+In certain cases where a vendor wants to provide an authenticated filesystem
+image to customers, it should be possible to do so without sharing the secret
+UBIFS authentication key. Instead, in addition the each HMAC a digital
+signature could be stored where the vendor shares the public key alongside the
+filesystem image. In case this filesystem has to be modified afterwards,
+UBIFS can exchange all digital signatures with HMACs on first mount similar
+to the way the IMA/EVM subsystem deals with such situations. The HMAC key
+will then have to be provided beforehand in the normal way.
+
+
+References
+==========
+
+[CRYPTSETUP2] http://www.saout.de/pipermail/dm-crypt/2017-November/005745.html
+
+[DMC-CBC-ATTACK] http://www.jakoblell.com/blog/2013/12/22/practical-malleability-attack-against-cbc-encrypted-luks-partitions/
+
+[DM-INTEGRITY] https://www.kernel.org/doc/Documentation/device-mapper/dm-integrity.rst
+
+[DM-VERITY] https://www.kernel.org/doc/Documentation/device-mapper/verity.rst
+
+[FSCRYPT-POLICY2] https://www.spinics.net/lists/linux-ext4/msg58710.html
+
+[UBIFS-WP] http://www.linux-mtd.infradead.org/doc/ubifs_whitepaper.pdf
diff --git a/Documentation/filesystems/ubifs.txt b/Documentation/filesystems/ubifs.txt
index a0a61d2..acc8044 100644
--- a/Documentation/filesystems/ubifs.txt
+++ b/Documentation/filesystems/ubifs.txt
@@ -91,6 +91,13 @@
compr=none override default compressor and set it to "none"
compr=lzo override default compressor and set it to "lzo"
compr=zlib override default compressor and set it to "zlib"
+auth_key= specify the key used for authenticating the filesystem.
+ Passing this option makes authentication mandatory.
+ The passed key must be present in the kernel keyring
+ and must be of type 'logon'
+auth_hash_name= The hash algorithm used for authentication. Used for
+ both hashing and for creating HMACs. Typical values
+ include "sha256" or "sha512"
Quick usage instructions
diff --git a/Documentation/filesystems/ufs.txt b/Documentation/filesystems/ufs.txt
deleted file mode 100644
index 7a602ad..0000000
--- a/Documentation/filesystems/ufs.txt
+++ /dev/null
@@ -1,60 +0,0 @@
-USING UFS
-=========
-
-mount -t ufs -o ufstype=type_of_ufs device dir
-
-
-UFS OPTIONS
-===========
-
-ufstype=type_of_ufs
- UFS is a file system widely used in different operating systems.
- The problem are differences among implementations. Features of
- some implementations are undocumented, so its hard to recognize
- type of ufs automatically. That's why user must specify type of
- ufs manually by mount option ufstype. Possible values are:
-
- old old format of ufs
- default value, supported as read-only
-
- 44bsd used in FreeBSD, NetBSD, OpenBSD
- supported as read-write
-
- ufs2 used in FreeBSD 5.x
- supported as read-write
-
- 5xbsd synonym for ufs2
-
- sun used in SunOS (Solaris)
- supported as read-write
-
- sunx86 used in SunOS for Intel (Solarisx86)
- supported as read-write
-
- hp used in HP-UX
- supported as read-only
-
- nextstep
- used in NextStep
- supported as read-only
-
- nextstep-cd
- used for NextStep CDROMs (block_size == 2048)
- supported as read-only
-
- openstep
- used in OpenStep
- supported as read-only
-
-
-POSSIBLE PROBLEMS
-=================
-
-See next section, if you have any.
-
-
-BUG REPORTS
-===========
-
-Any ufs bug report you can send to daniel.pirkl@email.cz or
-to dushistov@mail.ru (do not send partition tables bug reports).
diff --git a/Documentation/filesystems/vfs.rst b/Documentation/filesystems/vfs.rst
new file mode 100644
index 0000000..7d4d09d
--- /dev/null
+++ b/Documentation/filesystems/vfs.rst
@@ -0,0 +1,1428 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=========================================
+Overview of the Linux Virtual File System
+=========================================
+
+Original author: Richard Gooch <rgooch@atnf.csiro.au>
+
+- Copyright (C) 1999 Richard Gooch
+- Copyright (C) 2005 Pekka Enberg
+
+
+Introduction
+============
+
+The Virtual File System (also known as the Virtual Filesystem Switch) is
+the software layer in the kernel that provides the filesystem interface
+to userspace programs. It also provides an abstraction within the
+kernel which allows different filesystem implementations to coexist.
+
+VFS system calls open(2), stat(2), read(2), write(2), chmod(2) and so on
+are called from a process context. Filesystem locking is described in
+the document Documentation/filesystems/locking.rst.
+
+
+Directory Entry Cache (dcache)
+------------------------------
+
+The VFS implements the open(2), stat(2), chmod(2), and similar system
+calls. The pathname argument that is passed to them is used by the VFS
+to search through the directory entry cache (also known as the dentry
+cache or dcache). This provides a very fast look-up mechanism to
+translate a pathname (filename) into a specific dentry. Dentries live
+in RAM and are never saved to disc: they exist only for performance.
+
+The dentry cache is meant to be a view into your entire filespace. As
+most computers cannot fit all dentries in the RAM at the same time, some
+bits of the cache are missing. In order to resolve your pathname into a
+dentry, the VFS may have to resort to creating dentries along the way,
+and then loading the inode. This is done by looking up the inode.
+
+
+The Inode Object
+----------------
+
+An individual dentry usually has a pointer to an inode. Inodes are
+filesystem objects such as regular files, directories, FIFOs and other
+beasts. They live either on the disc (for block device filesystems) or
+in the memory (for pseudo filesystems). Inodes that live on the disc
+are copied into the memory when required and changes to the inode are
+written back to disc. A single inode can be pointed to by multiple
+dentries (hard links, for example, do this).
+
+To look up an inode requires that the VFS calls the lookup() method of
+the parent directory inode. This method is installed by the specific
+filesystem implementation that the inode lives in. Once the VFS has the
+required dentry (and hence the inode), we can do all those boring things
+like open(2) the file, or stat(2) it to peek at the inode data. The
+stat(2) operation is fairly simple: once the VFS has the dentry, it
+peeks at the inode data and passes some of it back to userspace.
+
+
+The File Object
+---------------
+
+Opening a file requires another operation: allocation of a file
+structure (this is the kernel-side implementation of file descriptors).
+The freshly allocated file structure is initialized with a pointer to
+the dentry and a set of file operation member functions. These are
+taken from the inode data. The open() file method is then called so the
+specific filesystem implementation can do its work. You can see that
+this is another switch performed by the VFS. The file structure is
+placed into the file descriptor table for the process.
+
+Reading, writing and closing files (and other assorted VFS operations)
+is done by using the userspace file descriptor to grab the appropriate
+file structure, and then calling the required file structure method to
+do whatever is required. For as long as the file is open, it keeps the
+dentry in use, which in turn means that the VFS inode is still in use.
+
+
+Registering and Mounting a Filesystem
+=====================================
+
+To register and unregister a filesystem, use the following API
+functions:
+
+.. code-block:: c
+
+ #include <linux/fs.h>
+
+ extern int register_filesystem(struct file_system_type *);
+ extern int unregister_filesystem(struct file_system_type *);
+
+The passed struct file_system_type describes your filesystem. When a
+request is made to mount a filesystem onto a directory in your
+namespace, the VFS will call the appropriate mount() method for the
+specific filesystem. New vfsmount referring to the tree returned by
+->mount() will be attached to the mountpoint, so that when pathname
+resolution reaches the mountpoint it will jump into the root of that
+vfsmount.
+
+You can see all filesystems that are registered to the kernel in the
+file /proc/filesystems.
+
+
+struct file_system_type
+-----------------------
+
+This describes the filesystem. As of kernel 2.6.39, the following
+members are defined:
+
+.. code-block:: c
+
+ struct file_system_operations {
+ const char *name;
+ int fs_flags;
+ struct dentry *(*mount) (struct file_system_type *, int,
+ const char *, void *);
+ void (*kill_sb) (struct super_block *);
+ struct module *owner;
+ struct file_system_type * next;
+ struct list_head fs_supers;
+ struct lock_class_key s_lock_key;
+ struct lock_class_key s_umount_key;
+ };
+
+``name``
+ the name of the filesystem type, such as "ext2", "iso9660",
+ "msdos" and so on
+
+``fs_flags``
+ various flags (i.e. FS_REQUIRES_DEV, FS_NO_DCACHE, etc.)
+
+``mount``
+ the method to call when a new instance of this filesystem should
+ be mounted
+
+``kill_sb``
+ the method to call when an instance of this filesystem should be
+ shut down
+
+
+``owner``
+ for internal VFS use: you should initialize this to THIS_MODULE
+ in most cases.
+
+``next``
+ for internal VFS use: you should initialize this to NULL
+
+ s_lock_key, s_umount_key: lockdep-specific
+
+The mount() method has the following arguments:
+
+``struct file_system_type *fs_type``
+ describes the filesystem, partly initialized by the specific
+ filesystem code
+
+``int flags``
+ mount flags
+
+``const char *dev_name``
+ the device name we are mounting.
+
+``void *data``
+ arbitrary mount options, usually comes as an ASCII string (see
+ "Mount Options" section)
+
+The mount() method must return the root dentry of the tree requested by
+caller. An active reference to its superblock must be grabbed and the
+superblock must be locked. On failure it should return ERR_PTR(error).
+
+The arguments match those of mount(2) and their interpretation depends
+on filesystem type. E.g. for block filesystems, dev_name is interpreted
+as block device name, that device is opened and if it contains a
+suitable filesystem image the method creates and initializes struct
+super_block accordingly, returning its root dentry to caller.
+
+->mount() may choose to return a subtree of existing filesystem - it
+doesn't have to create a new one. The main result from the caller's
+point of view is a reference to dentry at the root of (sub)tree to be
+attached; creation of new superblock is a common side effect.
+
+The most interesting member of the superblock structure that the mount()
+method fills in is the "s_op" field. This is a pointer to a "struct
+super_operations" which describes the next level of the filesystem
+implementation.
+
+Usually, a filesystem uses one of the generic mount() implementations
+and provides a fill_super() callback instead. The generic variants are:
+
+``mount_bdev``
+ mount a filesystem residing on a block device
+
+``mount_nodev``
+ mount a filesystem that is not backed by a device
+
+``mount_single``
+ mount a filesystem which shares the instance between all mounts
+
+A fill_super() callback implementation has the following arguments:
+
+``struct super_block *sb``
+ the superblock structure. The callback must initialize this
+ properly.
+
+``void *data``
+ arbitrary mount options, usually comes as an ASCII string (see
+ "Mount Options" section)
+
+``int silent``
+ whether or not to be silent on error
+
+
+The Superblock Object
+=====================
+
+A superblock object represents a mounted filesystem.
+
+
+struct super_operations
+-----------------------
+
+This describes how the VFS can manipulate the superblock of your
+filesystem. As of kernel 2.6.22, the following members are defined:
+
+.. code-block:: c
+
+ struct super_operations {
+ struct inode *(*alloc_inode)(struct super_block *sb);
+ void (*destroy_inode)(struct inode *);
+
+ void (*dirty_inode) (struct inode *, int flags);
+ int (*write_inode) (struct inode *, int);
+ void (*drop_inode) (struct inode *);
+ void (*delete_inode) (struct inode *);
+ void (*put_super) (struct super_block *);
+ int (*sync_fs)(struct super_block *sb, int wait);
+ int (*freeze_fs) (struct super_block *);
+ int (*unfreeze_fs) (struct super_block *);
+ int (*statfs) (struct dentry *, struct kstatfs *);
+ int (*remount_fs) (struct super_block *, int *, char *);
+ void (*clear_inode) (struct inode *);
+ void (*umount_begin) (struct super_block *);
+
+ int (*show_options)(struct seq_file *, struct dentry *);
+
+ ssize_t (*quota_read)(struct super_block *, int, char *, size_t, loff_t);
+ ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t);
+ int (*nr_cached_objects)(struct super_block *);
+ void (*free_cached_objects)(struct super_block *, int);
+ };
+
+All methods are called without any locks being held, unless otherwise
+noted. This means that most methods can block safely. All methods are
+only called from a process context (i.e. not from an interrupt handler
+or bottom half).
+
+``alloc_inode``
+ this method is called by alloc_inode() to allocate memory for
+ struct inode and initialize it. If this function is not
+ defined, a simple 'struct inode' is allocated. Normally
+ alloc_inode will be used to allocate a larger structure which
+ contains a 'struct inode' embedded within it.
+
+``destroy_inode``
+ this method is called by destroy_inode() to release resources
+ allocated for struct inode. It is only required if
+ ->alloc_inode was defined and simply undoes anything done by
+ ->alloc_inode.
+
+``dirty_inode``
+ this method is called by the VFS to mark an inode dirty.
+
+``write_inode``
+ this method is called when the VFS needs to write an inode to
+ disc. The second parameter indicates whether the write should
+ be synchronous or not, not all filesystems check this flag.
+
+``drop_inode``
+ called when the last access to the inode is dropped, with the
+ inode->i_lock spinlock held.
+
+ This method should be either NULL (normal UNIX filesystem
+ semantics) or "generic_delete_inode" (for filesystems that do
+ not want to cache inodes - causing "delete_inode" to always be
+ called regardless of the value of i_nlink)
+
+ The "generic_delete_inode()" behavior is equivalent to the old
+ practice of using "force_delete" in the put_inode() case, but
+ does not have the races that the "force_delete()" approach had.
+
+``delete_inode``
+ called when the VFS wants to delete an inode
+
+``put_super``
+ called when the VFS wishes to free the superblock
+ (i.e. unmount). This is called with the superblock lock held
+
+``sync_fs``
+ called when VFS is writing out all dirty data associated with a
+ superblock. The second parameter indicates whether the method
+ should wait until the write out has been completed. Optional.
+
+``freeze_fs``
+ called when VFS is locking a filesystem and forcing it into a
+ consistent state. This method is currently used by the Logical
+ Volume Manager (LVM).
+
+``unfreeze_fs``
+ called when VFS is unlocking a filesystem and making it writable
+ again.
+
+``statfs``
+ called when the VFS needs to get filesystem statistics.
+
+``remount_fs``
+ called when the filesystem is remounted. This is called with
+ the kernel lock held
+
+``clear_inode``
+ called then the VFS clears the inode. Optional
+
+``umount_begin``
+ called when the VFS is unmounting a filesystem.
+
+``show_options``
+ called by the VFS to show mount options for /proc/<pid>/mounts.
+ (see "Mount Options" section)
+
+``quota_read``
+ called by the VFS to read from filesystem quota file.
+
+``quota_write``
+ called by the VFS to write to filesystem quota file.
+
+``nr_cached_objects``
+ called by the sb cache shrinking function for the filesystem to
+ return the number of freeable cached objects it contains.
+ Optional.
+
+``free_cache_objects``
+ called by the sb cache shrinking function for the filesystem to
+ scan the number of objects indicated to try to free them.
+ Optional, but any filesystem implementing this method needs to
+ also implement ->nr_cached_objects for it to be called
+ correctly.
+
+ We can't do anything with any errors that the filesystem might
+ encountered, hence the void return type. This will never be
+ called if the VM is trying to reclaim under GFP_NOFS conditions,
+ hence this method does not need to handle that situation itself.
+
+ Implementations must include conditional reschedule calls inside
+ any scanning loop that is done. This allows the VFS to
+ determine appropriate scan batch sizes without having to worry
+ about whether implementations will cause holdoff problems due to
+ large scan batch sizes.
+
+Whoever sets up the inode is responsible for filling in the "i_op"
+field. This is a pointer to a "struct inode_operations" which describes
+the methods that can be performed on individual inodes.
+
+
+struct xattr_handlers
+---------------------
+
+On filesystems that support extended attributes (xattrs), the s_xattr
+superblock field points to a NULL-terminated array of xattr handlers.
+Extended attributes are name:value pairs.
+
+``name``
+ Indicates that the handler matches attributes with the specified
+ name (such as "system.posix_acl_access"); the prefix field must
+ be NULL.
+
+``prefix``
+ Indicates that the handler matches all attributes with the
+ specified name prefix (such as "user."); the name field must be
+ NULL.
+
+``list``
+ Determine if attributes matching this xattr handler should be
+ listed for a particular dentry. Used by some listxattr
+ implementations like generic_listxattr.
+
+``get``
+ Called by the VFS to get the value of a particular extended
+ attribute. This method is called by the getxattr(2) system
+ call.
+
+``set``
+ Called by the VFS to set the value of a particular extended
+ attribute. When the new value is NULL, called to remove a
+ particular extended attribute. This method is called by the the
+ setxattr(2) and removexattr(2) system calls.
+
+When none of the xattr handlers of a filesystem match the specified
+attribute name or when a filesystem doesn't support extended attributes,
+the various ``*xattr(2)`` system calls return -EOPNOTSUPP.
+
+
+The Inode Object
+================
+
+An inode object represents an object within the filesystem.
+
+
+struct inode_operations
+-----------------------
+
+This describes how the VFS can manipulate an inode in your filesystem.
+As of kernel 2.6.22, the following members are defined:
+
+.. code-block:: c
+
+ struct inode_operations {
+ int (*create) (struct inode *,struct dentry *, umode_t, bool);
+ struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int);
+ int (*link) (struct dentry *,struct inode *,struct dentry *);
+ int (*unlink) (struct inode *,struct dentry *);
+ int (*symlink) (struct inode *,struct dentry *,const char *);
+ int (*mkdir) (struct inode *,struct dentry *,umode_t);
+ int (*rmdir) (struct inode *,struct dentry *);
+ int (*mknod) (struct inode *,struct dentry *,umode_t,dev_t);
+ int (*rename) (struct inode *, struct dentry *,
+ struct inode *, struct dentry *, unsigned int);
+ int (*readlink) (struct dentry *, char __user *,int);
+ const char *(*get_link) (struct dentry *, struct inode *,
+ struct delayed_call *);
+ int (*permission) (struct inode *, int);
+ int (*get_acl)(struct inode *, int);
+ int (*setattr) (struct dentry *, struct iattr *);
+ int (*getattr) (const struct path *, struct kstat *, u32, unsigned int);
+ ssize_t (*listxattr) (struct dentry *, char *, size_t);
+ void (*update_time)(struct inode *, struct timespec *, int);
+ int (*atomic_open)(struct inode *, struct dentry *, struct file *,
+ unsigned open_flag, umode_t create_mode);
+ int (*tmpfile) (struct inode *, struct dentry *, umode_t);
+ };
+
+Again, all methods are called without any locks being held, unless
+otherwise noted.
+
+``create``
+ called by the open(2) and creat(2) system calls. Only required
+ if you want to support regular files. The dentry you get should
+ not have an inode (i.e. it should be a negative dentry). Here
+ you will probably call d_instantiate() with the dentry and the
+ newly created inode
+
+``lookup``
+ called when the VFS needs to look up an inode in a parent
+ directory. The name to look for is found in the dentry. This
+ method must call d_add() to insert the found inode into the
+ dentry. The "i_count" field in the inode structure should be
+ incremented. If the named inode does not exist a NULL inode
+ should be inserted into the dentry (this is called a negative
+ dentry). Returning an error code from this routine must only be
+ done on a real error, otherwise creating inodes with system
+ calls like create(2), mknod(2), mkdir(2) and so on will fail.
+ If you wish to overload the dentry methods then you should
+ initialise the "d_dop" field in the dentry; this is a pointer to
+ a struct "dentry_operations". This method is called with the
+ directory inode semaphore held
+
+``link``
+ called by the link(2) system call. Only required if you want to
+ support hard links. You will probably need to call
+ d_instantiate() just as you would in the create() method
+
+``unlink``
+ called by the unlink(2) system call. Only required if you want
+ to support deleting inodes
+
+``symlink``
+ called by the symlink(2) system call. Only required if you want
+ to support symlinks. You will probably need to call
+ d_instantiate() just as you would in the create() method
+
+``mkdir``
+ called by the mkdir(2) system call. Only required if you want
+ to support creating subdirectories. You will probably need to
+ call d_instantiate() just as you would in the create() method
+
+``rmdir``
+ called by the rmdir(2) system call. Only required if you want
+ to support deleting subdirectories
+
+``mknod``
+ called by the mknod(2) system call to create a device (char,
+ block) inode or a named pipe (FIFO) or socket. Only required if
+ you want to support creating these types of inodes. You will
+ probably need to call d_instantiate() just as you would in the
+ create() method
+
+``rename``
+ called by the rename(2) system call to rename the object to have
+ the parent and name given by the second inode and dentry.
+
+ The filesystem must return -EINVAL for any unsupported or
+ unknown flags. Currently the following flags are implemented:
+ (1) RENAME_NOREPLACE: this flag indicates that if the target of
+ the rename exists the rename should fail with -EEXIST instead of
+ replacing the target. The VFS already checks for existence, so
+ for local filesystems the RENAME_NOREPLACE implementation is
+ equivalent to plain rename.
+ (2) RENAME_EXCHANGE: exchange source and target. Both must
+ exist; this is checked by the VFS. Unlike plain rename, source
+ and target may be of different type.
+
+``get_link``
+ called by the VFS to follow a symbolic link to the inode it
+ points to. Only required if you want to support symbolic links.
+ This method returns the symlink body to traverse (and possibly
+ resets the current position with nd_jump_link()). If the body
+ won't go away until the inode is gone, nothing else is needed;
+ if it needs to be otherwise pinned, arrange for its release by
+ having get_link(..., ..., done) do set_delayed_call(done,
+ destructor, argument). In that case destructor(argument) will
+ be called once VFS is done with the body you've returned. May
+ be called in RCU mode; that is indicated by NULL dentry
+ argument. If request can't be handled without leaving RCU mode,
+ have it return ERR_PTR(-ECHILD).
+
+ If the filesystem stores the symlink target in ->i_link, the
+ VFS may use it directly without calling ->get_link(); however,
+ ->get_link() must still be provided. ->i_link must not be
+ freed until after an RCU grace period. Writing to ->i_link
+ post-iget() time requires a 'release' memory barrier.
+
+``readlink``
+ this is now just an override for use by readlink(2) for the
+ cases when ->get_link uses nd_jump_link() or object is not in
+ fact a symlink. Normally filesystems should only implement
+ ->get_link for symlinks and readlink(2) will automatically use
+ that.
+
+``permission``
+ called by the VFS to check for access rights on a POSIX-like
+ filesystem.
+
+ May be called in rcu-walk mode (mask & MAY_NOT_BLOCK). If in
+ rcu-walk mode, the filesystem must check the permission without
+ blocking or storing to the inode.
+
+ If a situation is encountered that rcu-walk cannot handle,
+ return
+ -ECHILD and it will be called again in ref-walk mode.
+
+``setattr``
+ called by the VFS to set attributes for a file. This method is
+ called by chmod(2) and related system calls.
+
+``getattr``
+ called by the VFS to get attributes of a file. This method is
+ called by stat(2) and related system calls.
+
+``listxattr``
+ called by the VFS to list all extended attributes for a given
+ file. This method is called by the listxattr(2) system call.
+
+``update_time``
+ called by the VFS to update a specific time or the i_version of
+ an inode. If this is not defined the VFS will update the inode
+ itself and call mark_inode_dirty_sync.
+
+``atomic_open``
+ called on the last component of an open. Using this optional
+ method the filesystem can look up, possibly create and open the
+ file in one atomic operation. If it wants to leave actual
+ opening to the caller (e.g. if the file turned out to be a
+ symlink, device, or just something filesystem won't do atomic
+ open for), it may signal this by returning finish_no_open(file,
+ dentry). This method is only called if the last component is
+ negative or needs lookup. Cached positive dentries are still
+ handled by f_op->open(). If the file was created, FMODE_CREATED
+ flag should be set in file->f_mode. In case of O_EXCL the
+ method must only succeed if the file didn't exist and hence
+ FMODE_CREATED shall always be set on success.
+
+``tmpfile``
+ called in the end of O_TMPFILE open(). Optional, equivalent to
+ atomically creating, opening and unlinking a file in given
+ directory.
+
+
+The Address Space Object
+========================
+
+The address space object is used to group and manage pages in the page
+cache. It can be used to keep track of the pages in a file (or anything
+else) and also track the mapping of sections of the file into process
+address spaces.
+
+There are a number of distinct yet related services that an
+address-space can provide. These include communicating memory pressure,
+page lookup by address, and keeping track of pages tagged as Dirty or
+Writeback.
+
+The first can be used independently to the others. The VM can try to
+either write dirty pages in order to clean them, or release clean pages
+in order to reuse them. To do this it can call the ->writepage method
+on dirty pages, and ->releasepage on clean pages with PagePrivate set.
+Clean pages without PagePrivate and with no external references will be
+released without notice being given to the address_space.
+
+To achieve this functionality, pages need to be placed on an LRU with
+lru_cache_add and mark_page_active needs to be called whenever the page
+is used.
+
+Pages are normally kept in a radix tree index by ->index. This tree
+maintains information about the PG_Dirty and PG_Writeback status of each
+page, so that pages with either of these flags can be found quickly.
+
+The Dirty tag is primarily used by mpage_writepages - the default
+->writepages method. It uses the tag to find dirty pages to call
+->writepage on. If mpage_writepages is not used (i.e. the address
+provides its own ->writepages) , the PAGECACHE_TAG_DIRTY tag is almost
+unused. write_inode_now and sync_inode do use it (through
+__sync_single_inode) to check if ->writepages has been successful in
+writing out the whole address_space.
+
+The Writeback tag is used by filemap*wait* and sync_page* functions, via
+filemap_fdatawait_range, to wait for all writeback to complete.
+
+An address_space handler may attach extra information to a page,
+typically using the 'private' field in the 'struct page'. If such
+information is attached, the PG_Private flag should be set. This will
+cause various VM routines to make extra calls into the address_space
+handler to deal with that data.
+
+An address space acts as an intermediate between storage and
+application. Data is read into the address space a whole page at a
+time, and provided to the application either by copying of the page, or
+by memory-mapping the page. Data is written into the address space by
+the application, and then written-back to storage typically in whole
+pages, however the address_space has finer control of write sizes.
+
+The read process essentially only requires 'readpage'. The write
+process is more complicated and uses write_begin/write_end or
+set_page_dirty to write data into the address_space, and writepage and
+writepages to writeback data to storage.
+
+Adding and removing pages to/from an address_space is protected by the
+inode's i_mutex.
+
+When data is written to a page, the PG_Dirty flag should be set. It
+typically remains set until writepage asks for it to be written. This
+should clear PG_Dirty and set PG_Writeback. It can be actually written
+at any point after PG_Dirty is clear. Once it is known to be safe,
+PG_Writeback is cleared.
+
+Writeback makes use of a writeback_control structure to direct the
+operations. This gives the the writepage and writepages operations some
+information about the nature of and reason for the writeback request,
+and the constraints under which it is being done. It is also used to
+return information back to the caller about the result of a writepage or
+writepages request.
+
+
+Handling errors during writeback
+--------------------------------
+
+Most applications that do buffered I/O will periodically call a file
+synchronization call (fsync, fdatasync, msync or sync_file_range) to
+ensure that data written has made it to the backing store. When there
+is an error during writeback, they expect that error to be reported when
+a file sync request is made. After an error has been reported on one
+request, subsequent requests on the same file descriptor should return
+0, unless further writeback errors have occurred since the previous file
+syncronization.
+
+Ideally, the kernel would report errors only on file descriptions on
+which writes were done that subsequently failed to be written back. The
+generic pagecache infrastructure does not track the file descriptions
+that have dirtied each individual page however, so determining which
+file descriptors should get back an error is not possible.
+
+Instead, the generic writeback error tracking infrastructure in the
+kernel settles for reporting errors to fsync on all file descriptions
+that were open at the time that the error occurred. In a situation with
+multiple writers, all of them will get back an error on a subsequent
+fsync, even if all of the writes done through that particular file
+descriptor succeeded (or even if there were no writes on that file
+descriptor at all).
+
+Filesystems that wish to use this infrastructure should call
+mapping_set_error to record the error in the address_space when it
+occurs. Then, after writing back data from the pagecache in their
+file->fsync operation, they should call file_check_and_advance_wb_err to
+ensure that the struct file's error cursor has advanced to the correct
+point in the stream of errors emitted by the backing device(s).
+
+
+struct address_space_operations
+-------------------------------
+
+This describes how the VFS can manipulate mapping of a file to page
+cache in your filesystem. The following members are defined:
+
+.. code-block:: c
+
+ struct address_space_operations {
+ int (*writepage)(struct page *page, struct writeback_control *wbc);
+ int (*readpage)(struct file *, struct page *);
+ int (*writepages)(struct address_space *, struct writeback_control *);
+ int (*set_page_dirty)(struct page *page);
+ int (*readpages)(struct file *filp, struct address_space *mapping,
+ struct list_head *pages, unsigned nr_pages);
+ int (*write_begin)(struct file *, struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned flags,
+ struct page **pagep, void **fsdata);
+ int (*write_end)(struct file *, struct address_space *mapping,
+ loff_t pos, unsigned len, unsigned copied,
+ struct page *page, void *fsdata);
+ sector_t (*bmap)(struct address_space *, sector_t);
+ void (*invalidatepage) (struct page *, unsigned int, unsigned int);
+ int (*releasepage) (struct page *, int);
+ void (*freepage)(struct page *);
+ ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
+ /* isolate a page for migration */
+ bool (*isolate_page) (struct page *, isolate_mode_t);
+ /* migrate the contents of a page to the specified target */
+ int (*migratepage) (struct page *, struct page *);
+ /* put migration-failed page back to right list */
+ void (*putback_page) (struct page *);
+ int (*launder_page) (struct page *);
+
+ int (*is_partially_uptodate) (struct page *, unsigned long,
+ unsigned long);
+ void (*is_dirty_writeback) (struct page *, bool *, bool *);
+ int (*error_remove_page) (struct mapping *mapping, struct page *page);
+ int (*swap_activate)(struct file *);
+ int (*swap_deactivate)(struct file *);
+ };
+
+``writepage``
+ called by the VM to write a dirty page to backing store. This
+ may happen for data integrity reasons (i.e. 'sync'), or to free
+ up memory (flush). The difference can be seen in
+ wbc->sync_mode. The PG_Dirty flag has been cleared and
+ PageLocked is true. writepage should start writeout, should set
+ PG_Writeback, and should make sure the page is unlocked, either
+ synchronously or asynchronously when the write operation
+ completes.
+
+ If wbc->sync_mode is WB_SYNC_NONE, ->writepage doesn't have to
+ try too hard if there are problems, and may choose to write out
+ other pages from the mapping if that is easier (e.g. due to
+ internal dependencies). If it chooses not to start writeout, it
+ should return AOP_WRITEPAGE_ACTIVATE so that the VM will not
+ keep calling ->writepage on that page.
+
+ See the file "Locking" for more details.
+
+``readpage``
+ called by the VM to read a page from backing store. The page
+ will be Locked when readpage is called, and should be unlocked
+ and marked uptodate once the read completes. If ->readpage
+ discovers that it needs to unlock the page for some reason, it
+ can do so, and then return AOP_TRUNCATED_PAGE. In this case,
+ the page will be relocated, relocked and if that all succeeds,
+ ->readpage will be called again.
+
+``writepages``
+ called by the VM to write out pages associated with the
+ address_space object. If wbc->sync_mode is WBC_SYNC_ALL, then
+ the writeback_control will specify a range of pages that must be
+ written out. If it is WBC_SYNC_NONE, then a nr_to_write is
+ given and that many pages should be written if possible. If no
+ ->writepages is given, then mpage_writepages is used instead.
+ This will choose pages from the address space that are tagged as
+ DIRTY and will pass them to ->writepage.
+
+``set_page_dirty``
+ called by the VM to set a page dirty. This is particularly
+ needed if an address space attaches private data to a page, and
+ that data needs to be updated when a page is dirtied. This is
+ called, for example, when a memory mapped page gets modified.
+ If defined, it should set the PageDirty flag, and the
+ PAGECACHE_TAG_DIRTY tag in the radix tree.
+
+``readpages``
+ called by the VM to read pages associated with the address_space
+ object. This is essentially just a vector version of readpage.
+ Instead of just one page, several pages are requested.
+ readpages is only used for read-ahead, so read errors are
+ ignored. If anything goes wrong, feel free to give up.
+
+``write_begin``
+ Called by the generic buffered write code to ask the filesystem
+ to prepare to write len bytes at the given offset in the file.
+ The address_space should check that the write will be able to
+ complete, by allocating space if necessary and doing any other
+ internal housekeeping. If the write will update parts of any
+ basic-blocks on storage, then those blocks should be pre-read
+ (if they haven't been read already) so that the updated blocks
+ can be written out properly.
+
+ The filesystem must return the locked pagecache page for the
+ specified offset, in ``*pagep``, for the caller to write into.
+
+ It must be able to cope with short writes (where the length
+ passed to write_begin is greater than the number of bytes copied
+ into the page).
+
+ flags is a field for AOP_FLAG_xxx flags, described in
+ include/linux/fs.h.
+
+ A void * may be returned in fsdata, which then gets passed into
+ write_end.
+
+ Returns 0 on success; < 0 on failure (which is the error code),
+ in which case write_end is not called.
+
+``write_end``
+ After a successful write_begin, and data copy, write_end must be
+ called. len is the original len passed to write_begin, and
+ copied is the amount that was able to be copied.
+
+ The filesystem must take care of unlocking the page and
+ releasing it refcount, and updating i_size.
+
+ Returns < 0 on failure, otherwise the number of bytes (<=
+ 'copied') that were able to be copied into pagecache.
+
+``bmap``
+ called by the VFS to map a logical block offset within object to
+ physical block number. This method is used by the FIBMAP ioctl
+ and for working with swap-files. To be able to swap to a file,
+ the file must have a stable mapping to a block device. The swap
+ system does not go through the filesystem but instead uses bmap
+ to find out where the blocks in the file are and uses those
+ addresses directly.
+
+``invalidatepage``
+ If a page has PagePrivate set, then invalidatepage will be
+ called when part or all of the page is to be removed from the
+ address space. This generally corresponds to either a
+ truncation, punch hole or a complete invalidation of the address
+ space (in the latter case 'offset' will always be 0 and 'length'
+ will be PAGE_SIZE). Any private data associated with the page
+ should be updated to reflect this truncation. If offset is 0
+ and length is PAGE_SIZE, then the private data should be
+ released, because the page must be able to be completely
+ discarded. This may be done by calling the ->releasepage
+ function, but in this case the release MUST succeed.
+
+``releasepage``
+ releasepage is called on PagePrivate pages to indicate that the
+ page should be freed if possible. ->releasepage should remove
+ any private data from the page and clear the PagePrivate flag.
+ If releasepage() fails for some reason, it must indicate failure
+ with a 0 return value. releasepage() is used in two distinct
+ though related cases. The first is when the VM finds a clean
+ page with no active users and wants to make it a free page. If
+ ->releasepage succeeds, the page will be removed from the
+ address_space and become free.
+
+ The second case is when a request has been made to invalidate
+ some or all pages in an address_space. This can happen through
+ the fadvise(POSIX_FADV_DONTNEED) system call or by the
+ filesystem explicitly requesting it as nfs and 9fs do (when they
+ believe the cache may be out of date with storage) by calling
+ invalidate_inode_pages2(). If the filesystem makes such a call,
+ and needs to be certain that all pages are invalidated, then its
+ releasepage will need to ensure this. Possibly it can clear the
+ PageUptodate bit if it cannot free private data yet.
+
+``freepage``
+ freepage is called once the page is no longer visible in the
+ page cache in order to allow the cleanup of any private data.
+ Since it may be called by the memory reclaimer, it should not
+ assume that the original address_space mapping still exists, and
+ it should not block.
+
+``direct_IO``
+ called by the generic read/write routines to perform direct_IO -
+ that is IO requests which bypass the page cache and transfer
+ data directly between the storage and the application's address
+ space.
+
+``isolate_page``
+ Called by the VM when isolating a movable non-lru page. If page
+ is successfully isolated, VM marks the page as PG_isolated via
+ __SetPageIsolated.
+
+``migrate_page``
+ This is used to compact the physical memory usage. If the VM
+ wants to relocate a page (maybe off a memory card that is
+ signalling imminent failure) it will pass a new page and an old
+ page to this function. migrate_page should transfer any private
+ data across and update any references that it has to the page.
+
+``putback_page``
+ Called by the VM when isolated page's migration fails.
+
+``launder_page``
+ Called before freeing a page - it writes back the dirty page.
+ To prevent redirtying the page, it is kept locked during the
+ whole operation.
+
+``is_partially_uptodate``
+ Called by the VM when reading a file through the pagecache when
+ the underlying blocksize != pagesize. If the required block is
+ up to date then the read can complete without needing the IO to
+ bring the whole page up to date.
+
+``is_dirty_writeback``
+ Called by the VM when attempting to reclaim a page. The VM uses
+ dirty and writeback information to determine if it needs to
+ stall to allow flushers a chance to complete some IO.
+ Ordinarily it can use PageDirty and PageWriteback but some
+ filesystems have more complex state (unstable pages in NFS
+ prevent reclaim) or do not set those flags due to locking
+ problems. This callback allows a filesystem to indicate to the
+ VM if a page should be treated as dirty or writeback for the
+ purposes of stalling.
+
+``error_remove_page``
+ normally set to generic_error_remove_page if truncation is ok
+ for this address space. Used for memory failure handling.
+ Setting this implies you deal with pages going away under you,
+ unless you have them locked or reference counts increased.
+
+``swap_activate``
+ Called when swapon is used on a file to allocate space if
+ necessary and pin the block lookup information in memory. A
+ return value of zero indicates success, in which case this file
+ can be used to back swapspace.
+
+``swap_deactivate``
+ Called during swapoff on files where swap_activate was
+ successful.
+
+
+The File Object
+===============
+
+A file object represents a file opened by a process. This is also known
+as an "open file description" in POSIX parlance.
+
+
+struct file_operations
+----------------------
+
+This describes how the VFS can manipulate an open file. As of kernel
+4.18, the following members are defined:
+
+.. code-block:: c
+
+ struct file_operations {
+ struct module *owner;
+ loff_t (*llseek) (struct file *, loff_t, int);
+ ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
+ ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
+ ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
+ ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
+ int (*iopoll)(struct kiocb *kiocb, bool spin);
+ int (*iterate) (struct file *, struct dir_context *);
+ int (*iterate_shared) (struct file *, struct dir_context *);
+ __poll_t (*poll) (struct file *, struct poll_table_struct *);
+ long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long);
+ long (*compat_ioctl) (struct file *, unsigned int, unsigned long);
+ int (*mmap) (struct file *, struct vm_area_struct *);
+ int (*open) (struct inode *, struct file *);
+ int (*flush) (struct file *, fl_owner_t id);
+ int (*release) (struct inode *, struct file *);
+ int (*fsync) (struct file *, loff_t, loff_t, int datasync);
+ int (*fasync) (int, struct file *, int);
+ int (*lock) (struct file *, int, struct file_lock *);
+ ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int);
+ unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long);
+ int (*check_flags)(int);
+ int (*flock) (struct file *, int, struct file_lock *);
+ ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int);
+ ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int);
+ int (*setlease)(struct file *, long, struct file_lock **, void **);
+ long (*fallocate)(struct file *file, int mode, loff_t offset,
+ loff_t len);
+ void (*show_fdinfo)(struct seq_file *m, struct file *f);
+ #ifndef CONFIG_MMU
+ unsigned (*mmap_capabilities)(struct file *);
+ #endif
+ ssize_t (*copy_file_range)(struct file *, loff_t, struct file *, loff_t, size_t, unsigned int);
+ loff_t (*remap_file_range)(struct file *file_in, loff_t pos_in,
+ struct file *file_out, loff_t pos_out,
+ loff_t len, unsigned int remap_flags);
+ int (*fadvise)(struct file *, loff_t, loff_t, int);
+ };
+
+Again, all methods are called without any locks being held, unless
+otherwise noted.
+
+``llseek``
+ called when the VFS needs to move the file position index
+
+``read``
+ called by read(2) and related system calls
+
+``read_iter``
+ possibly asynchronous read with iov_iter as destination
+
+``write``
+ called by write(2) and related system calls
+
+``write_iter``
+ possibly asynchronous write with iov_iter as source
+
+``iopoll``
+ called when aio wants to poll for completions on HIPRI iocbs
+
+``iterate``
+ called when the VFS needs to read the directory contents
+
+``iterate_shared``
+ called when the VFS needs to read the directory contents when
+ filesystem supports concurrent dir iterators
+
+``poll``
+ called by the VFS when a process wants to check if there is
+ activity on this file and (optionally) go to sleep until there
+ is activity. Called by the select(2) and poll(2) system calls
+
+``unlocked_ioctl``
+ called by the ioctl(2) system call.
+
+``compat_ioctl``
+ called by the ioctl(2) system call when 32 bit system calls are
+ used on 64 bit kernels.
+
+``mmap``
+ called by the mmap(2) system call
+
+``open``
+ called by the VFS when an inode should be opened. When the VFS
+ opens a file, it creates a new "struct file". It then calls the
+ open method for the newly allocated file structure. You might
+ think that the open method really belongs in "struct
+ inode_operations", and you may be right. I think it's done the
+ way it is because it makes filesystems simpler to implement.
+ The open() method is a good place to initialize the
+ "private_data" member in the file structure if you want to point
+ to a device structure
+
+``flush``
+ called by the close(2) system call to flush a file
+
+``release``
+ called when the last reference to an open file is closed
+
+``fsync``
+ called by the fsync(2) system call. Also see the section above
+ entitled "Handling errors during writeback".
+
+``fasync``
+ called by the fcntl(2) system call when asynchronous
+ (non-blocking) mode is enabled for a file
+
+``lock``
+ called by the fcntl(2) system call for F_GETLK, F_SETLK, and
+ F_SETLKW commands
+
+``get_unmapped_area``
+ called by the mmap(2) system call
+
+``check_flags``
+ called by the fcntl(2) system call for F_SETFL command
+
+``flock``
+ called by the flock(2) system call
+
+``splice_write``
+ called by the VFS to splice data from a pipe to a file. This
+ method is used by the splice(2) system call
+
+``splice_read``
+ called by the VFS to splice data from file to a pipe. This
+ method is used by the splice(2) system call
+
+``setlease``
+ called by the VFS to set or release a file lock lease. setlease
+ implementations should call generic_setlease to record or remove
+ the lease in the inode after setting it.
+
+``fallocate``
+ called by the VFS to preallocate blocks or punch a hole.
+
+``copy_file_range``
+ called by the copy_file_range(2) system call.
+
+``remap_file_range``
+ called by the ioctl(2) system call for FICLONERANGE and FICLONE
+ and FIDEDUPERANGE commands to remap file ranges. An
+ implementation should remap len bytes at pos_in of the source
+ file into the dest file at pos_out. Implementations must handle
+ callers passing in len == 0; this means "remap to the end of the
+ source file". The return value should the number of bytes
+ remapped, or the usual negative error code if errors occurred
+ before any bytes were remapped. The remap_flags parameter
+ accepts REMAP_FILE_* flags. If REMAP_FILE_DEDUP is set then the
+ implementation must only remap if the requested file ranges have
+ identical contents. If REMAP_CAN_SHORTEN is set, the caller is
+ ok with the implementation shortening the request length to
+ satisfy alignment or EOF requirements (or any other reason).
+
+``fadvise``
+ possibly called by the fadvise64() system call.
+
+Note that the file operations are implemented by the specific
+filesystem in which the inode resides. When opening a device node
+(character or block special) most filesystems will call special
+support routines in the VFS which will locate the required device
+driver information. These support routines replace the filesystem file
+operations with those for the device driver, and then proceed to call
+the new open() method for the file. This is how opening a device file
+in the filesystem eventually ends up calling the device driver open()
+method.
+
+
+Directory Entry Cache (dcache)
+==============================
+
+
+struct dentry_operations
+------------------------
+
+This describes how a filesystem can overload the standard dentry
+operations. Dentries and the dcache are the domain of the VFS and the
+individual filesystem implementations. Device drivers have no business
+here. These methods may be set to NULL, as they are either optional or
+the VFS uses a default. As of kernel 2.6.22, the following members are
+defined:
+
+.. code-block:: c
+
+ struct dentry_operations {
+ int (*d_revalidate)(struct dentry *, unsigned int);
+ int (*d_weak_revalidate)(struct dentry *, unsigned int);
+ int (*d_hash)(const struct dentry *, struct qstr *);
+ int (*d_compare)(const struct dentry *,
+ unsigned int, const char *, const struct qstr *);
+ int (*d_delete)(const struct dentry *);
+ int (*d_init)(struct dentry *);
+ void (*d_release)(struct dentry *);
+ void (*d_iput)(struct dentry *, struct inode *);
+ char *(*d_dname)(struct dentry *, char *, int);
+ struct vfsmount *(*d_automount)(struct path *);
+ int (*d_manage)(const struct path *, bool);
+ struct dentry *(*d_real)(struct dentry *, const struct inode *);
+ };
+
+``d_revalidate``
+ called when the VFS needs to revalidate a dentry. This is
+ called whenever a name look-up finds a dentry in the dcache.
+ Most local filesystems leave this as NULL, because all their
+ dentries in the dcache are valid. Network filesystems are
+ different since things can change on the server without the
+ client necessarily being aware of it.
+
+ This function should return a positive value if the dentry is
+ still valid, and zero or a negative error code if it isn't.
+
+ d_revalidate may be called in rcu-walk mode (flags &
+ LOOKUP_RCU). If in rcu-walk mode, the filesystem must
+ revalidate the dentry without blocking or storing to the dentry,
+ d_parent and d_inode should not be used without care (because
+ they can change and, in d_inode case, even become NULL under
+ us).
+
+ If a situation is encountered that rcu-walk cannot handle,
+ return
+ -ECHILD and it will be called again in ref-walk mode.
+
+``_weak_revalidate``
+ called when the VFS needs to revalidate a "jumped" dentry. This
+ is called when a path-walk ends at dentry that was not acquired
+ by doing a lookup in the parent directory. This includes "/",
+ "." and "..", as well as procfs-style symlinks and mountpoint
+ traversal.
+
+ In this case, we are less concerned with whether the dentry is
+ still fully correct, but rather that the inode is still valid.
+ As with d_revalidate, most local filesystems will set this to
+ NULL since their dcache entries are always valid.
+
+ This function has the same return code semantics as
+ d_revalidate.
+
+ d_weak_revalidate is only called after leaving rcu-walk mode.
+
+``d_hash``
+ called when the VFS adds a dentry to the hash table. The first
+ dentry passed to d_hash is the parent directory that the name is
+ to be hashed into.
+
+ Same locking and synchronisation rules as d_compare regarding
+ what is safe to dereference etc.
+
+``d_compare``
+ called to compare a dentry name with a given name. The first
+ dentry is the parent of the dentry to be compared, the second is
+ the child dentry. len and name string are properties of the
+ dentry to be compared. qstr is the name to compare it with.
+
+ Must be constant and idempotent, and should not take locks if
+ possible, and should not or store into the dentry. Should not
+ dereference pointers outside the dentry without lots of care
+ (eg. d_parent, d_inode, d_name should not be used).
+
+ However, our vfsmount is pinned, and RCU held, so the dentries
+ and inodes won't disappear, neither will our sb or filesystem
+ module. ->d_sb may be used.
+
+ It is a tricky calling convention because it needs to be called
+ under "rcu-walk", ie. without any locks or references on things.
+
+``d_delete``
+ called when the last reference to a dentry is dropped and the
+ dcache is deciding whether or not to cache it. Return 1 to
+ delete immediately, or 0 to cache the dentry. Default is NULL
+ which means to always cache a reachable dentry. d_delete must
+ be constant and idempotent.
+
+``d_init``
+ called when a dentry is allocated
+
+``d_release``
+ called when a dentry is really deallocated
+
+``d_iput``
+ called when a dentry loses its inode (just prior to its being
+ deallocated). The default when this is NULL is that the VFS
+ calls iput(). If you define this method, you must call iput()
+ yourself
+
+``d_dname``
+ called when the pathname of a dentry should be generated.
+ Useful for some pseudo filesystems (sockfs, pipefs, ...) to
+ delay pathname generation. (Instead of doing it when dentry is
+ created, it's done only when the path is needed.). Real
+ filesystems probably dont want to use it, because their dentries
+ are present in global dcache hash, so their hash should be an
+ invariant. As no lock is held, d_dname() should not try to
+ modify the dentry itself, unless appropriate SMP safety is used.
+ CAUTION : d_path() logic is quite tricky. The correct way to
+ return for example "Hello" is to put it at the end of the
+ buffer, and returns a pointer to the first char.
+ dynamic_dname() helper function is provided to take care of
+ this.
+
+ Example :
+
+.. code-block:: c
+
+ static char *pipefs_dname(struct dentry *dent, char *buffer, int buflen)
+ {
+ return dynamic_dname(dentry, buffer, buflen, "pipe:[%lu]",
+ dentry->d_inode->i_ino);
+ }
+
+``d_automount``
+ called when an automount dentry is to be traversed (optional).
+ This should create a new VFS mount record and return the record
+ to the caller. The caller is supplied with a path parameter
+ giving the automount directory to describe the automount target
+ and the parent VFS mount record to provide inheritable mount
+ parameters. NULL should be returned if someone else managed to
+ make the automount first. If the vfsmount creation failed, then
+ an error code should be returned. If -EISDIR is returned, then
+ the directory will be treated as an ordinary directory and
+ returned to pathwalk to continue walking.
+
+ If a vfsmount is returned, the caller will attempt to mount it
+ on the mountpoint and will remove the vfsmount from its
+ expiration list in the case of failure. The vfsmount should be
+ returned with 2 refs on it to prevent automatic expiration - the
+ caller will clean up the additional ref.
+
+ This function is only used if DCACHE_NEED_AUTOMOUNT is set on
+ the dentry. This is set by __d_instantiate() if S_AUTOMOUNT is
+ set on the inode being added.
+
+``d_manage``
+ called to allow the filesystem to manage the transition from a
+ dentry (optional). This allows autofs, for example, to hold up
+ clients waiting to explore behind a 'mountpoint' while letting
+ the daemon go past and construct the subtree there. 0 should be
+ returned to let the calling process continue. -EISDIR can be
+ returned to tell pathwalk to use this directory as an ordinary
+ directory and to ignore anything mounted on it and not to check
+ the automount flag. Any other error code will abort pathwalk
+ completely.
+
+ If the 'rcu_walk' parameter is true, then the caller is doing a
+ pathwalk in RCU-walk mode. Sleeping is not permitted in this
+ mode, and the caller can be asked to leave it and call again by
+ returning -ECHILD. -EISDIR may also be returned to tell
+ pathwalk to ignore d_automount or any mounts.
+
+ This function is only used if DCACHE_MANAGE_TRANSIT is set on
+ the dentry being transited from.
+
+``d_real``
+ overlay/union type filesystems implement this method to return
+ one of the underlying dentries hidden by the overlay. It is
+ used in two different modes:
+
+ Called from file_dentry() it returns the real dentry matching
+ the inode argument. The real dentry may be from a lower layer
+ already copied up, but still referenced from the file. This
+ mode is selected with a non-NULL inode argument.
+
+ With NULL inode the topmost real underlying dentry is returned.
+
+Each dentry has a pointer to its parent dentry, as well as a hash list
+of child dentries. Child dentries are basically like files in a
+directory.
+
+
+Directory Entry Cache API
+--------------------------
+
+There are a number of functions defined which permit a filesystem to
+manipulate dentries:
+
+``dget``
+ open a new handle for an existing dentry (this just increments
+ the usage count)
+
+``dput``
+ close a handle for a dentry (decrements the usage count). If
+ the usage count drops to 0, and the dentry is still in its
+ parent's hash, the "d_delete" method is called to check whether
+ it should be cached. If it should not be cached, or if the
+ dentry is not hashed, it is deleted. Otherwise cached dentries
+ are put into an LRU list to be reclaimed on memory shortage.
+
+``d_drop``
+ this unhashes a dentry from its parents hash list. A subsequent
+ call to dput() will deallocate the dentry if its usage count
+ drops to 0
+
+``d_delete``
+ delete a dentry. If there are no other open references to the
+ dentry then the dentry is turned into a negative dentry (the
+ d_iput() method is called). If there are other references, then
+ d_drop() is called instead
+
+``d_add``
+ add a dentry to its parents hash list and then calls
+ d_instantiate()
+
+``d_instantiate``
+ add a dentry to the alias hash list for the inode and updates
+ the "d_inode" member. The "i_count" member in the inode
+ structure should be set/incremented. If the inode pointer is
+ NULL, the dentry is called a "negative dentry". This function
+ is commonly called when an inode is created for an existing
+ negative dentry
+
+``d_lookup``
+ look up a dentry given its parent and path name component It
+ looks up the child of that given name from the dcache hash
+ table. If it is found, the reference count is incremented and
+ the dentry is returned. The caller must use dput() to free the
+ dentry when it finishes using it.
+
+
+Mount Options
+=============
+
+
+Parsing options
+---------------
+
+On mount and remount the filesystem is passed a string containing a
+comma separated list of mount options. The options can have either of
+these forms:
+
+ option
+ option=value
+
+The <linux/parser.h> header defines an API that helps parse these
+options. There are plenty of examples on how to use it in existing
+filesystems.
+
+
+Showing options
+---------------
+
+If a filesystem accepts mount options, it must define show_options() to
+show all the currently active options. The rules are:
+
+ - options MUST be shown which are not default or their values differ
+ from the default
+
+ - options MAY be shown which are enabled by default or have their
+ default value
+
+Options used only internally between a mount helper and the kernel (such
+as file descriptors), or which only have an effect during the mounting
+(such as ones controlling the creation of a journal) are exempt from the
+above rules.
+
+The underlying reason for the above rules is to make sure, that a mount
+can be accurately replicated (e.g. umounting and mounting again) based
+on the information found in /proc/mounts.
+
+
+Resources
+=========
+
+(Note some of these resources are not up-to-date with the latest kernel
+ version.)
+
+Creating Linux virtual filesystems. 2002
+ <http://lwn.net/Articles/13325/>
+
+The Linux Virtual File-system Layer by Neil Brown. 1999
+ <http://www.cse.unsw.edu.au/~neilb/oss/linux-commentary/vfs.html>
+
+A tour of the Linux VFS by Michael K. Johnson. 1996
+ <http://www.tldp.org/LDP/khg/HyperNews/get/fs/vfstour.html>
+
+A small trail through the Linux kernel by Andries Brouwer. 2001
+ <http://www.win.tue.nl/~aeb/linux/vfs/trail.html>
diff --git a/Documentation/filesystems/vfs.txt b/Documentation/filesystems/vfs.txt
deleted file mode 100644
index a6c6a8a..0000000
--- a/Documentation/filesystems/vfs.txt
+++ /dev/null
@@ -1,1253 +0,0 @@
-
- Overview of the Linux Virtual File System
-
- Original author: Richard Gooch <rgooch@atnf.csiro.au>
-
- Last updated on June 24, 2007.
-
- Copyright (C) 1999 Richard Gooch
- Copyright (C) 2005 Pekka Enberg
-
- This file is released under the GPLv2.
-
-
-Introduction
-============
-
-The Virtual File System (also known as the Virtual Filesystem Switch)
-is the software layer in the kernel that provides the filesystem
-interface to userspace programs. It also provides an abstraction
-within the kernel which allows different filesystem implementations to
-coexist.
-
-VFS system calls open(2), stat(2), read(2), write(2), chmod(2) and so
-on are called from a process context. Filesystem locking is described
-in the document Documentation/filesystems/Locking.
-
-
-Directory Entry Cache (dcache)
-------------------------------
-
-The VFS implements the open(2), stat(2), chmod(2), and similar system
-calls. The pathname argument that is passed to them is used by the VFS
-to search through the directory entry cache (also known as the dentry
-cache or dcache). This provides a very fast look-up mechanism to
-translate a pathname (filename) into a specific dentry. Dentries live
-in RAM and are never saved to disc: they exist only for performance.
-
-The dentry cache is meant to be a view into your entire filespace. As
-most computers cannot fit all dentries in the RAM at the same time,
-some bits of the cache are missing. In order to resolve your pathname
-into a dentry, the VFS may have to resort to creating dentries along
-the way, and then loading the inode. This is done by looking up the
-inode.
-
-
-The Inode Object
-----------------
-
-An individual dentry usually has a pointer to an inode. Inodes are
-filesystem objects such as regular files, directories, FIFOs and other
-beasts. They live either on the disc (for block device filesystems)
-or in the memory (for pseudo filesystems). Inodes that live on the
-disc are copied into the memory when required and changes to the inode
-are written back to disc. A single inode can be pointed to by multiple
-dentries (hard links, for example, do this).
-
-To look up an inode requires that the VFS calls the lookup() method of
-the parent directory inode. This method is installed by the specific
-filesystem implementation that the inode lives in. Once the VFS has
-the required dentry (and hence the inode), we can do all those boring
-things like open(2) the file, or stat(2) it to peek at the inode
-data. The stat(2) operation is fairly simple: once the VFS has the
-dentry, it peeks at the inode data and passes some of it back to
-userspace.
-
-
-The File Object
----------------
-
-Opening a file requires another operation: allocation of a file
-structure (this is the kernel-side implementation of file
-descriptors). The freshly allocated file structure is initialized with
-a pointer to the dentry and a set of file operation member functions.
-These are taken from the inode data. The open() file method is then
-called so the specific filesystem implementation can do its work. You
-can see that this is another switch performed by the VFS. The file
-structure is placed into the file descriptor table for the process.
-
-Reading, writing and closing files (and other assorted VFS operations)
-is done by using the userspace file descriptor to grab the appropriate
-file structure, and then calling the required file structure method to
-do whatever is required. For as long as the file is open, it keeps the
-dentry in use, which in turn means that the VFS inode is still in use.
-
-
-Registering and Mounting a Filesystem
-=====================================
-
-To register and unregister a filesystem, use the following API
-functions:
-
- #include <linux/fs.h>
-
- extern int register_filesystem(struct file_system_type *);
- extern int unregister_filesystem(struct file_system_type *);
-
-The passed struct file_system_type describes your filesystem. When a
-request is made to mount a filesystem onto a directory in your namespace,
-the VFS will call the appropriate mount() method for the specific
-filesystem. New vfsmount referring to the tree returned by ->mount()
-will be attached to the mountpoint, so that when pathname resolution
-reaches the mountpoint it will jump into the root of that vfsmount.
-
-You can see all filesystems that are registered to the kernel in the
-file /proc/filesystems.
-
-
-struct file_system_type
------------------------
-
-This describes the filesystem. As of kernel 2.6.39, the following
-members are defined:
-
-struct file_system_type {
- const char *name;
- int fs_flags;
- struct dentry *(*mount) (struct file_system_type *, int,
- const char *, void *);
- void (*kill_sb) (struct super_block *);
- struct module *owner;
- struct file_system_type * next;
- struct list_head fs_supers;
- struct lock_class_key s_lock_key;
- struct lock_class_key s_umount_key;
-};
-
- name: the name of the filesystem type, such as "ext2", "iso9660",
- "msdos" and so on
-
- fs_flags: various flags (i.e. FS_REQUIRES_DEV, FS_NO_DCACHE, etc.)
-
- mount: the method to call when a new instance of this
- filesystem should be mounted
-
- kill_sb: the method to call when an instance of this filesystem
- should be shut down
-
- owner: for internal VFS use: you should initialize this to THIS_MODULE in
- most cases.
-
- next: for internal VFS use: you should initialize this to NULL
-
- s_lock_key, s_umount_key: lockdep-specific
-
-The mount() method has the following arguments:
-
- struct file_system_type *fs_type: describes the filesystem, partly initialized
- by the specific filesystem code
-
- int flags: mount flags
-
- const char *dev_name: the device name we are mounting.
-
- void *data: arbitrary mount options, usually comes as an ASCII
- string (see "Mount Options" section)
-
-The mount() method must return the root dentry of the tree requested by
-caller. An active reference to its superblock must be grabbed and the
-superblock must be locked. On failure it should return ERR_PTR(error).
-
-The arguments match those of mount(2) and their interpretation
-depends on filesystem type. E.g. for block filesystems, dev_name is
-interpreted as block device name, that device is opened and if it
-contains a suitable filesystem image the method creates and initializes
-struct super_block accordingly, returning its root dentry to caller.
-
-->mount() may choose to return a subtree of existing filesystem - it
-doesn't have to create a new one. The main result from the caller's
-point of view is a reference to dentry at the root of (sub)tree to
-be attached; creation of new superblock is a common side effect.
-
-The most interesting member of the superblock structure that the
-mount() method fills in is the "s_op" field. This is a pointer to
-a "struct super_operations" which describes the next level of the
-filesystem implementation.
-
-Usually, a filesystem uses one of the generic mount() implementations
-and provides a fill_super() callback instead. The generic variants are:
-
- mount_bdev: mount a filesystem residing on a block device
-
- mount_nodev: mount a filesystem that is not backed by a device
-
- mount_single: mount a filesystem which shares the instance between
- all mounts
-
-A fill_super() callback implementation has the following arguments:
-
- struct super_block *sb: the superblock structure. The callback
- must initialize this properly.
-
- void *data: arbitrary mount options, usually comes as an ASCII
- string (see "Mount Options" section)
-
- int silent: whether or not to be silent on error
-
-
-The Superblock Object
-=====================
-
-A superblock object represents a mounted filesystem.
-
-
-struct super_operations
------------------------
-
-This describes how the VFS can manipulate the superblock of your
-filesystem. As of kernel 2.6.22, the following members are defined:
-
-struct super_operations {
- struct inode *(*alloc_inode)(struct super_block *sb);
- void (*destroy_inode)(struct inode *);
-
- void (*dirty_inode) (struct inode *, int flags);
- int (*write_inode) (struct inode *, int);
- void (*drop_inode) (struct inode *);
- void (*delete_inode) (struct inode *);
- void (*put_super) (struct super_block *);
- int (*sync_fs)(struct super_block *sb, int wait);
- int (*freeze_fs) (struct super_block *);
- int (*unfreeze_fs) (struct super_block *);
- int (*statfs) (struct dentry *, struct kstatfs *);
- int (*remount_fs) (struct super_block *, int *, char *);
- void (*clear_inode) (struct inode *);
- void (*umount_begin) (struct super_block *);
-
- int (*show_options)(struct seq_file *, struct dentry *);
-
- ssize_t (*quota_read)(struct super_block *, int, char *, size_t, loff_t);
- ssize_t (*quota_write)(struct super_block *, int, const char *, size_t, loff_t);
- int (*nr_cached_objects)(struct super_block *);
- void (*free_cached_objects)(struct super_block *, int);
-};
-
-All methods are called without any locks being held, unless otherwise
-noted. This means that most methods can block safely. All methods are
-only called from a process context (i.e. not from an interrupt handler
-or bottom half).
-
- alloc_inode: this method is called by alloc_inode() to allocate memory
- for struct inode and initialize it. If this function is not
- defined, a simple 'struct inode' is allocated. Normally
- alloc_inode will be used to allocate a larger structure which
- contains a 'struct inode' embedded within it.
-
- destroy_inode: this method is called by destroy_inode() to release
- resources allocated for struct inode. It is only required if
- ->alloc_inode was defined and simply undoes anything done by
- ->alloc_inode.
-
- dirty_inode: this method is called by the VFS to mark an inode dirty.
-
- write_inode: this method is called when the VFS needs to write an
- inode to disc. The second parameter indicates whether the write
- should be synchronous or not, not all filesystems check this flag.
-
- drop_inode: called when the last access to the inode is dropped,
- with the inode->i_lock spinlock held.
-
- This method should be either NULL (normal UNIX filesystem
- semantics) or "generic_delete_inode" (for filesystems that do not
- want to cache inodes - causing "delete_inode" to always be
- called regardless of the value of i_nlink)
-
- The "generic_delete_inode()" behavior is equivalent to the
- old practice of using "force_delete" in the put_inode() case,
- but does not have the races that the "force_delete()" approach
- had.
-
- delete_inode: called when the VFS wants to delete an inode
-
- put_super: called when the VFS wishes to free the superblock
- (i.e. unmount). This is called with the superblock lock held
-
- sync_fs: called when VFS is writing out all dirty data associated with
- a superblock. The second parameter indicates whether the method
- should wait until the write out has been completed. Optional.
-
- freeze_fs: called when VFS is locking a filesystem and
- forcing it into a consistent state. This method is currently
- used by the Logical Volume Manager (LVM).
-
- unfreeze_fs: called when VFS is unlocking a filesystem and making it writable
- again.
-
- statfs: called when the VFS needs to get filesystem statistics.
-
- remount_fs: called when the filesystem is remounted. This is called
- with the kernel lock held
-
- clear_inode: called then the VFS clears the inode. Optional
-
- umount_begin: called when the VFS is unmounting a filesystem.
-
- show_options: called by the VFS to show mount options for
- /proc/<pid>/mounts. (see "Mount Options" section)
-
- quota_read: called by the VFS to read from filesystem quota file.
-
- quota_write: called by the VFS to write to filesystem quota file.
-
- nr_cached_objects: called by the sb cache shrinking function for the
- filesystem to return the number of freeable cached objects it contains.
- Optional.
-
- free_cache_objects: called by the sb cache shrinking function for the
- filesystem to scan the number of objects indicated to try to free them.
- Optional, but any filesystem implementing this method needs to also
- implement ->nr_cached_objects for it to be called correctly.
-
- We can't do anything with any errors that the filesystem might
- encountered, hence the void return type. This will never be called if
- the VM is trying to reclaim under GFP_NOFS conditions, hence this
- method does not need to handle that situation itself.
-
- Implementations must include conditional reschedule calls inside any
- scanning loop that is done. This allows the VFS to determine
- appropriate scan batch sizes without having to worry about whether
- implementations will cause holdoff problems due to large scan batch
- sizes.
-
-Whoever sets up the inode is responsible for filling in the "i_op" field. This
-is a pointer to a "struct inode_operations" which describes the methods that
-can be performed on individual inodes.
-
-struct xattr_handlers
----------------------
-
-On filesystems that support extended attributes (xattrs), the s_xattr
-superblock field points to a NULL-terminated array of xattr handlers. Extended
-attributes are name:value pairs.
-
- name: Indicates that the handler matches attributes with the specified name
- (such as "system.posix_acl_access"); the prefix field must be NULL.
-
- prefix: Indicates that the handler matches all attributes with the specified
- name prefix (such as "user."); the name field must be NULL.
-
- list: Determine if attributes matching this xattr handler should be listed
- for a particular dentry. Used by some listxattr implementations like
- generic_listxattr.
-
- get: Called by the VFS to get the value of a particular extended attribute.
- This method is called by the getxattr(2) system call.
-
- set: Called by the VFS to set the value of a particular extended attribute.
- When the new value is NULL, called to remove a particular extended
- attribute. This method is called by the the setxattr(2) and
- removexattr(2) system calls.
-
-When none of the xattr handlers of a filesystem match the specified attribute
-name or when a filesystem doesn't support extended attributes, the various
-*xattr(2) system calls return -EOPNOTSUPP.
-
-
-The Inode Object
-================
-
-An inode object represents an object within the filesystem.
-
-
-struct inode_operations
------------------------
-
-This describes how the VFS can manipulate an inode in your
-filesystem. As of kernel 2.6.22, the following members are defined:
-
-struct inode_operations {
- int (*create) (struct inode *,struct dentry *, umode_t, bool);
- struct dentry * (*lookup) (struct inode *,struct dentry *, unsigned int);
- int (*link) (struct dentry *,struct inode *,struct dentry *);
- int (*unlink) (struct inode *,struct dentry *);
- int (*symlink) (struct inode *,struct dentry *,const char *);
- int (*mkdir) (struct inode *,struct dentry *,umode_t);
- int (*rmdir) (struct inode *,struct dentry *);
- int (*mknod) (struct inode *,struct dentry *,umode_t,dev_t);
- int (*rename) (struct inode *, struct dentry *,
- struct inode *, struct dentry *, unsigned int);
- int (*readlink) (struct dentry *, char __user *,int);
- const char *(*get_link) (struct dentry *, struct inode *,
- struct delayed_call *);
- int (*permission) (struct inode *, int);
- int (*get_acl)(struct inode *, int);
- int (*setattr) (struct dentry *, struct iattr *);
- int (*getattr) (const struct path *, struct kstat *, u32, unsigned int);
- ssize_t (*listxattr) (struct dentry *, char *, size_t);
- void (*update_time)(struct inode *, struct timespec *, int);
- int (*atomic_open)(struct inode *, struct dentry *, struct file *,
- unsigned open_flag, umode_t create_mode);
- int (*tmpfile) (struct inode *, struct dentry *, umode_t);
-};
-
-Again, all methods are called without any locks being held, unless
-otherwise noted.
-
- create: called by the open(2) and creat(2) system calls. Only
- required if you want to support regular files. The dentry you
- get should not have an inode (i.e. it should be a negative
- dentry). Here you will probably call d_instantiate() with the
- dentry and the newly created inode
-
- lookup: called when the VFS needs to look up an inode in a parent
- directory. The name to look for is found in the dentry. This
- method must call d_add() to insert the found inode into the
- dentry. The "i_count" field in the inode structure should be
- incremented. If the named inode does not exist a NULL inode
- should be inserted into the dentry (this is called a negative
- dentry). Returning an error code from this routine must only
- be done on a real error, otherwise creating inodes with system
- calls like create(2), mknod(2), mkdir(2) and so on will fail.
- If you wish to overload the dentry methods then you should
- initialise the "d_dop" field in the dentry; this is a pointer
- to a struct "dentry_operations".
- This method is called with the directory inode semaphore held
-
- link: called by the link(2) system call. Only required if you want
- to support hard links. You will probably need to call
- d_instantiate() just as you would in the create() method
-
- unlink: called by the unlink(2) system call. Only required if you
- want to support deleting inodes
-
- symlink: called by the symlink(2) system call. Only required if you
- want to support symlinks. You will probably need to call
- d_instantiate() just as you would in the create() method
-
- mkdir: called by the mkdir(2) system call. Only required if you want
- to support creating subdirectories. You will probably need to
- call d_instantiate() just as you would in the create() method
-
- rmdir: called by the rmdir(2) system call. Only required if you want
- to support deleting subdirectories
-
- mknod: called by the mknod(2) system call to create a device (char,
- block) inode or a named pipe (FIFO) or socket. Only required
- if you want to support creating these types of inodes. You
- will probably need to call d_instantiate() just as you would
- in the create() method
-
- rename: called by the rename(2) system call to rename the object to
- have the parent and name given by the second inode and dentry.
-
- The filesystem must return -EINVAL for any unsupported or
- unknown flags. Currently the following flags are implemented:
- (1) RENAME_NOREPLACE: this flag indicates that if the target
- of the rename exists the rename should fail with -EEXIST
- instead of replacing the target. The VFS already checks for
- existence, so for local filesystems the RENAME_NOREPLACE
- implementation is equivalent to plain rename.
- (2) RENAME_EXCHANGE: exchange source and target. Both must
- exist; this is checked by the VFS. Unlike plain rename,
- source and target may be of different type.
-
- get_link: called by the VFS to follow a symbolic link to the
- inode it points to. Only required if you want to support
- symbolic links. This method returns the symlink body
- to traverse (and possibly resets the current position with
- nd_jump_link()). If the body won't go away until the inode
- is gone, nothing else is needed; if it needs to be otherwise
- pinned, arrange for its release by having get_link(..., ..., done)
- do set_delayed_call(done, destructor, argument).
- In that case destructor(argument) will be called once VFS is
- done with the body you've returned.
- May be called in RCU mode; that is indicated by NULL dentry
- argument. If request can't be handled without leaving RCU mode,
- have it return ERR_PTR(-ECHILD).
-
- readlink: this is now just an override for use by readlink(2) for the
- cases when ->get_link uses nd_jump_link() or object is not in
- fact a symlink. Normally filesystems should only implement
- ->get_link for symlinks and readlink(2) will automatically use
- that.
-
- permission: called by the VFS to check for access rights on a POSIX-like
- filesystem.
-
- May be called in rcu-walk mode (mask & MAY_NOT_BLOCK). If in rcu-walk
- mode, the filesystem must check the permission without blocking or
- storing to the inode.
-
- If a situation is encountered that rcu-walk cannot handle, return
- -ECHILD and it will be called again in ref-walk mode.
-
- setattr: called by the VFS to set attributes for a file. This method
- is called by chmod(2) and related system calls.
-
- getattr: called by the VFS to get attributes of a file. This method
- is called by stat(2) and related system calls.
-
- listxattr: called by the VFS to list all extended attributes for a
- given file. This method is called by the listxattr(2) system call.
-
- update_time: called by the VFS to update a specific time or the i_version of
- an inode. If this is not defined the VFS will update the inode itself
- and call mark_inode_dirty_sync.
-
- atomic_open: called on the last component of an open. Using this optional
- method the filesystem can look up, possibly create and open the file in
- one atomic operation. If it wants to leave actual opening to the
- caller (e.g. if the file turned out to be a symlink, device, or just
- something filesystem won't do atomic open for), it may signal this by
- returning finish_no_open(file, dentry). This method is only called if
- the last component is negative or needs lookup. Cached positive dentries
- are still handled by f_op->open(). If the file was created,
- FMODE_CREATED flag should be set in file->f_mode. In case of O_EXCL
- the method must only succeed if the file didn't exist and hence FMODE_CREATED
- shall always be set on success.
-
- tmpfile: called in the end of O_TMPFILE open(). Optional, equivalent to
- atomically creating, opening and unlinking a file in given directory.
-
-The Address Space Object
-========================
-
-The address space object is used to group and manage pages in the page
-cache. It can be used to keep track of the pages in a file (or
-anything else) and also track the mapping of sections of the file into
-process address spaces.
-
-There are a number of distinct yet related services that an
-address-space can provide. These include communicating memory
-pressure, page lookup by address, and keeping track of pages tagged as
-Dirty or Writeback.
-
-The first can be used independently to the others. The VM can try to
-either write dirty pages in order to clean them, or release clean
-pages in order to reuse them. To do this it can call the ->writepage
-method on dirty pages, and ->releasepage on clean pages with
-PagePrivate set. Clean pages without PagePrivate and with no external
-references will be released without notice being given to the
-address_space.
-
-To achieve this functionality, pages need to be placed on an LRU with
-lru_cache_add and mark_page_active needs to be called whenever the
-page is used.
-
-Pages are normally kept in a radix tree index by ->index. This tree
-maintains information about the PG_Dirty and PG_Writeback status of
-each page, so that pages with either of these flags can be found
-quickly.
-
-The Dirty tag is primarily used by mpage_writepages - the default
-->writepages method. It uses the tag to find dirty pages to call
-->writepage on. If mpage_writepages is not used (i.e. the address
-provides its own ->writepages) , the PAGECACHE_TAG_DIRTY tag is
-almost unused. write_inode_now and sync_inode do use it (through
-__sync_single_inode) to check if ->writepages has been successful in
-writing out the whole address_space.
-
-The Writeback tag is used by filemap*wait* and sync_page* functions,
-via filemap_fdatawait_range, to wait for all writeback to complete.
-
-An address_space handler may attach extra information to a page,
-typically using the 'private' field in the 'struct page'. If such
-information is attached, the PG_Private flag should be set. This will
-cause various VM routines to make extra calls into the address_space
-handler to deal with that data.
-
-An address space acts as an intermediate between storage and
-application. Data is read into the address space a whole page at a
-time, and provided to the application either by copying of the page,
-or by memory-mapping the page.
-Data is written into the address space by the application, and then
-written-back to storage typically in whole pages, however the
-address_space has finer control of write sizes.
-
-The read process essentially only requires 'readpage'. The write
-process is more complicated and uses write_begin/write_end or
-set_page_dirty to write data into the address_space, and writepage
-and writepages to writeback data to storage.
-
-Adding and removing pages to/from an address_space is protected by the
-inode's i_mutex.
-
-When data is written to a page, the PG_Dirty flag should be set. It
-typically remains set until writepage asks for it to be written. This
-should clear PG_Dirty and set PG_Writeback. It can be actually
-written at any point after PG_Dirty is clear. Once it is known to be
-safe, PG_Writeback is cleared.
-
-Writeback makes use of a writeback_control structure to direct the
-operations. This gives the the writepage and writepages operations some
-information about the nature of and reason for the writeback request,
-and the constraints under which it is being done. It is also used to
-return information back to the caller about the result of a writepage or
-writepages request.
-
-Handling errors during writeback
---------------------------------
-Most applications that do buffered I/O will periodically call a file
-synchronization call (fsync, fdatasync, msync or sync_file_range) to
-ensure that data written has made it to the backing store. When there
-is an error during writeback, they expect that error to be reported when
-a file sync request is made. After an error has been reported on one
-request, subsequent requests on the same file descriptor should return
-0, unless further writeback errors have occurred since the previous file
-syncronization.
-
-Ideally, the kernel would report errors only on file descriptions on
-which writes were done that subsequently failed to be written back. The
-generic pagecache infrastructure does not track the file descriptions
-that have dirtied each individual page however, so determining which
-file descriptors should get back an error is not possible.
-
-Instead, the generic writeback error tracking infrastructure in the
-kernel settles for reporting errors to fsync on all file descriptions
-that were open at the time that the error occurred. In a situation with
-multiple writers, all of them will get back an error on a subsequent fsync,
-even if all of the writes done through that particular file descriptor
-succeeded (or even if there were no writes on that file descriptor at all).
-
-Filesystems that wish to use this infrastructure should call
-mapping_set_error to record the error in the address_space when it
-occurs. Then, after writing back data from the pagecache in their
-file->fsync operation, they should call file_check_and_advance_wb_err to
-ensure that the struct file's error cursor has advanced to the correct
-point in the stream of errors emitted by the backing device(s).
-
-struct address_space_operations
--------------------------------
-
-This describes how the VFS can manipulate mapping of a file to page cache in
-your filesystem. The following members are defined:
-
-struct address_space_operations {
- int (*writepage)(struct page *page, struct writeback_control *wbc);
- int (*readpage)(struct file *, struct page *);
- int (*writepages)(struct address_space *, struct writeback_control *);
- int (*set_page_dirty)(struct page *page);
- int (*readpages)(struct file *filp, struct address_space *mapping,
- struct list_head *pages, unsigned nr_pages);
- int (*write_begin)(struct file *, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned flags,
- struct page **pagep, void **fsdata);
- int (*write_end)(struct file *, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned copied,
- struct page *page, void *fsdata);
- sector_t (*bmap)(struct address_space *, sector_t);
- void (*invalidatepage) (struct page *, unsigned int, unsigned int);
- int (*releasepage) (struct page *, int);
- void (*freepage)(struct page *);
- ssize_t (*direct_IO)(struct kiocb *, struct iov_iter *iter);
- /* isolate a page for migration */
- bool (*isolate_page) (struct page *, isolate_mode_t);
- /* migrate the contents of a page to the specified target */
- int (*migratepage) (struct page *, struct page *);
- /* put migration-failed page back to right list */
- void (*putback_page) (struct page *);
- int (*launder_page) (struct page *);
-
- int (*is_partially_uptodate) (struct page *, unsigned long,
- unsigned long);
- void (*is_dirty_writeback) (struct page *, bool *, bool *);
- int (*error_remove_page) (struct mapping *mapping, struct page *page);
- int (*swap_activate)(struct file *);
- int (*swap_deactivate)(struct file *);
-};
-
- writepage: called by the VM to write a dirty page to backing store.
- This may happen for data integrity reasons (i.e. 'sync'), or
- to free up memory (flush). The difference can be seen in
- wbc->sync_mode.
- The PG_Dirty flag has been cleared and PageLocked is true.
- writepage should start writeout, should set PG_Writeback,
- and should make sure the page is unlocked, either synchronously
- or asynchronously when the write operation completes.
-
- If wbc->sync_mode is WB_SYNC_NONE, ->writepage doesn't have to
- try too hard if there are problems, and may choose to write out
- other pages from the mapping if that is easier (e.g. due to
- internal dependencies). If it chooses not to start writeout, it
- should return AOP_WRITEPAGE_ACTIVATE so that the VM will not keep
- calling ->writepage on that page.
-
- See the file "Locking" for more details.
-
- readpage: called by the VM to read a page from backing store.
- The page will be Locked when readpage is called, and should be
- unlocked and marked uptodate once the read completes.
- If ->readpage discovers that it needs to unlock the page for
- some reason, it can do so, and then return AOP_TRUNCATED_PAGE.
- In this case, the page will be relocated, relocked and if
- that all succeeds, ->readpage will be called again.
-
- writepages: called by the VM to write out pages associated with the
- address_space object. If wbc->sync_mode is WBC_SYNC_ALL, then
- the writeback_control will specify a range of pages that must be
- written out. If it is WBC_SYNC_NONE, then a nr_to_write is given
- and that many pages should be written if possible.
- If no ->writepages is given, then mpage_writepages is used
- instead. This will choose pages from the address space that are
- tagged as DIRTY and will pass them to ->writepage.
-
- set_page_dirty: called by the VM to set a page dirty.
- This is particularly needed if an address space attaches
- private data to a page, and that data needs to be updated when
- a page is dirtied. This is called, for example, when a memory
- mapped page gets modified.
- If defined, it should set the PageDirty flag, and the
- PAGECACHE_TAG_DIRTY tag in the radix tree.
-
- readpages: called by the VM to read pages associated with the address_space
- object. This is essentially just a vector version of
- readpage. Instead of just one page, several pages are
- requested.
- readpages is only used for read-ahead, so read errors are
- ignored. If anything goes wrong, feel free to give up.
-
- write_begin:
- Called by the generic buffered write code to ask the filesystem to
- prepare to write len bytes at the given offset in the file. The
- address_space should check that the write will be able to complete,
- by allocating space if necessary and doing any other internal
- housekeeping. If the write will update parts of any basic-blocks on
- storage, then those blocks should be pre-read (if they haven't been
- read already) so that the updated blocks can be written out properly.
-
- The filesystem must return the locked pagecache page for the specified
- offset, in *pagep, for the caller to write into.
-
- It must be able to cope with short writes (where the length passed to
- write_begin is greater than the number of bytes copied into the page).
-
- flags is a field for AOP_FLAG_xxx flags, described in
- include/linux/fs.h.
-
- A void * may be returned in fsdata, which then gets passed into
- write_end.
-
- Returns 0 on success; < 0 on failure (which is the error code), in
- which case write_end is not called.
-
- write_end: After a successful write_begin, and data copy, write_end must
- be called. len is the original len passed to write_begin, and copied
- is the amount that was able to be copied.
-
- The filesystem must take care of unlocking the page and releasing it
- refcount, and updating i_size.
-
- Returns < 0 on failure, otherwise the number of bytes (<= 'copied')
- that were able to be copied into pagecache.
-
- bmap: called by the VFS to map a logical block offset within object to
- physical block number. This method is used by the FIBMAP
- ioctl and for working with swap-files. To be able to swap to
- a file, the file must have a stable mapping to a block
- device. The swap system does not go through the filesystem
- but instead uses bmap to find out where the blocks in the file
- are and uses those addresses directly.
-
- invalidatepage: If a page has PagePrivate set, then invalidatepage
- will be called when part or all of the page is to be removed
- from the address space. This generally corresponds to either a
- truncation, punch hole or a complete invalidation of the address
- space (in the latter case 'offset' will always be 0 and 'length'
- will be PAGE_SIZE). Any private data associated with the page
- should be updated to reflect this truncation. If offset is 0 and
- length is PAGE_SIZE, then the private data should be released,
- because the page must be able to be completely discarded. This may
- be done by calling the ->releasepage function, but in this case the
- release MUST succeed.
-
- releasepage: releasepage is called on PagePrivate pages to indicate
- that the page should be freed if possible. ->releasepage
- should remove any private data from the page and clear the
- PagePrivate flag. If releasepage() fails for some reason, it must
- indicate failure with a 0 return value.
- releasepage() is used in two distinct though related cases. The
- first is when the VM finds a clean page with no active users and
- wants to make it a free page. If ->releasepage succeeds, the
- page will be removed from the address_space and become free.
-
- The second case is when a request has been made to invalidate
- some or all pages in an address_space. This can happen
- through the fadvise(POSIX_FADV_DONTNEED) system call or by the
- filesystem explicitly requesting it as nfs and 9fs do (when
- they believe the cache may be out of date with storage) by
- calling invalidate_inode_pages2().
- If the filesystem makes such a call, and needs to be certain
- that all pages are invalidated, then its releasepage will
- need to ensure this. Possibly it can clear the PageUptodate
- bit if it cannot free private data yet.
-
- freepage: freepage is called once the page is no longer visible in
- the page cache in order to allow the cleanup of any private
- data. Since it may be called by the memory reclaimer, it
- should not assume that the original address_space mapping still
- exists, and it should not block.
-
- direct_IO: called by the generic read/write routines to perform
- direct_IO - that is IO requests which bypass the page cache
- and transfer data directly between the storage and the
- application's address space.
-
- isolate_page: Called by the VM when isolating a movable non-lru page.
- If page is successfully isolated, VM marks the page as PG_isolated
- via __SetPageIsolated.
-
- migrate_page: This is used to compact the physical memory usage.
- If the VM wants to relocate a page (maybe off a memory card
- that is signalling imminent failure) it will pass a new page
- and an old page to this function. migrate_page should
- transfer any private data across and update any references
- that it has to the page.
-
- putback_page: Called by the VM when isolated page's migration fails.
-
- launder_page: Called before freeing a page - it writes back the dirty page. To
- prevent redirtying the page, it is kept locked during the whole
- operation.
-
- is_partially_uptodate: Called by the VM when reading a file through the
- pagecache when the underlying blocksize != pagesize. If the required
- block is up to date then the read can complete without needing the IO
- to bring the whole page up to date.
-
- is_dirty_writeback: Called by the VM when attempting to reclaim a page.
- The VM uses dirty and writeback information to determine if it needs
- to stall to allow flushers a chance to complete some IO. Ordinarily
- it can use PageDirty and PageWriteback but some filesystems have
- more complex state (unstable pages in NFS prevent reclaim) or
- do not set those flags due to locking problems. This callback
- allows a filesystem to indicate to the VM if a page should be
- treated as dirty or writeback for the purposes of stalling.
-
- error_remove_page: normally set to generic_error_remove_page if truncation
- is ok for this address space. Used for memory failure handling.
- Setting this implies you deal with pages going away under you,
- unless you have them locked or reference counts increased.
-
- swap_activate: Called when swapon is used on a file to allocate
- space if necessary and pin the block lookup information in
- memory. A return value of zero indicates success,
- in which case this file can be used to back swapspace.
-
- swap_deactivate: Called during swapoff on files where swap_activate
- was successful.
-
-
-The File Object
-===============
-
-A file object represents a file opened by a process. This is also known
-as an "open file description" in POSIX parlance.
-
-
-struct file_operations
-----------------------
-
-This describes how the VFS can manipulate an open file. As of kernel
-4.18, the following members are defined:
-
-struct file_operations {
- struct module *owner;
- loff_t (*llseek) (struct file *, loff_t, int);
- ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
- ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
- ssize_t (*read_iter) (struct kiocb *, struct iov_iter *);
- ssize_t (*write_iter) (struct kiocb *, struct iov_iter *);
- int (*iterate) (struct file *, struct dir_context *);
- int (*iterate_shared) (struct file *, struct dir_context *);
- __poll_t (*poll) (struct file *, struct poll_table_struct *);
- long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long);
- long (*compat_ioctl) (struct file *, unsigned int, unsigned long);
- int (*mmap) (struct file *, struct vm_area_struct *);
- int (*open) (struct inode *, struct file *);
- int (*flush) (struct file *, fl_owner_t id);
- int (*release) (struct inode *, struct file *);
- int (*fsync) (struct file *, loff_t, loff_t, int datasync);
- int (*fasync) (int, struct file *, int);
- int (*lock) (struct file *, int, struct file_lock *);
- ssize_t (*sendpage) (struct file *, struct page *, int, size_t, loff_t *, int);
- unsigned long (*get_unmapped_area)(struct file *, unsigned long, unsigned long, unsigned long, unsigned long);
- int (*check_flags)(int);
- int (*flock) (struct file *, int, struct file_lock *);
- ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int);
- ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int);
- int (*setlease)(struct file *, long, struct file_lock **, void **);
- long (*fallocate)(struct file *file, int mode, loff_t offset,
- loff_t len);
- void (*show_fdinfo)(struct seq_file *m, struct file *f);
-#ifndef CONFIG_MMU
- unsigned (*mmap_capabilities)(struct file *);
-#endif
- ssize_t (*copy_file_range)(struct file *, loff_t, struct file *, loff_t, size_t, unsigned int);
- int (*clone_file_range)(struct file *, loff_t, struct file *, loff_t, u64);
- int (*dedupe_file_range)(struct file *, loff_t, struct file *, loff_t, u64);
- int (*fadvise)(struct file *, loff_t, loff_t, int);
-};
-
-Again, all methods are called without any locks being held, unless
-otherwise noted.
-
- llseek: called when the VFS needs to move the file position index
-
- read: called by read(2) and related system calls
-
- read_iter: possibly asynchronous read with iov_iter as destination
-
- write: called by write(2) and related system calls
-
- write_iter: possibly asynchronous write with iov_iter as source
-
- iterate: called when the VFS needs to read the directory contents
-
- iterate_shared: called when the VFS needs to read the directory contents
- when filesystem supports concurrent dir iterators
-
- poll: called by the VFS when a process wants to check if there is
- activity on this file and (optionally) go to sleep until there
- is activity. Called by the select(2) and poll(2) system calls
-
- unlocked_ioctl: called by the ioctl(2) system call.
-
- compat_ioctl: called by the ioctl(2) system call when 32 bit system calls
- are used on 64 bit kernels.
-
- mmap: called by the mmap(2) system call
-
- open: called by the VFS when an inode should be opened. When the VFS
- opens a file, it creates a new "struct file". It then calls the
- open method for the newly allocated file structure. You might
- think that the open method really belongs in
- "struct inode_operations", and you may be right. I think it's
- done the way it is because it makes filesystems simpler to
- implement. The open() method is a good place to initialize the
- "private_data" member in the file structure if you want to point
- to a device structure
-
- flush: called by the close(2) system call to flush a file
-
- release: called when the last reference to an open file is closed
-
- fsync: called by the fsync(2) system call. Also see the section above
- entitled "Handling errors during writeback".
-
- fasync: called by the fcntl(2) system call when asynchronous
- (non-blocking) mode is enabled for a file
-
- lock: called by the fcntl(2) system call for F_GETLK, F_SETLK, and F_SETLKW
- commands
-
- get_unmapped_area: called by the mmap(2) system call
-
- check_flags: called by the fcntl(2) system call for F_SETFL command
-
- flock: called by the flock(2) system call
-
- splice_write: called by the VFS to splice data from a pipe to a file. This
- method is used by the splice(2) system call
-
- splice_read: called by the VFS to splice data from file to a pipe. This
- method is used by the splice(2) system call
-
- setlease: called by the VFS to set or release a file lock lease. setlease
- implementations should call generic_setlease to record or remove
- the lease in the inode after setting it.
-
- fallocate: called by the VFS to preallocate blocks or punch a hole.
-
- copy_file_range: called by the copy_file_range(2) system call.
-
- clone_file_range: called by the ioctl(2) system call for FICLONERANGE and
- FICLONE commands.
-
- dedupe_file_range: called by the ioctl(2) system call for FIDEDUPERANGE
- command.
-
- fadvise: possibly called by the fadvise64() system call.
-
-Note that the file operations are implemented by the specific
-filesystem in which the inode resides. When opening a device node
-(character or block special) most filesystems will call special
-support routines in the VFS which will locate the required device
-driver information. These support routines replace the filesystem file
-operations with those for the device driver, and then proceed to call
-the new open() method for the file. This is how opening a device file
-in the filesystem eventually ends up calling the device driver open()
-method.
-
-
-Directory Entry Cache (dcache)
-==============================
-
-
-struct dentry_operations
-------------------------
-
-This describes how a filesystem can overload the standard dentry
-operations. Dentries and the dcache are the domain of the VFS and the
-individual filesystem implementations. Device drivers have no business
-here. These methods may be set to NULL, as they are either optional or
-the VFS uses a default. As of kernel 2.6.22, the following members are
-defined:
-
-struct dentry_operations {
- int (*d_revalidate)(struct dentry *, unsigned int);
- int (*d_weak_revalidate)(struct dentry *, unsigned int);
- int (*d_hash)(const struct dentry *, struct qstr *);
- int (*d_compare)(const struct dentry *,
- unsigned int, const char *, const struct qstr *);
- int (*d_delete)(const struct dentry *);
- int (*d_init)(struct dentry *);
- void (*d_release)(struct dentry *);
- void (*d_iput)(struct dentry *, struct inode *);
- char *(*d_dname)(struct dentry *, char *, int);
- struct vfsmount *(*d_automount)(struct path *);
- int (*d_manage)(const struct path *, bool);
- struct dentry *(*d_real)(struct dentry *, const struct inode *);
-};
-
- d_revalidate: called when the VFS needs to revalidate a dentry. This
- is called whenever a name look-up finds a dentry in the
- dcache. Most local filesystems leave this as NULL, because all their
- dentries in the dcache are valid. Network filesystems are different
- since things can change on the server without the client necessarily
- being aware of it.
-
- This function should return a positive value if the dentry is still
- valid, and zero or a negative error code if it isn't.
-
- d_revalidate may be called in rcu-walk mode (flags & LOOKUP_RCU).
- If in rcu-walk mode, the filesystem must revalidate the dentry without
- blocking or storing to the dentry, d_parent and d_inode should not be
- used without care (because they can change and, in d_inode case, even
- become NULL under us).
-
- If a situation is encountered that rcu-walk cannot handle, return
- -ECHILD and it will be called again in ref-walk mode.
-
- d_weak_revalidate: called when the VFS needs to revalidate a "jumped" dentry.
- This is called when a path-walk ends at dentry that was not acquired by
- doing a lookup in the parent directory. This includes "/", "." and "..",
- as well as procfs-style symlinks and mountpoint traversal.
-
- In this case, we are less concerned with whether the dentry is still
- fully correct, but rather that the inode is still valid. As with
- d_revalidate, most local filesystems will set this to NULL since their
- dcache entries are always valid.
-
- This function has the same return code semantics as d_revalidate.
-
- d_weak_revalidate is only called after leaving rcu-walk mode.
-
- d_hash: called when the VFS adds a dentry to the hash table. The first
- dentry passed to d_hash is the parent directory that the name is
- to be hashed into.
-
- Same locking and synchronisation rules as d_compare regarding
- what is safe to dereference etc.
-
- d_compare: called to compare a dentry name with a given name. The first
- dentry is the parent of the dentry to be compared, the second is
- the child dentry. len and name string are properties of the dentry
- to be compared. qstr is the name to compare it with.
-
- Must be constant and idempotent, and should not take locks if
- possible, and should not or store into the dentry.
- Should not dereference pointers outside the dentry without
- lots of care (eg. d_parent, d_inode, d_name should not be used).
-
- However, our vfsmount is pinned, and RCU held, so the dentries and
- inodes won't disappear, neither will our sb or filesystem module.
- ->d_sb may be used.
-
- It is a tricky calling convention because it needs to be called under
- "rcu-walk", ie. without any locks or references on things.
-
- d_delete: called when the last reference to a dentry is dropped and the
- dcache is deciding whether or not to cache it. Return 1 to delete
- immediately, or 0 to cache the dentry. Default is NULL which means to
- always cache a reachable dentry. d_delete must be constant and
- idempotent.
-
- d_init: called when a dentry is allocated
-
- d_release: called when a dentry is really deallocated
-
- d_iput: called when a dentry loses its inode (just prior to its
- being deallocated). The default when this is NULL is that the
- VFS calls iput(). If you define this method, you must call
- iput() yourself
-
- d_dname: called when the pathname of a dentry should be generated.
- Useful for some pseudo filesystems (sockfs, pipefs, ...) to delay
- pathname generation. (Instead of doing it when dentry is created,
- it's done only when the path is needed.). Real filesystems probably
- dont want to use it, because their dentries are present in global
- dcache hash, so their hash should be an invariant. As no lock is
- held, d_dname() should not try to modify the dentry itself, unless
- appropriate SMP safety is used. CAUTION : d_path() logic is quite
- tricky. The correct way to return for example "Hello" is to put it
- at the end of the buffer, and returns a pointer to the first char.
- dynamic_dname() helper function is provided to take care of this.
-
- Example :
-
- static char *pipefs_dname(struct dentry *dent, char *buffer, int buflen)
- {
- return dynamic_dname(dentry, buffer, buflen, "pipe:[%lu]",
- dentry->d_inode->i_ino);
- }
-
- d_automount: called when an automount dentry is to be traversed (optional).
- This should create a new VFS mount record and return the record to the
- caller. The caller is supplied with a path parameter giving the
- automount directory to describe the automount target and the parent
- VFS mount record to provide inheritable mount parameters. NULL should
- be returned if someone else managed to make the automount first. If
- the vfsmount creation failed, then an error code should be returned.
- If -EISDIR is returned, then the directory will be treated as an
- ordinary directory and returned to pathwalk to continue walking.
-
- If a vfsmount is returned, the caller will attempt to mount it on the
- mountpoint and will remove the vfsmount from its expiration list in
- the case of failure. The vfsmount should be returned with 2 refs on
- it to prevent automatic expiration - the caller will clean up the
- additional ref.
-
- This function is only used if DCACHE_NEED_AUTOMOUNT is set on the
- dentry. This is set by __d_instantiate() if S_AUTOMOUNT is set on the
- inode being added.
-
- d_manage: called to allow the filesystem to manage the transition from a
- dentry (optional). This allows autofs, for example, to hold up clients
- waiting to explore behind a 'mountpoint' whilst letting the daemon go
- past and construct the subtree there. 0 should be returned to let the
- calling process continue. -EISDIR can be returned to tell pathwalk to
- use this directory as an ordinary directory and to ignore anything
- mounted on it and not to check the automount flag. Any other error
- code will abort pathwalk completely.
-
- If the 'rcu_walk' parameter is true, then the caller is doing a
- pathwalk in RCU-walk mode. Sleeping is not permitted in this mode,
- and the caller can be asked to leave it and call again by returning
- -ECHILD. -EISDIR may also be returned to tell pathwalk to
- ignore d_automount or any mounts.
-
- This function is only used if DCACHE_MANAGE_TRANSIT is set on the
- dentry being transited from.
-
- d_real: overlay/union type filesystems implement this method to return one of
- the underlying dentries hidden by the overlay. It is used in two
- different modes:
-
- Called from file_dentry() it returns the real dentry matching the inode
- argument. The real dentry may be from a lower layer already copied up,
- but still referenced from the file. This mode is selected with a
- non-NULL inode argument.
-
- With NULL inode the topmost real underlying dentry is returned.
-
-Each dentry has a pointer to its parent dentry, as well as a hash list
-of child dentries. Child dentries are basically like files in a
-directory.
-
-
-Directory Entry Cache API
---------------------------
-
-There are a number of functions defined which permit a filesystem to
-manipulate dentries:
-
- dget: open a new handle for an existing dentry (this just increments
- the usage count)
-
- dput: close a handle for a dentry (decrements the usage count). If
- the usage count drops to 0, and the dentry is still in its
- parent's hash, the "d_delete" method is called to check whether
- it should be cached. If it should not be cached, or if the dentry
- is not hashed, it is deleted. Otherwise cached dentries are put
- into an LRU list to be reclaimed on memory shortage.
-
- d_drop: this unhashes a dentry from its parents hash list. A
- subsequent call to dput() will deallocate the dentry if its
- usage count drops to 0
-
- d_delete: delete a dentry. If there are no other open references to
- the dentry then the dentry is turned into a negative dentry
- (the d_iput() method is called). If there are other
- references, then d_drop() is called instead
-
- d_add: add a dentry to its parents hash list and then calls
- d_instantiate()
-
- d_instantiate: add a dentry to the alias hash list for the inode and
- updates the "d_inode" member. The "i_count" member in the
- inode structure should be set/incremented. If the inode
- pointer is NULL, the dentry is called a "negative
- dentry". This function is commonly called when an inode is
- created for an existing negative dentry
-
- d_lookup: look up a dentry given its parent and path name component
- It looks up the child of that given name from the dcache
- hash table. If it is found, the reference count is incremented
- and the dentry is returned. The caller must use dput()
- to free the dentry when it finishes using it.
-
-Mount Options
-=============
-
-Parsing options
----------------
-
-On mount and remount the filesystem is passed a string containing a
-comma separated list of mount options. The options can have either of
-these forms:
-
- option
- option=value
-
-The <linux/parser.h> header defines an API that helps parse these
-options. There are plenty of examples on how to use it in existing
-filesystems.
-
-Showing options
----------------
-
-If a filesystem accepts mount options, it must define show_options()
-to show all the currently active options. The rules are:
-
- - options MUST be shown which are not default or their values differ
- from the default
-
- - options MAY be shown which are enabled by default or have their
- default value
-
-Options used only internally between a mount helper and the kernel
-(such as file descriptors), or which only have an effect during the
-mounting (such as ones controlling the creation of a journal) are exempt
-from the above rules.
-
-The underlying reason for the above rules is to make sure, that a
-mount can be accurately replicated (e.g. umounting and mounting again)
-based on the information found in /proc/mounts.
-
-Resources
-=========
-
-(Note some of these resources are not up-to-date with the latest kernel
- version.)
-
-Creating Linux virtual filesystems. 2002
- <http://lwn.net/Articles/13325/>
-
-The Linux Virtual File-system Layer by Neil Brown. 1999
- <http://www.cse.unsw.edu.au/~neilb/oss/linux-commentary/vfs.html>
-
-A tour of the Linux VFS by Michael K. Johnson. 1996
- <http://www.tldp.org/LDP/khg/HyperNews/get/fs/vfstour.html>
-
-A small trail through the Linux kernel by Andries Brouwer. 2001
- <http://www.win.tue.nl/~aeb/linux/vfs/trail.html>
diff --git a/Documentation/filesystems/virtiofs.rst b/Documentation/filesystems/virtiofs.rst
new file mode 100644
index 0000000..4f338e3
--- /dev/null
+++ b/Documentation/filesystems/virtiofs.rst
@@ -0,0 +1,60 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===================================================
+virtiofs: virtio-fs host<->guest shared file system
+===================================================
+
+- Copyright (C) 2019 Red Hat, Inc.
+
+Introduction
+============
+The virtiofs file system for Linux implements a driver for the paravirtualized
+VIRTIO "virtio-fs" device for guest<->host file system sharing. It allows a
+guest to mount a directory that has been exported on the host.
+
+Guests often require access to files residing on the host or remote systems.
+Use cases include making files available to new guests during installation,
+booting from a root file system located on the host, persistent storage for
+stateless or ephemeral guests, and sharing a directory between guests.
+
+Although it is possible to use existing network file systems for some of these
+tasks, they require configuration steps that are hard to automate and they
+expose the storage network to the guest. The virtio-fs device was designed to
+solve these problems by providing file system access without networking.
+
+Furthermore the virtio-fs device takes advantage of the co-location of the
+guest and host to increase performance and provide semantics that are not
+possible with network file systems.
+
+Usage
+=====
+Mount file system with tag ``myfs`` on ``/mnt``:
+
+.. code-block:: sh
+
+ guest# mount -t virtiofs myfs /mnt
+
+Please see https://virtio-fs.gitlab.io/ for details on how to configure QEMU
+and the virtiofsd daemon.
+
+Internals
+=========
+Since the virtio-fs device uses the FUSE protocol for file system requests, the
+virtiofs file system for Linux is integrated closely with the FUSE file system
+client. The guest acts as the FUSE client while the host acts as the FUSE
+server. The /dev/fuse interface between the kernel and userspace is replaced
+with the virtio-fs device interface.
+
+FUSE requests are placed into a virtqueue and processed by the host. The
+response portion of the buffer is filled in by the host and the guest handles
+the request completion.
+
+Mapping /dev/fuse to virtqueues requires solving differences in semantics
+between /dev/fuse and virtqueues. Each time the /dev/fuse device is read, the
+FUSE client may choose which request to transfer, making it possible to
+prioritize certain requests over others. Virtqueues have queue semantics and
+it is not possible to change the order of requests that have been enqueued.
+This is especially important if the virtqueue becomes full since it is then
+impossible to add high priority requests. In order to address this difference,
+the virtio-fs device uses a "hiprio" virtqueue specifically for requests that
+have priority over normal requests.
diff --git a/Documentation/filesystems/xfs-delayed-logging-design.txt b/Documentation/filesystems/xfs-delayed-logging-design.txt
index 2ce3643..9a6dd28 100644
--- a/Documentation/filesystems/xfs-delayed-logging-design.txt
+++ b/Documentation/filesystems/xfs-delayed-logging-design.txt
@@ -34,7 +34,7 @@
D A+B+C+D X+n+m+o
<object written to disk>
E E Y (> X+n+m+o)
- F E+F Yٍ+p
+ F E+F Y+p
In other words, each time an object is relogged, the new transaction contains
the aggregation of all the previous changes currently held only in the log.
diff --git a/Documentation/filesystems/xfs-self-describing-metadata.txt b/Documentation/filesystems/xfs-self-describing-metadata.txt
index 05aa455..8db0121 100644
--- a/Documentation/filesystems/xfs-self-describing-metadata.txt
+++ b/Documentation/filesystems/xfs-self-describing-metadata.txt
@@ -110,7 +110,7 @@
determine the scope of the problem.
Different types of metadata have different owner identifiers. For example,
-directory, attribute and extent tree blocks are all owned by an inode, whilst
+directory, attribute and extent tree blocks are all owned by an inode, while
freespace btree blocks are owned by an allocation group. Hence the size and
contents of the owner field are determined by the type of metadata object we are
looking at. The owner information can also identify misplaced writes (e.g.
@@ -222,7 +222,7 @@
xfs_foo_read_verify(
struct xfs_buf *bp)
{
- struct xfs_mount *mp = bp->b_target->bt_mount;
+ struct xfs_mount *mp = bp->b_mount;
if ((xfs_sb_version_hascrc(&mp->m_sb) &&
!xfs_verify_cksum(bp->b_addr, BBTOB(bp->b_length),
@@ -245,7 +245,7 @@
xfs_foo_verify(
struct xfs_buf *bp)
{
- struct xfs_mount *mp = bp->b_target->bt_mount;
+ struct xfs_mount *mp = bp->b_mount;
struct xfs_ondisk_hdr *hdr = bp->b_addr;
if (hdr->magic != cpu_to_be32(XFS_FOO_MAGIC))
@@ -272,7 +272,7 @@
xfs_foo_verify(
struct xfs_buf *bp)
{
- struct xfs_mount *mp = bp->b_target->bt_mount;
+ struct xfs_mount *mp = bp->b_mount;
struct xfs_ondisk_hdr *hdr = bp->b_addr;
if (hdr->magic == cpu_to_be32(XFS_FOO_CRC_MAGIC)) {
@@ -297,7 +297,7 @@
xfs_foo_write_verify(
struct xfs_buf *bp)
{
- struct xfs_mount *mp = bp->b_target->bt_mount;
+ struct xfs_mount *mp = bp->b_mount;
struct xfs_buf_log_item *bip = bp->b_fspriv;
if (!xfs_foo_verify(bp)) {
diff --git a/Documentation/filesystems/xfs.txt b/Documentation/filesystems/xfs.txt
deleted file mode 100644
index a9ae82f..0000000
--- a/Documentation/filesystems/xfs.txt
+++ /dev/null
@@ -1,469 +0,0 @@
-
-The SGI XFS Filesystem
-======================
-
-XFS is a high performance journaling filesystem which originated
-on the SGI IRIX platform. It is completely multi-threaded, can
-support large files and large filesystems, extended attributes,
-variable block sizes, is extent based, and makes extensive use of
-Btrees (directories, extents, free space) to aid both performance
-and scalability.
-
-Refer to the documentation at https://xfs.wiki.kernel.org/
-for further details. This implementation is on-disk compatible
-with the IRIX version of XFS.
-
-
-Mount Options
-=============
-
-When mounting an XFS filesystem, the following options are accepted.
-For boolean mount options, the names with the (*) suffix is the
-default behaviour.
-
- allocsize=size
- Sets the buffered I/O end-of-file preallocation size when
- doing delayed allocation writeout (default size is 64KiB).
- Valid values for this option are page size (typically 4KiB)
- through to 1GiB, inclusive, in power-of-2 increments.
-
- The default behaviour is for dynamic end-of-file
- preallocation size, which uses a set of heuristics to
- optimise the preallocation size based on the current
- allocation patterns within the file and the access patterns
- to the file. Specifying a fixed allocsize value turns off
- the dynamic behaviour.
-
- attr2
- noattr2
- The options enable/disable an "opportunistic" improvement to
- be made in the way inline extended attributes are stored
- on-disk. When the new form is used for the first time when
- attr2 is selected (either when setting or removing extended
- attributes) the on-disk superblock feature bit field will be
- updated to reflect this format being in use.
-
- The default behaviour is determined by the on-disk feature
- bit indicating that attr2 behaviour is active. If either
- mount option it set, then that becomes the new default used
- by the filesystem.
-
- CRC enabled filesystems always use the attr2 format, and so
- will reject the noattr2 mount option if it is set.
-
- discard
- nodiscard (*)
- Enable/disable the issuing of commands to let the block
- device reclaim space freed by the filesystem. This is
- useful for SSD devices, thinly provisioned LUNs and virtual
- machine images, but may have a performance impact.
-
- Note: It is currently recommended that you use the fstrim
- application to discard unused blocks rather than the discard
- mount option because the performance impact of this option
- is quite severe.
-
- grpid/bsdgroups
- nogrpid/sysvgroups (*)
- These options define what group ID a newly created file
- gets. When grpid is set, it takes the group ID of the
- directory in which it is created; otherwise it takes the
- fsgid of the current process, unless the directory has the
- setgid bit set, in which case it takes the gid from the
- parent directory, and also gets the setgid bit set if it is
- a directory itself.
-
- filestreams
- Make the data allocator use the filestreams allocation mode
- across the entire filesystem rather than just on directories
- configured to use it.
-
- ikeep
- noikeep (*)
- When ikeep is specified, XFS does not delete empty inode
- clusters and keeps them around on disk. When noikeep is
- specified, empty inode clusters are returned to the free
- space pool.
-
- inode32
- inode64 (*)
- When inode32 is specified, it indicates that XFS limits
- inode creation to locations which will not result in inode
- numbers with more than 32 bits of significance.
-
- When inode64 is specified, it indicates that XFS is allowed
- to create inodes at any location in the filesystem,
- including those which will result in inode numbers occupying
- more than 32 bits of significance.
-
- inode32 is provided for backwards compatibility with older
- systems and applications, since 64 bits inode numbers might
- cause problems for some applications that cannot handle
- large inode numbers. If applications are in use which do
- not handle inode numbers bigger than 32 bits, the inode32
- option should be specified.
-
-
- largeio
- nolargeio (*)
- If "nolargeio" is specified, the optimal I/O reported in
- st_blksize by stat(2) will be as small as possible to allow
- user applications to avoid inefficient read/modify/write
- I/O. This is typically the page size of the machine, as
- this is the granularity of the page cache.
-
- If "largeio" specified, a filesystem that was created with a
- "swidth" specified will return the "swidth" value (in bytes)
- in st_blksize. If the filesystem does not have a "swidth"
- specified but does specify an "allocsize" then "allocsize"
- (in bytes) will be returned instead. Otherwise the behaviour
- is the same as if "nolargeio" was specified.
-
- logbufs=value
- Set the number of in-memory log buffers. Valid numbers
- range from 2-8 inclusive.
-
- The default value is 8 buffers.
-
- If the memory cost of 8 log buffers is too high on small
- systems, then it may be reduced at some cost to performance
- on metadata intensive workloads. The logbsize option below
- controls the size of each buffer and so is also relevant to
- this case.
-
- logbsize=value
- Set the size of each in-memory log buffer. The size may be
- specified in bytes, or in kilobytes with a "k" suffix.
- Valid sizes for version 1 and version 2 logs are 16384 (16k)
- and 32768 (32k). Valid sizes for version 2 logs also
- include 65536 (64k), 131072 (128k) and 262144 (256k). The
- logbsize must be an integer multiple of the log
- stripe unit configured at mkfs time.
-
- The default value for for version 1 logs is 32768, while the
- default value for version 2 logs is MAX(32768, log_sunit).
-
- logdev=device and rtdev=device
- Use an external log (metadata journal) and/or real-time device.
- An XFS filesystem has up to three parts: a data section, a log
- section, and a real-time section. The real-time section is
- optional, and the log section can be separate from the data
- section or contained within it.
-
- noalign
- Data allocations will not be aligned at stripe unit
- boundaries. This is only relevant to filesystems created
- with non-zero data alignment parameters (sunit, swidth) by
- mkfs.
-
- norecovery
- The filesystem will be mounted without running log recovery.
- If the filesystem was not cleanly unmounted, it is likely to
- be inconsistent when mounted in "norecovery" mode.
- Some files or directories may not be accessible because of this.
- Filesystems mounted "norecovery" must be mounted read-only or
- the mount will fail.
-
- nouuid
- Don't check for double mounted file systems using the file
- system uuid. This is useful to mount LVM snapshot volumes,
- and often used in combination with "norecovery" for mounting
- read-only snapshots.
-
- noquota
- Forcibly turns off all quota accounting and enforcement
- within the filesystem.
-
- uquota/usrquota/uqnoenforce/quota
- User disk quota accounting enabled, and limits (optionally)
- enforced. Refer to xfs_quota(8) for further details.
-
- gquota/grpquota/gqnoenforce
- Group disk quota accounting enabled and limits (optionally)
- enforced. Refer to xfs_quota(8) for further details.
-
- pquota/prjquota/pqnoenforce
- Project disk quota accounting enabled and limits (optionally)
- enforced. Refer to xfs_quota(8) for further details.
-
- sunit=value and swidth=value
- Used to specify the stripe unit and width for a RAID device
- or a stripe volume. "value" must be specified in 512-byte
- block units. These options are only relevant to filesystems
- that were created with non-zero data alignment parameters.
-
- The sunit and swidth parameters specified must be compatible
- with the existing filesystem alignment characteristics. In
- general, that means the only valid changes to sunit are
- increasing it by a power-of-2 multiple. Valid swidth values
- are any integer multiple of a valid sunit value.
-
- Typically the only time these mount options are necessary if
- after an underlying RAID device has had it's geometry
- modified, such as adding a new disk to a RAID5 lun and
- reshaping it.
-
- swalloc
- Data allocations will be rounded up to stripe width boundaries
- when the current end of file is being extended and the file
- size is larger than the stripe width size.
-
- wsync
- When specified, all filesystem namespace operations are
- executed synchronously. This ensures that when the namespace
- operation (create, unlink, etc) completes, the change to the
- namespace is on stable storage. This is useful in HA setups
- where failover must not result in clients seeing
- inconsistent namespace presentation during or after a
- failover event.
-
-
-Deprecated Mount Options
-========================
-
- Name Removal Schedule
- ---- ----------------
-
-
-Removed Mount Options
-=====================
-
- Name Removed
- ---- -------
- delaylog/nodelaylog v4.0
- ihashsize v4.0
- irixsgid v4.0
- osyncisdsync/osyncisosync v4.0
- barrier v4.19
- nobarrier v4.19
-
-
-sysctls
-=======
-
-The following sysctls are available for the XFS filesystem:
-
- fs.xfs.stats_clear (Min: 0 Default: 0 Max: 1)
- Setting this to "1" clears accumulated XFS statistics
- in /proc/fs/xfs/stat. It then immediately resets to "0".
-
- fs.xfs.xfssyncd_centisecs (Min: 100 Default: 3000 Max: 720000)
- The interval at which the filesystem flushes metadata
- out to disk and runs internal cache cleanup routines.
-
- fs.xfs.filestream_centisecs (Min: 1 Default: 3000 Max: 360000)
- The interval at which the filesystem ages filestreams cache
- references and returns timed-out AGs back to the free stream
- pool.
-
- fs.xfs.speculative_prealloc_lifetime
- (Units: seconds Min: 1 Default: 300 Max: 86400)
- The interval at which the background scanning for inodes
- with unused speculative preallocation runs. The scan
- removes unused preallocation from clean inodes and releases
- the unused space back to the free pool.
-
- fs.xfs.error_level (Min: 0 Default: 3 Max: 11)
- A volume knob for error reporting when internal errors occur.
- This will generate detailed messages & backtraces for filesystem
- shutdowns, for example. Current threshold values are:
-
- XFS_ERRLEVEL_OFF: 0
- XFS_ERRLEVEL_LOW: 1
- XFS_ERRLEVEL_HIGH: 5
-
- fs.xfs.panic_mask (Min: 0 Default: 0 Max: 255)
- Causes certain error conditions to call BUG(). Value is a bitmask;
- OR together the tags which represent errors which should cause panics:
-
- XFS_NO_PTAG 0
- XFS_PTAG_IFLUSH 0x00000001
- XFS_PTAG_LOGRES 0x00000002
- XFS_PTAG_AILDELETE 0x00000004
- XFS_PTAG_ERROR_REPORT 0x00000008
- XFS_PTAG_SHUTDOWN_CORRUPT 0x00000010
- XFS_PTAG_SHUTDOWN_IOERROR 0x00000020
- XFS_PTAG_SHUTDOWN_LOGERROR 0x00000040
- XFS_PTAG_FSBLOCK_ZERO 0x00000080
-
- This option is intended for debugging only.
-
- fs.xfs.irix_symlink_mode (Min: 0 Default: 0 Max: 1)
- Controls whether symlinks are created with mode 0777 (default)
- or whether their mode is affected by the umask (irix mode).
-
- fs.xfs.irix_sgid_inherit (Min: 0 Default: 0 Max: 1)
- Controls files created in SGID directories.
- If the group ID of the new file does not match the effective group
- ID or one of the supplementary group IDs of the parent dir, the
- ISGID bit is cleared if the irix_sgid_inherit compatibility sysctl
- is set.
-
- fs.xfs.inherit_sync (Min: 0 Default: 1 Max: 1)
- Setting this to "1" will cause the "sync" flag set
- by the xfs_io(8) chattr command on a directory to be
- inherited by files in that directory.
-
- fs.xfs.inherit_nodump (Min: 0 Default: 1 Max: 1)
- Setting this to "1" will cause the "nodump" flag set
- by the xfs_io(8) chattr command on a directory to be
- inherited by files in that directory.
-
- fs.xfs.inherit_noatime (Min: 0 Default: 1 Max: 1)
- Setting this to "1" will cause the "noatime" flag set
- by the xfs_io(8) chattr command on a directory to be
- inherited by files in that directory.
-
- fs.xfs.inherit_nosymlinks (Min: 0 Default: 1 Max: 1)
- Setting this to "1" will cause the "nosymlinks" flag set
- by the xfs_io(8) chattr command on a directory to be
- inherited by files in that directory.
-
- fs.xfs.inherit_nodefrag (Min: 0 Default: 1 Max: 1)
- Setting this to "1" will cause the "nodefrag" flag set
- by the xfs_io(8) chattr command on a directory to be
- inherited by files in that directory.
-
- fs.xfs.rotorstep (Min: 1 Default: 1 Max: 256)
- In "inode32" allocation mode, this option determines how many
- files the allocator attempts to allocate in the same allocation
- group before moving to the next allocation group. The intent
- is to control the rate at which the allocator moves between
- allocation groups when allocating extents for new files.
-
-Deprecated Sysctls
-==================
-
-None at present.
-
-
-Removed Sysctls
-===============
-
- Name Removed
- ---- -------
- fs.xfs.xfsbufd_centisec v4.0
- fs.xfs.age_buffer_centisecs v4.0
-
-
-Error handling
-==============
-
-XFS can act differently according to the type of error found during its
-operation. The implementation introduces the following concepts to the error
-handler:
-
- -failure speed:
- Defines how fast XFS should propagate an error upwards when a specific
- error is found during the filesystem operation. It can propagate
- immediately, after a defined number of retries, after a set time period,
- or simply retry forever.
-
- -error classes:
- Specifies the subsystem the error configuration will apply to, such as
- metadata IO or memory allocation. Different subsystems will have
- different error handlers for which behaviour can be configured.
-
- -error handlers:
- Defines the behavior for a specific error.
-
-The filesystem behavior during an error can be set via sysfs files. Each
-error handler works independently - the first condition met by an error handler
-for a specific class will cause the error to be propagated rather than reset and
-retried.
-
-The action taken by the filesystem when the error is propagated is context
-dependent - it may cause a shut down in the case of an unrecoverable error,
-it may be reported back to userspace, or it may even be ignored because
-there's nothing useful we can with the error or anyone we can report it to (e.g.
-during unmount).
-
-The configuration files are organized into the following hierarchy for each
-mounted filesystem:
-
- /sys/fs/xfs/<dev>/error/<class>/<error>/
-
-Where:
- <dev>
- The short device name of the mounted filesystem. This is the same device
- name that shows up in XFS kernel error messages as "XFS(<dev>): ..."
-
- <class>
- The subsystem the error configuration belongs to. As of 4.9, the defined
- classes are:
-
- - "metadata": applies metadata buffer write IO
-
- <error>
- The individual error handler configurations.
-
-
-Each filesystem has "global" error configuration options defined in their top
-level directory:
-
- /sys/fs/xfs/<dev>/error/
-
- fail_at_unmount (Min: 0 Default: 1 Max: 1)
- Defines the filesystem error behavior at unmount time.
-
- If set to a value of 1, XFS will override all other error configurations
- during unmount and replace them with "immediate fail" characteristics.
- i.e. no retries, no retry timeout. This will always allow unmount to
- succeed when there are persistent errors present.
-
- If set to 0, the configured retry behaviour will continue until all
- retries and/or timeouts have been exhausted. This will delay unmount
- completion when there are persistent errors, and it may prevent the
- filesystem from ever unmounting fully in the case of "retry forever"
- handler configurations.
-
- Note: there is no guarantee that fail_at_unmount can be set whilst an
- unmount is in progress. It is possible that the sysfs entries are
- removed by the unmounting filesystem before a "retry forever" error
- handler configuration causes unmount to hang, and hence the filesystem
- must be configured appropriately before unmount begins to prevent
- unmount hangs.
-
-Each filesystem has specific error class handlers that define the error
-propagation behaviour for specific errors. There is also a "default" error
-handler defined, which defines the behaviour for all errors that don't have
-specific handlers defined. Where multiple retry constraints are configuredi for
-a single error, the first retry configuration that expires will cause the error
-to be propagated. The handler configurations are found in the directory:
-
- /sys/fs/xfs/<dev>/error/<class>/<error>/
-
- max_retries (Min: -1 Default: Varies Max: INTMAX)
- Defines the allowed number of retries of a specific error before
- the filesystem will propagate the error. The retry count for a given
- error context (e.g. a specific metadata buffer) is reset every time
- there is a successful completion of the operation.
-
- Setting the value to "-1" will cause XFS to retry forever for this
- specific error.
-
- Setting the value to "0" will cause XFS to fail immediately when the
- specific error is reported.
-
- Setting the value to "N" (where 0 < N < Max) will make XFS retry the
- operation "N" times before propagating the error.
-
- retry_timeout_seconds (Min: -1 Default: Varies Max: 1 day)
- Define the amount of time (in seconds) that the filesystem is
- allowed to retry its operations when the specific error is
- found.
-
- Setting the value to "-1" will allow XFS to retry forever for this
- specific error.
-
- Setting the value to "0" will cause XFS to fail immediately when the
- specific error is reported.
-
- Setting the value to "N" (where 0 < N < Max) will allow XFS to retry the
- operation for up to "N" seconds before propagating the error.
-
-Note: The default behaviour for a specific error handler is dependent on both
-the class and error context. For example, the default values for
-"metadata/ENODEV" are "0" rather than "-1" so that this error handler defaults
-to "fail immediately" behaviour. This is done because ENODEV is a fatal,
-unrecoverable error no matter how many times the metadata IO is retried.