--- name: "dfsa" section: "7" platform: "linux" package: "mosix" version: "1.5.7-2" release: "Debian 3.0 (Woody)" platforms: ["linux"] url: "https://new.carta.tech/dfsa" --- # dfsa(7) Direct File-System Access ## NAME **DFSA** – Direct File-System Access ## DESCRIPTION *DFSA* is an option within *MOSIX(7)* that allows remote processes (eg. processes that currently run away from their home-node) under some conditions to save much time by performing most of their I/O and file-system operations directly where they run instead of having to contact their home-node. Moreover, processes that fulfill those conditions are more likely to migrate automatically away from their home-node to a less loaded node. *DFSA* operates over suitable, cluster-wide shared file-systems that fulfill certain requirements. The only file-system to currently comply with those requirements is the MOSIX File-System (See *MFS(7)*), but work is currently being undertaken to make the GFS file system comply as well. To use *DFSA* without violating access permissions, the permission-scheme (user and group ID's) must be identical, or at least compatible throughout the MOSIX cluster. Each partition that is to operate in *DFSA* mode, must be assigned a unique *DFSA* index (currently in the range of 1-8) that must be identical on all nodes in the MOSIX cluster. To request a particular partition to operate in *DFSA* mode, mount it with the *-odfsa={n}* flag, or if already mounted, type: *mount* *-o* *remount* *{mount-point}* *-odfsa={n}* *(1* *\<=* *n* *\<=* *8)*. You should do the same on all the nodes in the cluster either at about the same time or before MOSIX is configured (failure to assign all *DFSA* mount-points on some of the nodes is not fatal, but may result in serious performance degradation; simultaneous use of the same index for different partitions, however, is likely to cause various faults.) To disassociate a partition from *DFSA*, run: *mount* *-o* *remount* *{mount-point}* *-odfsa=0*. You may also designate symbolic-links to operate in *DFSA* mode: this is equivalent to a declaration that the given links are identical on all nodes and point to the same partition. It saves remote processes who use those link(s) the need to contact their home node every use in order to read those links. To declare a symbolic-link as identical, type: *echo* *{symbolic-link}* *\>* */proc/mosix/admin/dfsalinks*, where the symbolic-link must be an absolute-pathname, pointing at an existing file (or directory or another symbolic-link) on an already-mounted partition that is capable of *DFSA* (but it is not required to be already associated with a *DFSA* index). To remove a symbolic-link declaration, type: *echo* *-{symbolic-link}* *\>* */proc/mosix/admin/dfsalinks*: If you intend to re-define a declared symbolic link, you must first remove its declaration, then re-declare after the change is made. To see a list of all currently-declared symbolic links, type: *cat* */proc/mosix/admin/dfsalinks*. To cancel all symbolic-link declarations, type: *echo* *-* *\>* */proc/mosix/admin/dfsalinks*. The number of declared symbolic-links is currently limited to 8 and their path-name length is limited to 128 characters. ## REQUIREMENTS FROM A COMPLYING FILE-SYSTEM - Consistency All operations, even if performed from different nodes, must look as if they were sequentially synchornized on a single node. In the case of client-server file-systems, this requirement in particular does not allow private client-caches of either the data or the directory structure and in the case of shared-hardware file-systems, it probably calls for some sort of a hardware invalidation signal. - Times The time-stamps on files and between files of the same partition must be consistent and advancing - at least among all processes originating from the same home-node (unless the clock is deliberately set backwards), regardless from which node modifications are made. - New Functions The file-system must populate the following two new super-block methods: - *identify* Given a "dentry", encapsulate identifying information about it into a finite, rather-small structure, in a way that is sufficient to be able to re-establish that open file/directory on another node. - *reconstruct* Given only a mount-structure ("vfsmnt") and information that was provided by "identify", produce a live new "dentry". While not enforced by *DFSA*, in order for the *getcwd* system-call to work correctly on a shared file-system, regardless of where the call is made from, it is also highly recommended to populate the following new inode-method: - *checkpath* Given a "dentry", ensure that following its path via the directory-cache ("dcache") will truly reflect its current position on the shared file-system - and if not, make the necessary fixes by adjusting the "dentry" around the directory cache: The "dcache" of shared file-systems cannot be trusted, since processes running on other nodes can move (or remove) a directory at any time. - Removal The file-system must ensure that files/directories are not cleared when unlinked, for as long as any process in the cluster still holds them open. There are several possible techniques to achieve this, but given the distributed nature of the file-system, some form of garbage-collection is probably also called upon. ## SUPPORTED SYSTEM CALLS The following system-calls are supported by *DFSA* and will normally run directly by the process without resorting to its home-node, except for the "hard cases" (below). *read,* *readv,* *write,* *writev,* *readahead*\ *lseek,* *llseek*\ *open,* *creat,* *close*\ *dup,* *dup2,* *fcntl/fcntl64* *(F\_DUPFD,F\_GETFL,F\_SETFL)*\ *getdents,* *getdents64,* *old\_readdir*\ *fsync,* *fdatasync*\ *chdir,* *fchdir,* *getcwd*\ *stat,* *stat64,* *newstat,* *lstat,* *lstat64,* *newlstat,* *fstat,* *fstat64,* *newfstat*\ *access*\ *truncate,* *truncate64,* *ftruncate,* *ftruncate64*\ *chmod,* *chown,* *chown16,* *lchown,* *lchown16,* *fchmod,* *fchown,* *fchown16*\ *utime,* *utimes*\ *symlink,* *readlink*\ *mkdir,* *rmdir*\ *link,* *unlink,* *rename* Some examples of "hard cases" are: - \* If not all nodes have the same mounted *DFSA* partitions, or they do - but with different mount-flags. - \* If the calling process is being traced. - \* If the calling process has a non-standard root-directory. - \* If the calling process has an emulating personality that causes it to use an alternate root (but this is currently not relevant for the i386 architecture). - \* If the calling process shares either its files or current directory as a result of the *clone* system-call. - \* Operations during re-configuration of *DFSA* on either the home-node or the node where the process runs. - \* Operations involving special files (eg. other than regular, directories or symbolic-links) - \* Operations on files that were commonly opened and still shared with other related processes. - \* *dup2*, where the second file-descriptor is an already open non-DFSA file (that requires closing). - \* *chdir/fchdir* when the previous directory is non-DFSA. - \* *link/rename* that fail due to an attempt to cross-device link. - \* *open/dup/dup2/fcntl(F\_DUPFD)* that requires an allowable-increase in the maximal file-descriptor index (initially 1023!). - \* When the home-node has pending requests for the process (such as signals, requests for "ps" information, request to migrate or consider migration, etc.) - \* Use of path-names that leave the *DFSA* partition, as demonstrated by the following example: */mfs* is a *DFSA* file-system ;\ \ */mfstmp* is a symbolic link to */mfs/2/tmp* and is declared in */proc/mosix/admin/dfsalinks*;\ */mtmp* is a symbolic link to */mfstmp*, and is declared under */proc/mosix/admin/dfsalinks*;\ */mfs2* is a symbolic link to */mfs/2*, but is not declared ;\ On node 2, */fie* is a symbolic link to */tmp/foo*: then the following are accepted as simple cases (and identical): */mfs/2/tmp/foo*\ *//mfs//2/tmp/foo*\ */./mfs/2/tmp/foo*\ */mfstmp/foo*\ */mtmp/foo*\ */mfs/2/fie*\ *2/tmp/foo* (when in the /mfs directory) but not the following: - */tmp/../mfs/tmp/foo* (the kernel is not allowed to assume that each node has an accessible "/tmp" directory!) - */mfs/2/../../mfs/2/tmp/foo* (the second ".." steps out of the "/mfs" *DFSA* partition) - */mfs2/tmp/foo* (/mfs2 is not declared: no assurance was provided that it is identical on all nodes) - *mfs/2/foo* (or *mfstmp/foo*) when in the root directory (too difficult to recognize those cases) ## CAVEATS It was impossible to maintain 100% Linux/Posix compatibility on *DFSA* file-systems, but the deviations are kept to the very minimum: A process that received a signal may continue running a few *DFSA* system-calls before it actually receives and handles the signal. (in contrast, any POSIX process that receives a signal may possibly complete the next system-call, but cannot issue any new ones beyond that). Simultaneous mapping and I/O on the same *DFSA* file creates unpredictable results as follows: - :( execution (and library and all other file-mappings) is not always protected against other process(es) modifying the file (eg. either the writing-process or the executing/mapping process may fail to receive the "ETXTBSY" error). - :( The *MS\_INVALIDATE* *flag* *of* *msync* may fail to ensure that previous "write"(s) to a mapped *DFSA* file are discarded. - :( When a process modifies memory that is mapped as *MAP\_SHARED* to a *DFSA* file, but has not yet written it back (using *msync,* *munmap,* *exec* or *exit*, it is possible that another process that reads that file as it migrates will first see some of the changes but later (as opposed to normal behaviour) see the old values (or some of them) again. ## SEE ALSO [mosix(7)](/mosix/7.md), MFS(7).