VFS: A Virtual File System

go/vfs provides a virtual file system implementation for Go that is robust, extensive, and provides compatibility with multiple Go versions.

Why this Library?

This library predates the Go 1.16 efforts to add an FS layer to the io/fs package and, as such, contains some anachronisms that may seem somewhat surprising to users familiar with the post-1.16 standard library. However, with recent updates to go/vfs, we are now mostly at parity with io/fs.

This library is different in that users may target pre- or post-Go 1.16, toggling compatibility features on or off using the go116 build tag (go build -tags=go116) to enable compatibility with Go 1.16+. By default, go/vfs targets Go versions prior to 1.16, though this default will be changing possibly by the Go 1.17 release.

In particular, we have exposed mostly identical interfaces for constructs like DirEntry and similar such that using go/vfs will require virtually no changes to your code if you migrate to later versions of Go. Likewise, backporting your code using go/vfs should require few changes when targeting earlier pre-1.16 Go installs.

go/vfs also provides, out of the box, a .zip-based VFS layer, a VFS layer that interacts with most file systems, and a layered VFS type that can transparently merge multiple VFSes into a single interface with the option to override missing or older files with newer entries.

In particular, we also offer a Chroot() and SetRoot() functionality that changes the internal root of the VFS to "lock" the file system view to a particular path, inhibiting alternations to paths that exist outside that root while using the same VFS object. These are similar to io/fs.Sub().

Likewise, our persistent FS type deliberately eschews a traditional file system view and will lock itself to the path you've configured during its initialization. You can confidently interact with VFS instances knowing that errant path specifications will not expose other parts of your file system.

Core Interfaces

VFS exposes three primary interfaces: FS, VFS, and MutableFS.

FS is mostly compatible with io/fs, exposing a unified type that contains each of the core methods that appear in various segregated interfaces in io/fs (with the exception of SubFS) and may act as a stand-in for each of the interfaces presented in this package. SubFS is deliberately not included in go/vfs.FS as implementers of our VFS interface may choose whether to support the SetRoot/Chroot mechanics independently of SubFS or they may choose to expose Sub for SubFS compatibility.

FS also differs from the file system interface types in io/fs by requiring that its constituents implement Lstat as well. This is not included in the io/fs package for any file system type.

VFS is the primary wrapper type for the go/vfs library and is the target type that implementers should focus on when authoring their own VFS backends. This type provides--and requires--all of the expected primitives that would be available in your typical file system, such as Abs, Cwd, and a few other utility methods like ReadOnly and SetReadOnly. Some methods like MaskedAbs are explained elsewhere in this document.

MutableFS exposes methods that provide mutational capabilities for the underlying file system. File systems that expose these methods should provide some backing for their intended functions; e.g., Mkdir should do something analogously to creating a directory, and OpenFile should provide a File implementation that handles Write. Implementers may choose to chain together other constructs, like bytes.Buffer for backing stores that don't provide a byte stream or anything analogous to one (such as databases).

Not all file systems may expose all interfaces at any given time. In fact, we have supplyed a WrapAsMutable() function that returns any type implementing VFS inside a type that implements MutableFS with all mutable functions returning an ErrReadOnly when called. This wrapper may be used for VFS types that are only intended to interact with read only constructs, such as archives, saving the developer some time when offering compatibility with each of the go/vfs interfaces.

Quick Start

go/vfs is quite easy to use. This section highlight the most basic usage patterns to help you get started as fast as possible. More detailed documentation may be found in the docs/ directory included with every go/vfs distribution.

Loading a File System

To load and attach a file system:

fs, err := vfs.NewVFS("path/to/directory")
if err != nil {
    // Handle errors.
}

// ...
fp, err := vfs.Open("somefile.txt")

// ... do something with `fp` ...

// Read the entire contents of a file. Note that ReadFile is analogous to
// io.ReadFile or ioutils.ReadFile.
contents, err := vfs.ReadFile(fs, "anotherfile.txt")

// vfs.WriteFile is analogous to ioutil.WriteFile an io.WriteFile.
if err := vfs.WriteFile(fs, "writeme.txt", []byte("Hello!"), os.FileMode(0644); err != nil {
    // Handle error... note that `fs` must implement MutableFS or else an error
    // will be raised here.
}

// Get a mutable instance of the file system to do naughty things with:
mut, err := fs.Mutable()
if err == nil {
    if err := mut.Mkdir("bananas"); err == nil {
        vfs.WriteFile(mut, "tasty.txt", []byte("Tasty bananas."), os.FileMode(0644)
    }
}

Loading an Archive

Presently, VFS supports loading .zip archives out of the box:

fs, err := vfs.NewZipFS("path/to/zipfile.zip")
if err != nil {
    // Handle errors.
}

if content, err := vfs.ReadFile(fs, "some/file.txt"); err == nil {
    fmt.Println(content)
}

Loading Multiple File Systems

It's possible layer multiple file systems together. The first file system supplied to the AddFS method will be checked first for files.

Given a directory structured:

animals/cats/
animals/cats/mainecoon.jpg
animals/cats/funny_cat.png
animals/dogs/
animals/dogs/barking.mp4

and an archive containing:

animals.zip => animals/birds/dancing.mp4
animals.zip => animals/dogs/barking.mp4

we can layer these file systems as:

fs := vfs.NewLayeredFS()
if ffs, err := vfs.NewVFS("animals"); err == nil {
    fs.AddFS(ffs)
}
if zipfs, err := vfs.NewZipFS("animals.zip"); err == nil {
    fs.AddFS(zipfs)
}

In this example, the first file system (persistent FS; using the host file system on disk) will take precdence over the .zip archive. This means that these calls will succeed and read the requested files from the file system:

contents, err := vfs.ReadFile(fs, "cats/mainecoon.jpg")

while this call will read from the .zip archive:

contents, err := vfs.ReadFile(fs, "birds/dancing.mp4")

and this call will read barking.mp4 from the file system rather than the ZIP file:

contents, err := vfs.ReadFile(fs, "dogs/barking.mp4")

File System Types

In this distribution, there are three primary file system types, each of which were highlighted above:

Persistent File System: vfs.NewVFS

The persistent file system type accepts a single directory as its argument. This argument is bound as the VFS "root" directory and files or directories that exist outside this structure cannot be read. Only those files and directoreis present in the working directory of the persistent file system root or its children can be viewed.

Symlinks are handled differently depending on how this VFS is configured. We expose a HandleSymlinks() function that accepts a single boolean argument. If this argument is set to true, the persistent VFS will attempt to resolve and follow symbolic links if and only if they exist within the confines of the VFS file system. If the argument to HandleSymlinks() is false or otherwise left untouched (this is the default), symlinks are ignored and no effort is made to resolve their target.

Zip Archive File System: vfs.NewZipFS

Layered File System: vfs.NewLayeredFS

License

go/vfs is distributed under the terms and conditions of the NCSA license. See LICENSE for details.

In short, the NCSA is analogous to both the 2-clause BSD and MIT licenses with additional dispensations made for included documentation.