1
2
3A FileSystem Abstraction System for Go
4
5[](https://github.com/spf13/afero/actions/workflows/test.yml) [](https://godoc.org/github.com/spf13/afero) [](https://gitter.im/spf13/afero?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
6
7# Overview
8
9Afero is a filesystem framework providing a simple, uniform and universal API
10interacting with any filesystem, as an abstraction layer providing interfaces,
11types and methods. Afero has an exceptionally clean interface and simple design
12without needless constructors or initialization methods.
13
14Afero is also a library providing a base set of interoperable backend
15filesystems that make it easy to work with afero while retaining all the power
16and benefit of the os and ioutil packages.
17
18Afero provides significant improvements over using the os package alone, most
19notably the ability to create mock and testing filesystems without relying on the disk.
20
21It is suitable for use in any situation where you would consider using the OS
22package as it provides an additional abstraction that makes it easy to use a
23memory backed file system during testing. It also adds support for the http
24filesystem for full interoperability.
25
26
27## Afero Features
28
29* A single consistent API for accessing a variety of filesystems
30* Interoperation between a variety of file system types
31* A set of interfaces to encourage and enforce interoperability between backends
32* An atomic cross platform memory backed file system
33* Support for compositional (union) file systems by combining multiple file systems acting as one
34* Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
35* A set of utility functions ported from io, ioutil & hugo to be afero aware
36* Wrapper for go 1.16 filesystem abstraction `io/fs.FS`
37
38# Using Afero
39
40Afero is easy to use and easier to adopt.
41
42A few different ways you could use Afero:
43
44* Use the interfaces alone to define your own file system.
45* Wrapper for the OS packages.
46* Define different filesystems for different parts of your application.
47* Use Afero for mock filesystems while testing
48
49## Step 1: Install Afero
50
51First use go get to install the latest version of the library.
52
53 $ go get github.com/spf13/afero
54
55Next include Afero in your application.
56```go
57import "github.com/spf13/afero"
58```
59
60## Step 2: Declare a backend
61
62First define a package variable and set it to a pointer to a filesystem.
63```go
64var AppFs = afero.NewMemMapFs()
65
66or
67
68var AppFs = afero.NewOsFs()
69```
70It is important to note that if you repeat the composite literal you
71will be using a completely new and isolated filesystem. In the case of
72OsFs it will still use the same underlying filesystem but will reduce
73the ability to drop in other filesystems as desired.
74
75## Step 3: Use it like you would the OS package
76
77Throughout your application use any function and method like you normally
78would.
79
80So if my application before had:
81```go
82os.Open("/tmp/foo")
83```
84We would replace it with:
85```go
86AppFs.Open("/tmp/foo")
87```
88
89`AppFs` being the variable we defined above.
90
91
92## List of all available functions
93
94File System Methods Available:
95```go
96Chmod(name string, mode os.FileMode) : error
97Chown(name string, uid, gid int) : error
98Chtimes(name string, atime time.Time, mtime time.Time) : error
99Create(name string) : File, error
100Mkdir(name string, perm os.FileMode) : error
101MkdirAll(path string, perm os.FileMode) : error
102Name() : string
103Open(name string) : File, error
104OpenFile(name string, flag int, perm os.FileMode) : File, error
105Remove(name string) : error
106RemoveAll(path string) : error
107Rename(oldname, newname string) : error
108Stat(name string) : os.FileInfo, error
109```
110File Interfaces and Methods Available:
111```go
112io.Closer
113io.Reader
114io.ReaderAt
115io.Seeker
116io.Writer
117io.WriterAt
118
119Name() : string
120Readdir(count int) : []os.FileInfo, error
121Readdirnames(n int) : []string, error
122Stat() : os.FileInfo, error
123Sync() : error
124Truncate(size int64) : error
125WriteString(s string) : ret int, err error
126```
127In some applications it may make sense to define a new package that
128simply exports the file system variable for easy access from anywhere.
129
130## Using Afero's utility functions
131
132Afero provides a set of functions to make it easier to use the underlying file systems.
133These functions have been primarily ported from io & ioutil with some developed for Hugo.
134
135The afero utilities support all afero compatible backends.
136
137The list of utilities includes:
138
139```go
140DirExists(path string) (bool, error)
141Exists(path string) (bool, error)
142FileContainsBytes(filename string, subslice []byte) (bool, error)
143GetTempDir(subPath string) string
144IsDir(path string) (bool, error)
145IsEmpty(path string) (bool, error)
146ReadDir(dirname string) ([]os.FileInfo, error)
147ReadFile(filename string) ([]byte, error)
148SafeWriteReader(path string, r io.Reader) (err error)
149TempDir(dir, prefix string) (name string, err error)
150TempFile(dir, prefix string) (f File, err error)
151Walk(root string, walkFn filepath.WalkFunc) error
152WriteFile(filename string, data []byte, perm os.FileMode) error
153WriteReader(path string, r io.Reader) (err error)
154```
155For a complete list see [Afero's GoDoc](https://godoc.org/github.com/spf13/afero)
156
157They are available under two different approaches to use. You can either call
158them directly where the first parameter of each function will be the file
159system, or you can declare a new `Afero`, a custom type used to bind these
160functions as methods to a given filesystem.
161
162### Calling utilities directly
163
164```go
165fs := new(afero.MemMapFs)
166f, err := afero.TempFile(fs,"", "ioutil-test")
167
168```
169
170### Calling via Afero
171
172```go
173fs := afero.NewMemMapFs()
174afs := &afero.Afero{Fs: fs}
175f, err := afs.TempFile("", "ioutil-test")
176```
177
178## Using Afero for Testing
179
180There is a large benefit to using a mock filesystem for testing. It has a
181completely blank state every time it is initialized and can be easily
182reproducible regardless of OS. You could create files to your heart’s content
183and the file access would be fast while also saving you from all the annoying
184issues with deleting temporary files, Windows file locking, etc. The MemMapFs
185backend is perfect for testing.
186
187* Much faster than performing I/O operations on disk
188* Avoid security issues and permissions
189* Far more control. 'rm -rf /' with confidence
190* Test setup is far more easier to do
191* No test cleanup needed
192
193One way to accomplish this is to define a variable as mentioned above.
194In your application this will be set to afero.NewOsFs() during testing you
195can set it to afero.NewMemMapFs().
196
197It wouldn't be uncommon to have each test initialize a blank slate memory
198backend. To do this I would define my `appFS = afero.NewOsFs()` somewhere
199appropriate in my application code. This approach ensures that Tests are order
200independent, with no test relying on the state left by an earlier test.
201
202Then in my tests I would initialize a new MemMapFs for each test:
203```go
204func TestExist(t *testing.T) {
205 appFS := afero.NewMemMapFs()
206 // create test files and directories
207 appFS.MkdirAll("src/a", 0755)
208 afero.WriteFile(appFS, "src/a/b", []byte("file b"), 0644)
209 afero.WriteFile(appFS, "src/c", []byte("file c"), 0644)
210 name := "src/c"
211 _, err := appFS.Stat(name)
212 if os.IsNotExist(err) {
213 t.Errorf("file \"%s\" does not exist.\n", name)
214 }
215}
216```
217
218# Available Backends
219
220## Operating System Native
221
222### OsFs
223
224The first is simply a wrapper around the native OS calls. This makes it
225very easy to use as all of the calls are the same as the existing OS
226calls. It also makes it trivial to have your code use the OS during
227operation and a mock filesystem during testing or as needed.
228
229```go
230appfs := afero.NewOsFs()
231appfs.MkdirAll("src/a", 0755)
232```
233
234## Memory Backed Storage
235
236### MemMapFs
237
238Afero also provides a fully atomic memory backed filesystem perfect for use in
239mocking and to speed up unnecessary disk io when persistence isn’t
240necessary. It is fully concurrent and will work within go routines
241safely.
242
243```go
244mm := afero.NewMemMapFs()
245mm.MkdirAll("src/a", 0755)
246```
247
248#### InMemoryFile
249
250As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
251backed file implementation. This can be used in other memory backed file
252systems with ease. Plans are to add a radix tree memory stored file
253system using InMemoryFile.
254
255## Network Interfaces
256
257### SftpFs
258
259Afero has experimental support for secure file transfer protocol (sftp). Which can
260be used to perform file operations over a encrypted channel.
261
262### GCSFs
263
264Afero has experimental support for Google Cloud Storage (GCS). You can either set the
265`GOOGLE_APPLICATION_CREDENTIALS_JSON` env variable to your JSON credentials or use `opts` in
266`NewGcsFS` to configure access to your GCS bucket.
267
268Some known limitations of the existing implementation:
269* No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
270* No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
271* Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
272
273
274## Filtering Backends
275
276### BasePathFs
277
278The BasePathFs restricts all operations to a given path within an Fs.
279The given file name to the operations on this Fs will be prepended with
280the base path before calling the source Fs.
281
282```go
283bp := afero.NewBasePathFs(afero.NewOsFs(), "/base/path")
284```
285
286### ReadOnlyFs
287
288A thin wrapper around the source Fs providing a read only view.
289
290```go
291fs := afero.NewReadOnlyFs(afero.NewOsFs())
292_, err := fs.Create("/file.txt")
293// err = syscall.EPERM
294```
295
296# RegexpFs
297
298A filtered view on file names, any file NOT matching
299the passed regexp will be treated as non-existing.
300Files not matching the regexp provided will not be created.
301Directories are not filtered.
302
303```go
304fs := afero.NewRegexpFs(afero.NewMemMapFs(), regexp.MustCompile(`\.txt$`))
305_, err := fs.Create("/file.html")
306// err = syscall.ENOENT
307```
308
309### HttpFs
310
311Afero provides an http compatible backend which can wrap any of the existing
312backends.
313
314The Http package requires a slightly specific version of Open which
315returns an http.File type.
316
317Afero provides an httpFs file system which satisfies this requirement.
318Any Afero FileSystem can be used as an httpFs.
319
320```go
321httpFs := afero.NewHttpFs(<ExistingFS>)
322fileserver := http.FileServer(httpFs.Dir(<PATH>))
323http.Handle("/", fileserver)
324```
325
326## Composite Backends
327
328Afero provides the ability have two filesystems (or more) act as a single
329file system.
330
331### CacheOnReadFs
332
333The CacheOnReadFs will lazily make copies of any accessed files from the base
334layer into the overlay. Subsequent reads will be pulled from the overlay
335directly permitting the request is within the cache duration of when it was
336created in the overlay.
337
338If the base filesystem is writeable, any changes to files will be
339done first to the base, then to the overlay layer. Write calls to open file
340handles like `Write()` or `Truncate()` to the overlay first.
341
342To writing files to the overlay only, you can use the overlay Fs directly (not
343via the union Fs).
344
345Cache files in the layer for the given time.Duration, a cache duration of 0
346means "forever" meaning the file will not be re-requested from the base ever.
347
348A read-only base will make the overlay also read-only but still copy files
349from the base to the overlay when they're not present (or outdated) in the
350caching layer.
351
352```go
353base := afero.NewOsFs()
354layer := afero.NewMemMapFs()
355ufs := afero.NewCacheOnReadFs(base, layer, 100 * time.Second)
356```
357
358### CopyOnWriteFs()
359
360The CopyOnWriteFs is a read only base file system with a potentially
361writeable layer on top.
362
363Read operations will first look in the overlay and if not found there, will
364serve the file from the base.
365
366Changes to the file system will only be made in the overlay.
367
368Any attempt to modify a file found only in the base will copy the file to the
369overlay layer before modification (including opening a file with a writable
370handle).
371
372Removing and Renaming files present only in the base layer is not currently
373permitted. If a file is present in the base layer and the overlay, only the
374overlay will be removed/renamed.
375
376```go
377 base := afero.NewOsFs()
378 roBase := afero.NewReadOnlyFs(base)
379 ufs := afero.NewCopyOnWriteFs(roBase, afero.NewMemMapFs())
380
381 fh, _ = ufs.Create("/home/test/file2.txt")
382 fh.WriteString("This is a test")
383 fh.Close()
384```
385
386In this example all write operations will only occur in memory (MemMapFs)
387leaving the base filesystem (OsFs) untouched.
388
389
390## Desired/possible backends
391
392The following is a short list of possible backends we hope someone will
393implement:
394
395* SSH
396* S3
397
398# About the project
399
400## What's in the name
401
402Afero comes from the latin roots Ad-Facere.
403
404**"Ad"** is a prefix meaning "to".
405
406**"Facere"** is a form of the root "faciō" making "make or do".
407
408The literal meaning of afero is "to make" or "to do" which seems very fitting
409for a library that allows one to make files and directories and do things with them.
410
411The English word that shares the same roots as Afero is "affair". Affair shares
412the same concept but as a noun it means "something that is made or done" or "an
413object of a particular type".
414
415It's also nice that unlike some of my other libraries (hugo, cobra, viper) it
416Googles very well.
417
418## Release Notes
419
420See the [Releases Page](https://github.com/spf13/afero/releases).
421
422## Contributing
423
4241. Fork it
4252. Create your feature branch (`git checkout -b my-new-feature`)
4263. Commit your changes (`git commit -am 'Add some feature'`)
4274. Push to the branch (`git push origin my-new-feature`)
4285. Create new Pull Request
429
430## Contributors
431
432Names in no particular order:
433
434* [spf13](https://github.com/spf13)
435* [jaqx0r](https://github.com/jaqx0r)
436* [mbertschler](https://github.com/mbertschler)
437* [xor-gate](https://github.com/xor-gate)
438
439## License
440
441Afero is released under the Apache 2.0 license. See
442[LICENSE.txt](https://github.com/spf13/afero/blob/master/LICENSE.txt)
View as plain text