UBIFS - UBI File-System

Table of contents

  1. Big red note
  2. Overview
  3. Source code
  4. Mailing list
  5. User-space tools
  6. Scalability
  7. Write-back support
  8. Write-back knobs in Linux
  9. UBIFS write-buffer
  10. UBIFS in synchronous mode vs JFFS2
  11. Synchronization exceptions for buggy applications
  12. Compression
  13. Checksumming
  14. Read-ahead
  15. Space for superuser
  16. Extended attributes
  17. Mount options
  18. Flash space accounting issues
  19. Documentation
  20. How to send an UBIFS bugreport?
  21. Raw flash vs. FTL devices

Big red note

One thing people have to understand when dealing with UBIFS is that UBIFS is very different to any traditional file system - it does not work on top of block devices (like hard drives, MMC/SD cards, USB flash drives, SSDs, etc). UBIFS was designed to work on top of raw flash, which has nothing to do with block devices. This is why UBIFS does not work on MMC cards and the like - they look like block devices to the outside world because they implement FTL (Flash Translation Layer) support in hardware, which simply speaking emulates a block device on top of the built-in raw flash. Please, make sure you understand the differences between raw flash and, say, MMC flash before dealing with UBIFS. This section should help.

Overview

UBIFS is a new flash file system developed by Nokia engineers with help of the University of Szeged. In a way, UBIFS may be considered as the next generation of the JFFS2 file-system.

JFFS2 file system works on top of MTD devices, but UBIFS works on top of UBI volumes and cannot operate on top of MTD devices. In other words, there are 3 subsystems involved:

Here is a list of some of UBIFS features:

Source code

UBIFS is in mainline since 17 July 2008 and the first kernel release which contains UBIFS is 2.6.27. But since UBIFS is a very new file system, it is recommended to pick the latest updates and fixes from the linux-next branch of the UBIFS git tree:

git://git.infradead.org/ubifs-2.6.git

Here is the corresponding Git-web view.

The git tree has master and linux-next branches. The master branch contains the most recent stuff which is often incomplete, buggy, or not tested very well. This branch may be re-based from time to time. The linux-next branch contains stable UBIFS updates and fixes. This branch is included to the linux-next git tree and goes to main-line. The linux-next branch is never re-based. Thus, unless you are an active UBIFS developer, use the linux-next branch.

There are also 2.6.24, 2.6.25, 2.6.26 2.6.27, 2.6.28, 2.6.29, 2.6.30, and 2.6.31 kernel back-ports which may be found here:

git://git.infradead.org/~dedekind/ubifs-v2.6.31.git
git://git.infradead.org/~dedekind/ubifs-v2.6.30.git
git://git.infradead.org/~dedekind/ubifs-v2.6.29.git
git://git.infradead.org/~dedekind/ubifs-v2.6.28.git
git://git.infradead.org/~dedekind/ubifs-v2.6.27.git
git://git.infradead.org/~dedekind/ubifs-v2.6.26.git
git://git.infradead.org/~dedekind/ubifs-v2.6.25.git
git://git.infradead.org/~dedekind/ubifs-v2.6.24.git

Mailing list

You are welcome to send feed-back, bug-reports, patches, etc to the MTD mailing list. Feel free to ask questions, but please, check the UBIFS FAQ before doing this.

User-space tools

There is only one UBIFS user-space tool at the moment - mkfs.ubifs, which creates UBIFS images. The tool may be found in the MTD utils repository (the mkfs.ubifs sub-directory):

git://git.infradead.org/mtd-utils.git

The images produced by mkfs.ubifs may be written to UBI volumes using ubiupdatevol or may be further fed to the ubinize tool to create an UBI image which may be put to the raw flash.

Scalability

All the data structures UBIFS is using are trees, so it scales logarithmically in terms of flash size. However, UBI scales linearly (see here) which makes overall UBI/UBIFS stack scalability to be linear. But UBIFS authors believe it is possible to create logarithmically scalable UBI2 and improve the situation. Current UBI should be OK for 2-16GiB raw flashes, depending on the I/O speed and the requirements.

Note, although UBI scalability is linear, it anyway scales much better than JFFS2, which was originally designed for small ~32MiB NOR flashes. JFFS2 has scalability issues on the "file system level", while UBI/UBIFS stack has scalability issues only on lower "raw flash level". The following table describes the issues in more details.

Scalability issue JFFS2 UBIFS
Mount time linearly depends on the flash size True, the dependency is linear, because JFFS2 has to scan whole flash media when mounting. UBIFS mount time does not depend on the flash size. But UBI needs to scan the flash media, which is actually quicker than JFFS2 scanning. So overall, UBI/UBIFS has this linear dependency.
Memory consumption linearly depends on the flash size True, the dependency is linear. UBIFS memory does depend on the flash size in the current implementation, because the LPT shrinker is not implemented. But it is not difficult to implement the LPT shrinker and get rid of the dependency. It is not implemented only because the memory consumption is too small to make the coding work worth it. UBI memory consumption linearly depends on flash size. Thus, overall UBI/UBIFS stack has the linear dependency.
Mount time linearly depends on the file system contents True, the more data are stored on the file system, the longer it takes to mount it, because JFFS2 has to do more scanning work. False, mount time does not depend on the file system contents. At the worst case (if there was an unclean reboot), UBIFS has to scan and replay the journal which has fixed and configurable size.
Full file system checking is required after each mount True. JFFS2 has to check whole file system just after it has been mounted in case of NAND flash. The checking involves reading all nodes for each inode and checking their CRC checksums, which consumes a lot of CPU. For example, this may be observed by running top just after JFFS2 has been mounted. This slows down overall system boot-up time. Fundamentally, this is needed because JFFS2 does not store space accounting information (i.e., free/dirty space) on the flash media but instead, gathers this information by scanning the flash media. False. UBIFS does not scan/check whole file system because it stores the space accounting information on the flash media in the so-called LPT (Logical eraseblock Properties Tree) tree.
Memory consumption linearly depends on file system contents True. JFFS2 keeps a small data structure in RAM for each node on flash, so the more data are stored on the flash media, the more memory JFFS2 consumes. False. UBIFS memory consumption does not depend on how much data are stored on the flash media.
File access time linearly depends on its size True. JFFS2 has to keep in RAM so-called "fragment tree" for each inode corresponding to an opened file. The fragment tree is an in-memory RB-tree which is indexed by file offset and refers on-flash nodes corresponding to this offset. The fragment tree is not stored on the flash media. Instead, it is built on-the-flight when the file is opened for the first time. To build the fragment tree, JFFS2 has to read each data node corresponding to this inode from the flash. This means, the larger is the file, the longer it takes to open it for the first time. And the larger is the file the more memory it takes when it is opened. Depending on the system, JFFS2 becomes nearly unusable starting from certain file size. False. UBIFS stores all the indexing information on the media in the indexing B-tree. Whenever a piece of data has to be read from the file system, the B-tree is looked-up and the corresponding flash address to read is found. There is a TNC cache which caches the B-tree nodes when the B-tree is looked-up, and the cache is shrinkable, which means it can be shrunk when the kernel needs more memory.
File-system performance depends on I/O history True. Since JFFS2 is fully synchronous, it writes data to the flash media as soon as the data arrives. If one changes few bytes in the middle of a file, JFFS2 writes a data node which contains those bytes to the flash. If there are many random small writes all over the place, the file system becomes fragmented. JFFS2 merges small fragments to 4KiB chunks, which involves re-compression and re-writing the data. But this "de-fragmentation" is happening during garbage collection and at random time, because JFFS2 wear-leveling algorithm is based on random eraseblock selection. So if there were a lot of small writes, JFFS2 becomes slower some time later - the performance just goes down out of the blue which makes the system less predictable. False. UBIFS always writes in 4KiB chunks. This does not hurt the performance much because of the write-back support: the data changes do not go to the flash straight away - they are instead deferred and are done later, when (hopefully) more data are changed at the same data page. And write-back usually happens in background.

Write-back support

UBIFS supports write-back, which means that file changes do not go to the flash media straight away, but they are cached and go to the flash later, when it is absolutely necessary. This helps to greatly reduce the amount of I/O which results in better performance. Write-back caching is a standard technique which is used by most file systems like ext3 or XFS.

In contrast, JFFS2 does not have write-back support and all the JFFS2 file system changes go the flash synchronously. Well, this is not completely true and JFFS2 does have a small buffer of a NAND page size (if the underlying flash is NAND). This buffer contains last written data and is flushed once it is full. However, because the amount of cached data are very small, JFFS2 is very close to a synchronous file system.

Write-back support requires the application programmers to take extra care about synchronizing important files in time. Otherwise the files may corrupt or disappear in case of power-cuts, which happens very often in many embedded devices. Let's take a glimpse at Linux manual pages:

$ man 2 write
....
NOTES
       A  successful return from write() does not make any guarantee that data
       has been committed to disk.  In fact, on some buggy implementations, it
       does  not  even guarantee that space has successfully been reserved for
       the data.  The only way to be sure is to call fsync(2)  after  you  are
       done writing all your data.
...

This is true for UBIFS (except of the "some buggy implementations" part, because UBIFS does reserves space for cached dirty data). This is also true for JFFS2, as well as for any other Linux file system.

However, some (perhaps not very good) user-space programmers do not take write-back into account. They do not read manual pages carefully. When such applications are used in embedded systems which run JFFS2 - they work fine, because JFFS2 is almost synchronous. Of course, the applications are buggy, but they appear to work well enough with JFFS2. But the bugs show up when UBIFS is used instead. Please, be careful and check/test your applications with respect to power cut tolerance if you switch from JFFS2 to UBIFS. The following is a list of useful hints and advices.

Let us stress that the above items are true for any Linux file system, including JFFS2.

fsync() may be called for directories - it synchronizes the directory inode meta-data. The "sync" flag may also be set for directories to make the directory inode synchronous. But the flag is inherited, which means all new children of this directory will also have this flag. New files and sub-directories of this directory will also be synchronous, and their children, and so forth. This feature is very useful if one needs to create a whole sub-tree of synchronous files and directories, or to make all new children of some directory to be synchronous by default (e.g., /etc).

The fdatasync() call for directories is "no-op" in UBIFS and all UBIFS operations which change directory entries are synchronous. However, you should not assume this for portability (e.g., this is not true for ext2). Similarly, the "dirsync" inode flag has no effect in UBIFS.

The functions mentioned above work on file-descriptors, not on streams (FILE *). To synchronize a stream, you should first get its file descriptor using the fileno() libc function, then flush the stream using fflush(), and then synchronize the file using fsync() or fdatasync(). You may use other synchronization methods, but remember to flush the stream before synchronizing the file. The fflush() function flushes the libc-level buffers, while sync(), fsync(), etc flush kernel-level buffers.

Please, refer this FAQ entry for information about how to atomically update the contents of a file. Also, the Theodore Tso's article is a good reading.

Write-back knobs in Linux

Linux has several knobs in "/proc/sys/vm" which you may use to tune write-back. The knobs are global, so they affect all file-systems. Please, refer the "Documentation/sysctl/vm.txt" file fore more information. The file may be found in the Linux kernel source tree. Below are interesting knobs described in UBIFS context and in a simplified form.

Note, UBIFS additionally has small write-buffers which are synchronized every 3-5 seconds. This means that most of the dirty data are delayed by dirty_expire_centisecs centi-seconds, but the last few KiB are additionally delayed by 3-5 seconds.

UBIFS write-buffer

UBIFS is asynchronous file-system (read this section for more information). As other Linux file-system, it utilizes the page cache. The page cache is a generic Linux memory-management mechanism. It may be very large and cache a lot of data. When you write to a file, the data are written to the page cache, marked as dirty, and the write returns (unless the file is synchronous). Later the data are written-back.

Write-buffer is an additional UBIFS buffer, which is implemented inside UBIFS, and it sits between the page cache and the flash. This means that write-back actually writes to the write-buffer, not directly to the flash.

The write-buffer is designated to speed-up UBIFS on NAND flashes. NAND flashes consist of NAND pages, which are usually 512, 2KiB or 4KiB in size. NAND page is the minimal read/write unit of NAND flash (see this section).

Write-buffer size is equivalent to NAND page size (so it is tiny comparing to the page cache). It's purpose is to accumulate small writes, and write full NAND pages instead of partially filled. Indeed, imagine we have to write 4 512-byte nodes with half a second interval, and NAND page size is 2KiB. Without write-buffer we would have to write 4 NAND pages and waste 6KiB of flash space, while write-buffer allows us to write only once and waste nothing. This means we write less, we create less dirty space so UBIFS garbage collector will have to do less work, we save power.

Well, the example shows an ideal situation, and even with the write-buffer we may waste space, for example in case of synchronous I/O, or if the data arrives with long time intervals. This is because the write-buffer has an associated timer, which flushes it every 3-5 seconds, even if it isn't full. We do this for data integrity reasons.

Of course, when UBIFS has to write a lot of data, it does not use write buffer. Only the last part of the data which is smaller than the NAND page ends up in the write-buffer and waits more for data, until it is flushed by the timer.

The write-buffer implementation is a little more complex, and we actually have several of them - one for each journal head. But this does not change the basic idea behind the write-buffer.

Few notes with regards to synchronization:

Take into account that write-buffers delay the data synchronization timeout defined by "dirty_expire_centisecs" (see here) by 3-5 seconds. However, since write-buffers are small, only few data are delayed.

UBIFS in synchronous mode vs JFFS2

When UBIFS is mounted in synchronous mode (-o sync mount options) - all file system operations become synchronous. This means that all data are written to flash before the file-system operations return.

For example, if you write 10MiB of data to a file f.dat using the write() call, and UBIFS is in synchronous mode, then UBIFS guarantees that all 10MiB of data and the meta-data (file size and date changes) will reach the flash media before write() returns. And if a power cut happens after the write() call returns, the file will contain the written data.

The same is true for situations when f.dat has was opened with O_SYNC or has the sync flag (see man 2 chattr).

It is well-known that the JFFS2 file-system is synchronous (except a small write-buffer). However, UBIFS in synchronous mode is not the same as JFFS2 and provides somewhat less guarantees that JFFS2 does with respect to sudden power cuts.

In JFFS2 all the meta-data (like inode atime/mtime/ctime, inode size, UID/GID, etc) are stored in the data node headers. Data nodes carry 4KiB of (compressed) data. This means that the meta-data information is duplicated in many places, but this also means that every time JFFS2 writes a data node to the flash media, it updates inode size as well. So when JFFS2 mounts it scans the flash media, finds the latest data node, and fetches the inode size from there.

In practice this means that JFFS2 will write these 10MiB of data sequentially, from the beginning to the end. And if you have a power cut, you will just lose some amount of data at the end of the inode. For example, if JFFS2 starts writing those 10MiB of data, write 5MiB, and a power cut happens, you will end up with a 5MiB f.dat file. You lose only the last 5MiB.

Things are a little bit more complex in case of UBIFS, where data are stored in data nodes and meta-data are stored in (separate) inode nodes. The meta-data are not duplicated in each data node, like in JFFS2. UBIFS never writes data nodes beyond the on-flash inode size. If it has to write a data node and the data node is beyond the on-flash inode size (the in-memory inode has up-to-data size, but it is dirty and was not flushed yet), then UBIFS first writes the inode to the media, and then it starts writing the data. And if you have an interrupt, you lose data nodes and you have holes (or old data nodes, if you are overwriting). Lets consider an example.

Note, if the I/O was not synchronous, UBIFS would skip the last step and would just return. And the actual write-back would then happen in back-ground. But power cuts during write-back could anyway lead to files with holes at the end.

Thus, synchronous I/O in UBIFS provides less guarantees than JFFS2 I/O - UBIFS has an effect of holes at the end of files. In ideal world applications should not assume anything about the contents of files which were not synchronized before a power-cut has happened. And "mainstream" file-systems like ext3 do not provide JFSS2-like guarantees.

However, UBIFS is sometimes used as a JFFS2 replacement and people may want it to behave the same way as JFFS2 if it is mounted synchronously. This is doable, but needs some non-trivial development, so this was not implemented so far. On the other hand, there was no strong demand. You may implement this as an exercise, or you may try to convince UBIFS authors to do this.

Synchronization exceptions for buggy applications

As this section describes, UBIFS is an asynchronous file-system, and applications should synchronize their files whenever it is required. The same applies to most Linux file-systems, e.g. XFS.

However, many applications ignore this and do not synchronize files properly. And there was a huge war between user-space and kernel developers related to ext4 delayed allocation feature. Please, see the Theodore Tso's blog post. More information may be found in this LWN article.

In short, the flame war was about 2 cases. The first case was about the atomic re-name, where many user-space programs did not synchronize the copy before re-naming it. The second case was about applications which truncate files, then change them. There was no final agreement, but the "we cannot ignore the real world" argument found ext4 developers' understanding, and there were 2 ext4 changes which help both problems.

Roughly speaking, the first change made ext4 synchronize files on close if they were previously truncated. This was a hack from file-system point of view, but it "fixed" applications which truncate files, write new contents, and close the files without synchronizing them.

The second change made ext4 synchronize the renamed file.

Well, this is not exactly correct description, because ext4 does not write the files synchronously, but actually initiates asynchronous write-out of the files, so the performance hit is not very high. For the truncation case this means that the file is synchronized soon after it is closed. For the re-name case this means that ext4 writes data before it writes the re-name meta-data.

However, the application writers should never rely on these things, because this is not portable. Instead, they should properly synchronize files. The ext4 fixes were because there were many broken user-space applications in the wild already.

We have plans to implement these features in UBIFS, but this has not been done yet. The problem is that UBI/MTD are fully synchronous and we cannot initiate asynchronous write-out, so we'd have to synchronously write files on close/rename, which is slow. So implementing these features would require implementing asynchronous I/O in UBI, which is a big job. But feel free to do this :-).

Compression

UBIFS supports on-the-flight compression, which means it compresses data before writing them to the flash media, and de-compresses before reading them, and this is absolutely transparent to the users. UBIFS compresses only regular files data. Directories, device nodes and so on are not compressed. Meta-data and the indexing information are not compressed as well.

At the moment UBIFS supports LZO and zlib compressors. Zlib provides better compression ratio, but LZO is faster in both compression and decompression. LZO is the default compressor for UBIFS and for the mkfs.ubifs utility. And of course you may disable UBIFS compression altogether using the "-x none" mkfs.ubifs option.

UBIFS splits all data on 4KiB chunks and compresses each chunk independently. This is not optimal, because larger chunks of data would compress better, but this still provides noticeable flash space economy. For example, real-life root file-system image for an ARM platform becomes ~40% smaller with LZO compression and ~50% smaller with zlib compression. This means that you may fit a 300MiB rootfs image into a 256MiB UBI volume and still have about 100MiB of free space. However, the figures may be different depending on the contents of the file-system. For example, if your file-system mostly contains mp3 files, UBIFS will be unable to efficiently compress them, just because mp3 files are already compressed.

In UBIFS it is possible to enable or disable compression individually for each inode by setting or cleaning their compression flag. Note, the compression flag of directories is inherited, which means that when files and sub-directories are created, they inherit the compression flag of the parent directory. Please, refer this section for instruction about how the compression flag may be toggled.

It is also possible to somewhat combine LZO and zlib compressors, see this FAQ section.

It's also worth noting that JFFS2 LZO compression is a little bit different to UBIFS zlib compression. UBIFS uses the crypto-API deflate method, while JFFS2 uses zlib library directly. As a result, UBIFS and JFFS2 use different zlib compression options. Namely, JFFS2 uses deflate level 3 and window bits 15, while UBIFS uses deflate level 6 and window bits -11 (minus makes zlib avoid putting a header to the output data stream). Experiments with compressing ARM code showed that JFFS2 compression ratio is slightly smaller, decompression speed is also slightly slower, but compression speed is a bit faster.

Checksumming

Every piece of information UBIFS writes to the media has a CRC-32 checksum, and UBIFS verifies the checksum for every piece of information it reads from the media. CRC-32 is quite strong function and any data corruption will most probably be noticed. The same is true for UBI.

CRC-32 loads the CPU and makes the file-system slower - this is the price we pay for providing very high data integrity level. But UBIFS allows to switch the data checksumming off using the no_chk_data_crc mount option. If UBIFS is mounted with this option, it does not check CRC-32 checksum for data, but it does check it for the internal indexing information. And this option only affects reading, not writing, because UBIFS always calculates CRC-32 when writing the data.

Disabling checksum verification for data speeds-up file-system read speed and reduces CPU usage. But of course, it also lowers the file-system integrity level, so you should decide whether you want to use it or not depending on your system requirements. In general, if you use SLC NAND flash or NOR flash, it is probably fine to disable CRC-32. In case of MLC NAND flash, you should probably be more careful. However, see this FAQ section for more information about UBIFS on MLC NAND.

Note, currently UBIFS cannot disable CRC-32 calculations on write, because UBIFS recovery process depends on in. When recovering from an unclean reboot and re-playing the journal, UBIFS has to be able to detect broken and half-written UBIFS nodes and drop them, and UBIFS depends on the CRC-32 checksum here. So the no_chk_data_crc mount option does not improve UBIFS write speed. However, UBIFS writes speed should not be a problem for a great deal of standard work-loads because of write-back support.

In other words, if you use UBIFS with data CRC-32 checking disabled, you still have the CRC-32 checksum attached to each piece of data, and you may mount UBIFS with default options to enable CRC-32 checking at any time (e.g., when you suspect the file-system might be corrupted because you visited the Large Hadron Collider and exposed your flash to proton beams).

Read-ahead

Read-ahead is an optimization technique which makes the file system read a little bit more data than users actually ask. The idea is that files are often read sequentially from the beginning to the end, so the file system tries to make next data available before the user actually asks for them.

Linux VFS is capable of doing read-ahead and this does not require any support from the file system. This probably works well for traditional block-based file systems, however this does not work well for UBIFS. UBIFS works with UBI API, which works with MTD API, which is synchronous. MTD API is pretty trivial and does not have any request queues. This means that VFS blocks UBIFS readers and makes them wait for read-ahead process. In opposite, block-device API is asynchronous and readers do not wait for read-ahead.

VFS read-ahead was designed for hard drives, and it was benchmarked with hard-drives. But the nature of raw flash devices is very different to the nature of Hard Drives Raw flash devices do not heave such a huge seek time as hard drives do, so the techniques which work for HDDs do not necessarily work well for flash.

That said, VFS read-ahead only slows UBIFS down instead of improving it, so UBIFS disables VFS read-ahead. But UBIFS has its own internal read-ahead, which we call "bulk-read". You may enable bulk-read using the "bulk_read" UBIFS mount option.

Some flashes may read faster if the data are read at one go, rather than at several read requests. For example, OneNAND can do "read-while-load" if it reads more than one NAND page. So UBIFS may benefit from reading large data chunks at one go, and this is exactly what bulk-read does.

If UBIFS notices that a file is being read sequentially (at least 3 sequential 4KiB blocks has been read), and if UBIFS sees that the further file data resides sequentially at the same eraseblock, it starts reading data ahead using large read requests, which makes it possible to read at higher rates. So UBIFS reads more than it is asked to, and it pushes the read-ahead data to the file caches, so the data become instantly available for the further user read requests.

Here is an example. Suppose the user is reading a file sequentially. We are lucky and the file is not fragmented on the media. Suppose LEB 25 contains data nodes belonging to this file, and the data nodes are logically (in terms of logical file offset) and physically (in terms of LEB/offset addresses) sequential. Suppose user requests to read data node at LEB 25 offset 0. In this case UBIFS will actually read whole LEB 25 at one go, then populate the file cache with all the read data. And when the user asks the next piece of data, it will already be in the cache.

Obviously, the bulk-read feature may slow UBIFS down in some work-loads, so you should be careful. It is also worth noting that bulk-read feature cannot help on highly fragmented file-systems. Although UBIFS does not fragment file-systems (e.g., the Garbage-Collector does not re-order data nodes), but UBIFS does not try to de-fragment them. For example, if you write a file sequentially, it won't be fragmented. But if you write more than one file at a time, they may become fragmented (well, this also depends on how write-back flushes the changes), and UBIFS won't automatically de-fragment them. However, it is possible to implement a background de-fragmentator. It is also possible to have per-inode journal head and avoid mixing data nodes belonging to different inodes in the same LEB. So there is room for improvements.

Space for superuser

UBIFS reserves some space for the superuser (root), which means that when the file-system is full for normal users, there is still little space for the super-user. File-systems like ext2 have a similar feature. mkfs.ubifs has -R option which may be used to specify the reserved space size.

By default only root may use the reserved space. But it is possible to extend the list of power users who are able to utilize the reserved space. UBIFS may record several user and group IDs at the superblock and allow them to utilize the reserved space as well. However, current mkfs.ubifs utility does not have corresponding command line options, but it should be trivial to implement them.

Note, UBIFS prints the amount of reserved space when mounts the file-system. See UBIFS messages in the system log.

Extended attributes

UBIFS supports extended attributes if the corresponding configuration option is enabled (no additional mount options are required). It supports the user, trusted, and security name-spaces. However, access control lists (ACL) support is not implemented.

Note, currently mkfs.ubifs ignores extended attributes and does not write them to the target file-system image.

Mount options

The following are UBIFS-specific mount options.

Example:

$ mount -t ubifs -o no_chk_data_crc ubi0:rootfs /mnt/ubifs

mounts UBIFS file-system to /mnt/ubifs and disables data CRC checking.

Besides, UBIFS supports the standard sync mount option which may be used to disable UBIFS write-back and write-buffer caching and make it fully synchronous. Note, UBIFS does not support "atime", so the atime mount option has no effect.

Flash space accounting issues

Traditional file systems like ext2 can easily calculate amount of free space. The calculation is usually quite precise and users are accustomed to this. However, the situation is very different in UBIFS - it cannot really report precise amount of free space which confuses users. Instead, it reports minimum amount of free space, which usually less than the real amount. Sometimes the mistake may be very high. For example, UBIFS may report (via the statfs() system call) that there is no free space, but one would still be able to write quite a lot.

To put it differently, UBIFS is often lying about the amount of free space it has. As a rule, it may fit considerably more bytes than it reports. However, it never reports more free space than it has. It reports less, and very rarely it may report the exact amount. And this is not because UBIFS authors are jerks, there are fundamental reasons for this, which are discussed below.

Effect of compression

The first factor is UBIFS on-flight compression. Users usually seem to expect that if file system reports N bytes of free space, than it is possible to create an N-byte file. And because of the compression, this is not quite true for UBIFS. Depending on how well the file data compresses, UBIFS may fit several times more than it reports.

When UBIFS calculates free space, it does not a-priori know anything about the data which is going to be written, so it cannot take into account the compression, so it always assumes the worst-case scenario when the data does not compress.

Well, this does not sound as a big issue. However, compression becomes an issue for free space reporting when compression is combined with write-back. Namely, UBIFS cannot know how well the cached dirty data would compress, and the only way to find this out is to actually compress it. See below.

Effect of write-back

Suppose there are X bytes of dirty file data in the page cache. They will be flushed to the flash media later, but they are in RAM so far. UBIFS (namely, the budgeting sub-system) has reserved X + O bytes on the flash media for this data, where O is file system overhead (e.g., the data has to be indexed, each data node has a header, etc).

The problem is that UBIFS cannot accurately calculate X and O, and it uses pessimistic worst-case calculations, so that when the cached data are flushed, they may take considerably less flash space than the reserved X + O. For example, this may lead to the following situations.

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:ubifs               49568     49568         0 100% /mnt/ubifs
$ sync
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi0:ubifs               49568     39164      7428  85% /mnt/ubifs

First time df reported zero free space, but after the sync it reported 15% free space. This is because there were a lot of cached dirty data, and UBIFS reserved all flash space for them. But once the data has reached the flash media, they took considerably less flash space.

Here are the reasons why UBIFS reserves more space than it is needed.

Thus, UBIFS reports more accurate free space value if it is synchronized.

Wastage

As it was mentioned above, UBIFS nodes do not cross LEB boundaries. Consider the following numbers:

Thus, if the vast majority of nodes on the flash were non-compressed data nodes, UBIFS would waste 1344 bytes at the ends of 126KiB LEBs. But real-life data are often compressible, so data node sizes vary, and the amount of wasted space at the ends of eraseblocks varies from 0 to 4255.

UBIFS is doing some job to put small nodes like directory entries to the ends of LEBs to lessen the amount of wasted space, but it is not ideal and UBIFS still may waste unnecessarily large chunks of flash space at the ends of eraseblocks.

When reporting free space, UBIFS does not know which kind of data are going to be written to the flash media, and in which sequence. Thus, it assumes the maximum possible wastage of 4255 bytes per LEB. This calculation is too pessimistic for most real-life situations and the average real-life wastage is considerably less than 4255 bytes per LEB. However, UBIFS reports the absolute minimum amount of free space user-space applications may count on.

The above means that the larger is LEB size, the better is UBIFS free space prediction. E.g., UBIFS is better in this respect on NANDs with 128KiB eraseblock size, comparing to NANDs with 16KiB eraseblock size.

Dirty space

Dirty space is the flash space occupied by UBIFS nodes which were invalidated because they were changed or removed. For example, if the contents of a file is re-written, than corresponding data nodes are invalidated and new data nodes are written to the flash media. The invalidated nodes comprise dirty space. There are other mechanisms how dirty space appears as well.

UBIFS cannot re-use dirty space straight away, because corresponding flash areas do not contain all 0xFF bytes. Before dirty space can be re-used, UBIFS has to garbage-collect corresponding LEBs. The idea of Garbage collector which reclaims dirty space is the same as in JFFS2. Please, refer the JFFS2 design document for more information.

Roughly, UBIFS garbage collector picks a victim LEB which has some dirty space and moves valid UBIFS nodes from the victim LEB to the LEB which was reserved for GC. This produces some amount of free space at the end of the reserved LEB. Then GC pick new victim LEB, and moves the data to the reserved LEB. When the reserved LEB is full, UBIFS picks another empty LEB (e.g., the old victim which had been made free a step ago), and continues moving nodes from the victim LEB to the new reserved LEB. The process continues until a full empty LEB is produced.

UBIFS has a notion of minimum I/O unit size, which characterizes minimum amount of data which may be written to the flash (see here for more information). Typically, UBIFS works on large-page NAND flashes and min. I/O size is 2KiB.

Consider a situation when GC picks eraseblocks with less than min. I/O unit size dirty space. When all nodes from the victim LEB have been moved to the reserved LEB, the last min. I/O unit of the reserved LEB has to be written to the flash media, which means no space would be reclaimed. The reason why the last min. I/O unit of the reserved LEB has to me written immediately is because the victim LEB cannot be erased before all the moved nodes have reached the media. Indeed, otherwise an unclean reboot would result in lost data.

Well, things are actually not that simple and UBIFS GC actually tries not to waste space, but it is not always possible and UBIFS GC is far from being ideal. Anyway, what matters is that UBIFS cannot always reclaim dirty space if the amount of it is less than min. I/O unit size.

When UBIFS reports free space to the users, it treats dirty space as available for new data, because after garbage-collection dirty space becomes free space. But we have just showed, UBIFS cannot reclaim all dirty space and turn it into free space. Worse, UBIFS does not precisely know how much dirty space it can reclaim. So it again uses pessimistic calculations.

Thus, the less dirty space the FS has, and the smaller is dirty space fragmentation, the more precise is UBIFS free space reporting. In practice this means that a file system which is close to be full has less accurate free space reporting comparing to a less full file system, because this file system presumably has more dirty space.

Note, to fix this issue, UBIFS would need to run GC in statfs(), which would turn as much dirty space as possible into free space, which would result in more precise free space reporting. However, this would make statfs() very slow. Another possibility would be to implement background GC in UBIFS (just like in JFFS2), which would lessen effect of dirty space with time.

Precise index size is not known

As you probably know, UBIFS maintain the FS index on flash. The index takes some flash space. There also UBIFS journal, which contain FS data. The FS data in the journal is not indexed, which means that on-flash index does not refer it. Instead, UBIFS keeps indexing information for the journal in RAM. When the file system is mounted, UBIFS has to scan the journal and build this part of the index in RAM. So the journal is like a small JFFS2 file system inside UBIFS.

The journal becomes indexed as the result of the commit operation. During the commit UBIFS updates the on-flash index and makes it refer information in the journal. Then UBIFS picks other LEBs for the new journal, so the journal changes is position after the commit.

UBIFS maintains precise accounting of the index size. That is, UBIFS always knows how many bytes the on-flash index takes. However, UBIFS does not know precisely how much will the index grow (or shrink) after the commit. This means, it does not know whether how much will the index size change after the journal data references will be included into the on-flash index. And UBIFS again makes pessimistic calculations here and assumes worst-case scenario.

However, after the commit operation, UBIFS knows exact index size again. The sync() system not only flushes all dirty data, but it also call makes UBIFS commit the journal. This means that file system synchronization the makes free space prediction mistake lower.

It is worth noting that this is not a fundamental thing. It is just an UBIFS implementation detail. UBIFS could calculate precise index size without actually running the commit operation, but the UBIFS authors found it difficult to implement. And the effect of the index size uncertainty should be low.

Documentation

If flash file systems is a completely new area for you, it is recommended to start from learning JFFS2, because many basic ideas are the same in UBIFS. Read the JFFS2 design document.

You may find the description of main JFFS2 issues, as well as very basic UBIFS ideas in the JFFS3 design document. Remember, the document in general is old and out-of-date. We do not use the "JFFS3" name anymore, and JFFS3 was re-named to UBIFS. The document was written when UBI did not exist and the document assumes that JFFS3 is talking directly to the MTD device, just like JFFS2. However, the JFFS2 overview, JFFS3 Requirements, and Introduction to JFFS3 chapters are still mostly valid and give a good introduction into basic UBIFS ideas like wandering tree and the journal. Although please note, that the superblock description is irrelevant for UBIFS. UBIFS is based on UBI and does not need that trick. However, the superblock location idea may be used to create new scalable UBI2 layer.

This web-page as well as the UBIFS FAQ contains a plenty of UBIFS information. And you have to study UBI as well, because UBIFS depends on the services provided by the UBI layer. See the UBI documentation and UBI FAQ sections.

Look at UBIFS presentation slides (ubifs.odp) which give another UBI/UBIFS overview. The slides were prepared in OpenOffice.org Impress 2.4, so you need OpenOffice to see them. The slides contain animation, so you have to watch them in "slide show" mode (use F5 key). And if you do not have any possibility to get OpenOffice, here is a pdf version, but it is very ugly because it does not store the animation and draws all animation steps at once.

There is an UBIFS white-paper document available as well. However, it might be rather difficult for newbies, so we recommend to start with the JFFS3 design document. The UBIFS white-paper gives a complete UBIFS design picture and describes the UBIFS internals. The white-paper does not contain some details which you may find at this web-page or in the UBIFS FAQ, and vice-versa.

And finally, there is UBIFS source code. The code has a great deal of comments, so we recommend to look there if you need all the details. And of course, you are welcome to ask questions at the UBIFS mailing list.

How to send an UBIFS bugreport?

When sending a bug report, please:

  1. make sure you use up-to-date UBIFS; pull the latest UBIFS patches if needed, see here; no one is interested in digging already fixed problems;
  2. make sure you have compiled the kernel symbols in (CONFIG_KALLSYMS_ALL=y in .config);
  3. mark the Enable debugging check-box in the UBIFS kernel configuration menu (CONFIG_UBIFS_FS_DEBUG=y in .config); please, do not enable other UBIFS debugging sub-options like extra debugging messages unless you know what you are doing; note, you should enable UBIFS debugging, not UBI debugging - distinguish between UBI and UBIFS please, they are different things;
  4. include all the messages UBIFS prints, not only those you see at the console, but also those you see when running dmesg; or before running your UBIFS test, which reproduces the error, just invoke dmesg -n8 command to make all kernel messages to go to the console; another possibility is to boot the kernel with ignore_loglevel option, in which makes the kernel print all messages to the console unconditionally;
  5. describe your flash device, attach the "cat /proc/mtd" output;
  6. specify which kernel version you are using;
  7. describe how the problem cab be reproduced;

The bugreport should be sent to the MTD mailing list. Please, do not send private e-mails to UBIFS authors, always CC the mailing list!

Note, sometimes UBIFS bugs may appear to be UBI bugs. If you suspect there are UBI problems, please, also enable UBI debugging. Please, refer the UBI debugging section for more information.

Raw flash vs. FTL devices

FTL stands for "Flash Translation Layer" and it is software which emulates a block device on top of flash hardware. At early days FTL ran on the host computer. For example, old PCMCIA flash devices were essentially raw flash devices, and the PCMCIA standard defined the media storage format for them. So the host computer had to run the FTL software driver which implemented PCMCIA FTL. However, nowadays FTL is usually firmware, and it is run by the controller which is built into the storage device. For example, if you look inside an USB flash drive, you'll find there a NAND chip (or several of them), and a micro-controller, which runs FTL firmware. Some USB flash drives are known to have quite powerful ARM processors inside. Similarly, MMC, eMMC, SD, SSD, and other FTL devices have a built-in controller which runs FTL firmware.

All FTL devices have an interface which provides block I/O access. Well, the interfaces are different and they are defined by different specifications, e.g., MMC, eMMC, SD, USB mass storage, ATA, and so on. But all of them provide block-based access to the device. By block-based access we mean that whole device is represented as an linear array of (usually 512-byte) blocks. Each block may be read or written.

Linux has an abstraction of a block device. For example hard drives are block devices. Linux has many file systems and the block I/O subsystem, which includes elevators and so on which have been created to work with block devices (historically - hard drives). So the idea is that the same software may be used with FTL devices. For example, you may use FAT file system on your MMC card, or ext3 file system or your SSD.

Although most flashes on the commodity hardware have FTL, there are systems which have bare flashes and do not use FTL. Those are mostly various handheld devices and embedded systems. Raw flash devices are very different to block devices. They have different work model, they have tighter constraints and more issues than block devices. In case of FTL devices these constraints and issues are hidden, but in case of raw flash the software has to deal with them. Please, refer this table for the some more details about the difference between block devices and raw flashes.

UBIFS file system has been designed for raw flash. It doesn't work with block devices and it assumes the raw flash device model. In other words, it assumes the device has eraseblocks, which may be written to, read from, or erased. UBIFS takes care of writing all data out-of-place, doing garbage-collection and so on. UBIFS utilizes UBI, which is doing stuff like wear-leveling and bad eraseblock handling. All these things are not normally needed for block devices.

Very often people ask questions like "why would one need to use raw flash and why not just use eMMC, or something like this?". Well, there is no simple answer, and the following is what UBIFS developers think. Please, take into account the date of this writing (3 May 2009). The answer is given in form of a list of non-structured items, and the reader should structure it in a way which is appropriate for his system. And because mass storage systems mostly use NAND flash (modern FTL devices also have NAND flash arrays inside), we talk specifically about NAND flashes. Also, we'd like to emphasize that we do not give general recommendations and everything depends on system requirements.

So it is indeed difficult to give an answer. Just think about cons and pros, take into account your system requirements and decide. Nonetheless, raw flashes are used, mostly in the embedded world, and this is why UBIFS has been developed.

Last updated: 26 Oct 2008, dedekind Valid XHTML 1.0! Valid CSS!