Mythic Beasts

newubd - UML I/O performance improvements

User Mode Linux disk I/O is handled by the UBD (User-mode Block Device) driver. The stock UBD implementation has some significant performance issues, especially when dealing with larger I/O requests. newubd is a re-implementation of UBD that avoids these issues.

Background

The UBD driver is implemented as a separate thread dedicated to I/O. This accepts requests via a pipe from from the UML kernel process. Under Linux, a block device I/O request is made up of one or more buffers each made up of one or more segments. Unfortunately, the standard UBD driver handles I/O requests by sending each segment individually to the I/O thread, waiting for the I/O to complete, and then sending the next one. With each segment being a maximum of 4K in size, this has a very big impact on I/O performance, as the underlying hardware is often not able to treat large I/O operations as a series of sequential read/writes.

This approach has a significant impact on I/O performance on a lightly loaded machine, but is disastrous on a heavily loaded UML server where different guests are contending for I/O. Modern hard drives are characterised by high data transfer rates (~50MB/s) but low seek speeds (~10ms per seek). Individual large I/O operations will typically result in only a small number of seeks, as much of the data will be written sequentially on the disk. The current UBD implementation gives the host's I/O driver the opportunity to go and do something else after every 4K of data is transferred. Chances are, that something else will be elsewhere on the disk, with a result that in a worst case scenario, two unnecessary seeks are inserted every 4K. Based on the disk speeds above, such a worst case could see two 1MB I/O operations grow from 60ms to over 5s.

The solution

The solution is simple in principle, if not entirely trivial to implement. Each UBD I/O request should be passed to the host kernel as a single scatter or gather operation using writev()/readv(). This is what the newubd driver does.

We have developed a simple performance benchmark utility that will issue random seeks and randomly-sized read, write or synchronous write operations. This gives a measure of the effective seek time and transfer rate of a block device. The graph below shows the results of this benchmark, comparing write operations on the host (Linux 2.6.13) with UML (2.6.12) using stock UBD and newubd. It is clear that the newubd driver performance is far closer to the host's raw performance than the stock UBD driver.

Host vs UML IO results

The patch

A patch against 2.6.14.3 can be downloaded here: linux-2.6.14.3-newubd.patch. This has patch has been in production use on Mythic Beasts Virtual Dedicated Servers for some months with no problems. Please note that this patch does not support COW (Copy On Write).

The next step

The limitations of the current UBD implementation only became apparent to us due to work we are doing on creating a system for balancing I/O fairly between UML guests, in particular, to neutralise the impact of a UML guest running out of memory and "thrashing". This system is currently being tested and will be rolled out on all Mythic Beasts Virtual Dedicated Servers shortly.

Copyright © 2000-2014 Mythic Beasts Ltd. All Rights Reserved.