blinohod

- Скажите, Сеймур, я видел ваш Форд на парковке - почему вы купили именно эту машину? Я вот сейчас думаю, какую машину мне покупать, и никак не выберу...
- Все просто. Я пошел в ближайший автосалон, подошел к ближайшему продавцу, ткнул пальцем в ближайший автомобиль, и сказал: я ее покупаю.
- Но...
- Это самый оптимальный способ, коллега. Он позволяет мне посвящать максимум времени разработке суперкомпьютеров. Рекомендую.

( Еще прекрасного про мистера Крэя )

OpenVZ have just introduced kernel and tools support for container-in-a-file technology, also known as ploop. This post tries to summarize why ploop is needed, and why is it a superior technology to what we had before.

Before ploop: simfs and vzquota

First of all, a few facts about the pre-ploop era technologies and their limitations.

As you are probably aware, a container file system was just a directory on the host, which a new container was chroot()-ed into. Although it seems like a good and natural idea, there are a number of limitations.

Since containers are living on one same file system, they all share common properties of that file system (it's type, block size, and other options). That means we can not configure the above properties on a per-container basis.

One such property that deserves a special item in this list is file system journal. While journal is a good thing to have, because it helps to maintain file system integrity and improve reboot times (by eliminating fsck in many cases), it is also a bottleneck for containers. If one container will fill up in-memory journal (with lots of small operations leading to file metadata updates, e.g. file truncates), all the other containers I/O will block waiting for the journal to be written to disk. In some extreme cases we saw up to 15 seconds of such blockage.

Since many containers share the same file system with limited space, in order to limit containers disk space we had to develop per-directory disk quotas (i.e. vzquota).

Again, since many containers share the same file system, and the number of inodes on a file system is limited [for most file systems], vzquota should also be able to limit inodes on a per container (per directory) basis.

In order for in-container (aka second-level) disk quota (i.e. standard per-user and per-group UNIX dist quota) to work, we had to provide a dummy file system called simfs. Its sole purpose is to have a superblock which is needed for disk quota to work.

When doing a live migration without some sort of shared storage (like NAS or SAN), we sync the files to a destination system using rsync, which does the exact copy of all files, except that their i-node numbers on disk will change. If there are some apps that rely on files' i-node numbers being constant (which is normally the case), those apps are not surviving the migration.

Finally, a container backup or snapshot is harder to do because there is a lot of small files that need to be copied.

Introducing ploop

In order to address the above problems and ultimately make a world a better place, we decided to implement a container-in-a-file technology, not different from what various VM products are using, but working as effectively as all the other container bits and pieces in OpenVZ.

The main idea of ploop is to have an image file, use it as a block device, and create and use a file system on that device. Some readers will recognize that this is exactly what Linux loop device does! Right, the only thing is loop device is very inefficient (say, using it leads to double caching of data in memory) and its functionality is very limited.

Ploop implementation in the kernel have a modular and layered design.

The top layer is the main ploop module, which provides a virtual block device to be used for CT file system.

The middle layer is the image format module, which does translation of block device block numbers into image file block numbers. A simple format module which is called "raw" is doing trivial 1:1 translation, same as existing loop device.

More sophisticated format module is keeping the translation table and is able to dynamically grow and shrink the image file. That means, if you create a container with 2GB of disk space, the image file size will not be 2GB, but less -- the size of the actual data stored in the container. It is also possible to support other image formats by writing other ploop format modules, such as the one for QCOW2 (used by QEMU and KVM).

The bottom layer is the I/O module. Currently modules for direct I/O on an ext4 device, and for NFS are available. There are plans to also have a generic VFS module, which will be able to store images on any decent file system, but that needs an efficient direct I/O implementation in the VFS layer which is still being worked on.

Ploop benefits

In a nutshell:

• File system journal is not bottleneck anymore
• Large-size image files I/O instead of lots of small-size files I/O on management operations
• Disk space quota can be implemented based on virtual device sizes; no need for per-directory quotas
• Number of inodes doesn't have to be limited because this is not a shared resource anymore (each CT has its own file system)
• Live backup is easy and consistent
• Live migration is reliable and efficient
• Different containers may use file systems of different types and properties

In addition:

• Efficient container creation
• [Potential] support for QCOW2 and other image formats
• Support for different storage types

Ехал грека через реку,
Пролагал маршрут по звёздам,
Думал о природе рака,
Тряс укушенной рукой,

Размышлял, что к человеку
Ум приходит слишком поздно.
Выла вдалеке собака.
Ночь струилась над рекой.

Грека плыл почти наощупь.
В темноте шептались раки
И держали против греки
Круговую оборону.

Мир моложе был и проще.
В предвкушеньи знатной драки
Ехал грека через реку,
Заплатив обол Харону.

IBM HDD SAS 3,5" 300GB 15k стоит 5448.33 грн (курс $1 - 8,1 грн).
Но, на самом деле, это Seagate ST3300655SS, с наклейкой IBM.
Seagate ST3300655SS без наклейки IBM стоит - 2192 грн.

16 Февраля 2012 года, в 19:30 в помещении GL Club, Киев состоится техническая встреча Kiev.pm - группы пользователей Perl

На сегодняшний день зарегистрированы доклады:

* Gearman Managers: управление рабочими процессами - Олег Алистратов, Яндекс
* Nginx. По следам мастер-класса Игоря Сысоева - Денис Жданов, Cogniance
* Тестирование кода, использующего ORM - Виктор Турский, WebbyLab
* Redis и Protocol::Redis - Сергей Засенко

Посетителям:
Мы будем рады посетителям с любым уровнем знания Perl.
Зарегистрируйтесь до 14 Февраля (включительно). Регистрация обязательна.
Пожалуйста, не забудьте взять с собой документ, удостоверяющий личность.
Клуб находится по адресу ул. Николая Гринченка, 2/1, БЦ "Протасов" [карта]

Докладчикам: напишите тему доклада и приблизительную продолжительность на support@kiev.pm.org
Последний срок регистрации докладов - 14 Февраля

Будет здорово, если вы поделитесь этой информацией с друзьями и коллегами, которые могут быть заинтересованы в посещении.