bmaptool copy

This section describes the "copy" subcommand of bmaptool and it is structured like this:

  1. usage - discusses the "copy" subcommand usage in general
  2. optimizations - describes the optimizations the "copy" subcommand implements
  3. Tizen IVI example - provides an example of "bmaptool copy" usage in Tizen IVI

Usage

The "bmaptool copy" command copies a file or an image to a different file or a block device using bmap. Let's focus on the image flashing usage scenario. The other scenarios are very similar.

The following command flashes the "image.raw" raw image to the "/dev/sdg" block device (such as a USB stick), using the "image.bmap" block map file:

 $ bmaptool copy --bmap image.bmap image.raw /dev/sdg

Most often, the raw images are distributed in a compressed form (such as compressed with "bzip2"), in which case you can flash them with the same command:

 $ bmaptool copy --bmap image.bmap image.bz2 /dev/sdg

where "image.bz2" is the compressed raw image.

If the bmap file is stored at the same directory as the image, then bmaptool will automatically discover it, so you do not have to use the "--bmap" option:

 $ bmaptool copy image.bz2 /dev/sdg

If the image is stored at the remote FTP/HTTP/HTTPS/SSH server, you do not have to download it - bmaptool accepts both local paths and URLs:

 $ bmaptool copy http://my.server/blah/image.bz2 /dev/sdg

Of course, the above command assumes that the bmap file for the image is stored at the same place as "image.bz2".

Basically speaking, bemaptool works as follows.

  1. If the --bmap option was not used, search for the bmap file at the same place as the image file
  2. Read and parse the bmap file
  3. If the OpenPGP signature is present in the bmap file, or a detached OpenPGP signature was found, verify it
  4. Verify the SHA256 checksum of the bmap file
  5. Read the image ("image.bz2") chunk-by-chunk, de-compressing the chunks on-the-fly
  6. Verify the SHA256 checksum of all the mapped block ranges
  7. Write the range data to the destination file (/dev/sdg)

Note: bmaptool does not decompress the entire image to a temporary file. It uses stream decompression, instead.

Supported compression formats

At the moment, the compression type is detected by the file extension and the following compression types are supported: ".bz2", ".gz", ".gzip", ".xz", ".tar.gz" or ".tgz", ".tar.bz2" and ".tar.xz". If the image file has none of these extensions, it is assumed to be uncompressed.

Exclusive block device access

The "/dev/sdg" block device is opened in exclusive mode, which means that if, by mistake, you specify a block device which is being mounted or otherwise used, then bmaptool will refuse it. In other words, bmaptool accepts only those block devices that are not opened by anyone else (see the semantics of the "O_EXCL" flag of the "open()" syscall). For example, bmaptool will refuse your root partition. This is just a safety measure to make sure that bmaptool users do not destroy their data by mistake.

Proxy settings

Just like most of the standard Linux software, bmaptool uses the "http_proxy"/"https_proxy"/"ftp_proxy"/"no_proxy" environment variables for its proxy settings.

Copying without bmap

If the "--bmap" option is not specified and bmaptool fails to auto-discover the bmap file, it exits with an error. Please use the "--nobmap" option, if you want to copy without a bmap file. Of course, this will be slower than copying with bmap.

SHA-256 verification

The bmap file carries SHA256 checksums for the bmap file itself and for all the mapped areas of the image file. Bmaptool always verifies the checksums when copying the image and exits with an error message in case of a checksum mismatch. This guarantees data integrity. One may disable checksums verification with the "--no-verify" option.

OpenPGP signature verification

The bmap file can be signed with OpenPGP (AKA gpg), in which case bmaptool verifies the signature when copying the data. If the signature is bad, bmaptool exits with an error message. One may disable signature verification with the "--no-sig-verify" option.

OpenPGP signatures guarantee bmap file integrity and authoring. And since the bmap file contains SHA256 checksums of the mapped image regions, the signature also guarantees integrity authoring of the mapped image areas. Unmapped image areas are dropped when flashing, so their contents should not matter, but in an intact image the contents of the unmapped areas has to be all zero bytes. However, the current bmaptool implementation does not verify unmapped areas because this does not seem to add any value.

Bmaptool supports detached OpenPGP signatures, as well as "in-band" "clearsign" signatures (see the "--clearsign" gpg option). The latter means that the signature is stored in the bmap file in "clearsign" format.

Bmaptool automatically checks the bmap file for the clearsign signature and verifies the signature if it is present. If there is no clearsign signature, bmaptool tries to automatically discover the detached signature by looking for a file with basename equivalent to the bmap file name, but with a ".sig" or ".asc" extension. The detached signature is searched for at the same path as the bmap file path (be that a local path or an URL).

Optimizations

Bmaptool flashes images a lot more quickly than tools like "dd", mostly because of the bmap, which contains the list of mapped block ranges. However, there are additional optimizations which speed up flashing to block devices.

The main optimization is that bmaptool switches the I/O scheduler of the destination block device to "noop", because the no-op scheduler is fastest when it comes to writing sequentially, directly to the block device. When "bmaptool copy" finishes, the original I/O scheduler is restored.

The other optimization bmaptool has is using 2 separate threads for reading and writing:

  • the first thread reads the presumably compressed chunks of image data, decompresses them, and verifies the SHA256 checksums
  • the second thread writes the mapped data

This speeds up flashing substantially when the image is compressed.

Tizen IVI example

Tizen IVI uses raw images and publishes them in a bzip2-compressed form, along with the bmap file at download.tizen.org.

Let's take the latest Tizen IVI milestone release image as an example. To flash it to a USB stick, you can execute the following command:

 $ bmaptool copy http://download.tizen.org/releases/milestone/tizen/ivi/latest/images/ivi-release-efi-i586/tizen_20130729.2_ivi-release-efi-i586-sdb.raw.bz2 /dev/usb_stick

to flash directly from download.tizen.org. The alternative is to first download the image and the bmap file, and then flash them to the USB stick. Here are the files to download:

  1. tizen_20130729.2_ivi-release-efi-i586-sdb.raw.bz2 (185MiB) - the compressed Tizen IVI raw image
  2. tizen_20130729.2_ivi-release-efi-i586-sdb.bmap (6.3KiB)- the block map of the image

Make sure they are saved in the same directory. Flash the image to the USB stick using this command:

 $ bmaptool copy <your_path>/tizen_20130729.2_ivi-release-efi-i586-sdb.raw.bz2 /dev/usb_stick