From dedekind1 at gmail.com  Tue Aug  6 04:30:31 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue,  6 Aug 2013 11:30:31 +0300
Subject: [bmap-tools] [PATCH 2/5] packaging: use opensuse_bs macro
In-Reply-To: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
References: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1375777834-3374-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

The <CI_CNT>.<B_CNT> trick is OBS-specific, so wrap it with the 'opensuse_bs'
macro.

Change-Id: I817757ffdacddc32077b2876185220e880655ad7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 packaging/bmap-tools.spec | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index d0a667d..c75c14f 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -5,7 +5,13 @@
 Name:       bmap-tools
 Summary:    Bmap Tools
 Version:    2.4
-Release:    %{rc_str}.<CI_CNT>.<B_CNT>
+
+%if 0%{?opensuse_bs}
+Release: %{rc_str}.<CI_CNT>.<B_CNT>
+%else
+Release: %{rc_str}.0.0
+%endif
+
 Group:      Development/Tools/Other
 License:    GPL-2.0
 BuildArch:  noarch
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug  6 04:30:32 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue,  6 Aug 2013 11:30:32 +0300
Subject: [bmap-tools] [PATCH 3/5] packaging: improve the summary text in RPM
	packaging
In-Reply-To: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
References: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1375777834-3374-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I71b36a801c4a4898bf73c8226f7a7495a2b8407c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 packaging/bmap-tools.spec | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index c75c14f..be6e63c 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -3,7 +3,7 @@
 %define rc_str %{?rc_num:0.rc%{rc_num}}%{!?rc_num:1}
 
 Name:       bmap-tools
-Summary:    Bmap Tools
+Summary:    Tools to generate block map (AKA bmap) and flash images using bmap
 Version:    2.4
 
 %if 0%{?opensuse_bs}
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug  6 04:30:30 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue,  6 Aug 2013 11:30:30 +0300
Subject: [bmap-tools] [PATCH 1/5] README: add more information for
	opensource community
Message-ID: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Plus some minor re-structuring.

Change-Id: I66c60057949ec6bd0ccdcb29c7fabf691a36bb43
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 docs/README | 47 ++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 11 deletions(-)

diff --git a/docs/README b/docs/README
index 401112f..b8aabaa 100644
--- a/docs/README
+++ b/docs/README
@@ -1,20 +1,43 @@
+Summary
+~~~~~~~
+
 The bmap-tools project implements bmap-related tools and API modules. The
 entire project is written in python, and requires python 2.7+.
 
+Currently the main user of this project is Tizen IVI, but the project is
+generic and can be used everywhere, when dealing with raw images.
+
+The project author and maintainer is Artem Bityutskiy <dedekind1 at gmail.com>.
+Please, feel free to contact me if you have questions.
+
 The project is documented here:
 https://source.tizen.org/documentation/reference/bmaptool
 
-Currently the main user of this project is Tizen IVI, but the project is
-generic and can be used everywhere, when dealing with raw images.
+The project mailing list is (no need to subscribe to post there):
+bmap-tools at lists.infradead.org
+
+Mailing list archives:
+http://lists.infradead.org/pipermail/bmap-tools/
+
+Subscribe here:
+http://lists.infradead.org/mailman/listinfo/bmap-tools
+
+The project git tree is here:
+git://git.infradead.org/users/dedekind/bmap-tools.git
+
+The git tree web view is here:
+http://git.infradead.org/users/dedekind/bmap-tools.git
+
+Signed release tarballs are available here:
+ftp://ftp.infradead.org/pub/bmap-tools/
 
-The project author and maintainer is 'Artem.Bityutskiy at linux.intel.com'.
-Please, feel free to contact me if you have questions about the project or
-patches. Suggestions and comments about the code are welcome as well.
+Packages for various distributions are available here:
+* The latest release: http://download.tizen.org/tools/latest-release/
+* The latest pre-release: http://download.tizen.org/tools/pre-release/
+* Older releases: http://download.tizen.org/tools/archive
 
-The debianizations and rpmization was done by
-'Ed Bartosh <eduard.bartosh at intel.com>', and he is the only person I know of
-who understands how that works :-). The bmap-tools packages for
-Debian/Ubuntu/Fedora/OpenSuse are built by the "OBS" service.
+Please, contribute by sending patches to the mailing list, feel free to CC
+me: Artem Bityutskiy <dedekind1 at gmail.com>
 
 The project structure
 ~~~~~~~~~~~~~~~~~~~~~
@@ -88,9 +111,11 @@ release, though.
 Credits
 ~~~~~~~
 
-* Ed Bartosh <eduard.bartosh at intel.com> - for helping me with learning python
+* Ed Bartosh <eduard.bartosh at intel.com> for helping me with learning python
   (this is my first python project) and working with the Tizen IVI
-  infrastructure.
+  infrastructure. Ed also implemented the packaging.
 * Alexander Kanevskiy <alexander.kanevskiy at intel.com> and
   Kevin Wang <kevin.a.wang at intel.com> for helping with integrating this stuff
   to the Tizen IVI infrastructure.
+* Simon McVittie <simon.mcvittie at collabora.co.uk> for improving Debian
+  packaging and fixing bmaptool.
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug  6 04:30:34 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue,  6 Aug 2013 11:30:34 +0300
Subject: [bmap-tools] [PATCH 5/5] Release version 2.5
In-Reply-To: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
References: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1375777834-3374-5-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ibeed08d6913893452bec9587297f2417808a6057
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool                     |  2 +-
 debian/changelog             |  8 ++++++++
 docs/RELEASE_NOTES           | 20 ++++++++++++++++++++
 packaging/bmap-tools.changes |  5 +++++
 packaging/bmap-tools.dsc     |  2 +-
 packaging/bmap-tools.spec    |  2 +-
 setup.py                     |  2 +-
 7 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/bmaptool b/bmaptool
index 2b1bda0..10a35e9 100755
--- a/bmaptool
+++ b/bmaptool
@@ -46,7 +46,7 @@ also contribute to the mapped blocks and are also copied.
 #   * Too few public methods (R0903)
 # pylint: disable=R0903
 
-VERSION = "2.4"
+VERSION = "2.5"
 
 import argparse
 import sys
diff --git a/debian/changelog b/debian/changelog
index 73fd992..a1723ab 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+bmap-tools (2.5) unstable; urgency=low
+
+  * Do not fail when lacking permisssions for accessing block device's sysfs
+    files.
+  * Improve debian packaging.
+
+ -- Artem Bityutskiy <artem.bityutskiy at linux.intel.com>  Mon, 05 Aug 2013 10:05:09 +0300
+
 bmap-tools (2.4) unstable; urgency=low
 
   * Add support for ssh:// URLs.
diff --git a/docs/RELEASE_NOTES b/docs/RELEASE_NOTES
index 6c6f7e1..e2bf833 100644
--- a/docs/RELEASE_NOTES
+++ b/docs/RELEASE_NOTES
@@ -1,3 +1,23 @@
+Bug-fix release 2.5
+~~~~~~~~~~~~~~~~~~~
+
+1. bmaptool (or more precisely, the BmapCopy class) has an optimization where
+   we switch to the "noop" I/O scheduler when writing directly to block
+   devices. We also lessen the allowed amount of dirty data for this block
+   device in order to create less memory pressure on the system. These tweaks
+   are done by touching the corresponding sysfs files of the block device. The
+   old bmaptool behavior was that it failed when it could not modify these
+   files. However, there are systems where users can write to some block
+   devices (USB sticks, for example), but they do not have permissions to
+   change the sysfs files, and bmaptool did not work for normal users on such
+   systems. In version 2.5 we change the behavior and do not fail anymore if we
+   do not have enough permissions for changing sysfs files, simply because this
+   is an optimization, although a quite important one. However, we do print a
+   warning message.
+
+2. Many improvements and fixes in the Debian packaging, which should make it
+   simpler for distributions to package bmap-tools.
+
 Bug-fix release 2.4
 ~~~~~~~~~~~~~~~~~~~
 
diff --git a/packaging/bmap-tools.changes b/packaging/bmap-tools.changes
index 893e776..00d6ada 100644
--- a/packaging/bmap-tools.changes
+++ b/packaging/bmap-tools.changes
@@ -1,3 +1,8 @@
+Mon Aug  5 07:05:59 UTC 2013 - Artem Bityutskiy <artem.bityutskiy at linux.intel.com> 2.5-1
+- Do not fail when lacking permisssions for accessing block device's sysfs
+  files.
+- Improve debian packaging.
+
 Wed Jun  5 15:16:42 UTC 2013 - Artem Bityutskiy <artem.bityutskiy at linux.intel.com> 2.4-1
 - Add ssh:// URLs support.
 
diff --git a/packaging/bmap-tools.dsc b/packaging/bmap-tools.dsc
index 35097e3..09a925c 100644
--- a/packaging/bmap-tools.dsc
+++ b/packaging/bmap-tools.dsc
@@ -2,7 +2,7 @@ Format: 1.0
 Source: bmap-tools
 Binary: bmap-tools
 Architecture: all
-Version: 2.4-1
+Version: 2.5-1
 Maintainer: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
 Standards-Version: 3.8.4
 Build-Depends: debhelper (>= 7), python-all, python-distribute
diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index be6e63c..213a5bb 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -4,7 +4,7 @@
 
 Name:       bmap-tools
 Summary:    Tools to generate block map (AKA bmap) and flash images using bmap
-Version:    2.4
+Version:    2.5
 
 %if 0%{?opensuse_bs}
 Release: %{rc_str}.<CI_CNT>.<B_CNT>
diff --git a/setup.py b/setup.py
index ca61d37..72fbd36 100644
--- a/setup.py
+++ b/setup.py
@@ -7,7 +7,7 @@ setup(
     description="Bmap tools",
     author="Artem Bityutskiy",
     author_email="artem.bityutskiy at linux.intel.com",
-    version="2.4",
+    version="2.5",
     scripts=['bmaptool'],
     packages=find_packages(exclude=["test*"]),
     license='GPLv2',
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug  6 04:30:33 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue,  6 Aug 2013 11:30:33 +0300
Subject: [bmap-tools] [PATCH 4/5] make_a_release.sh: a script for cutting
	releases
In-Reply-To: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
References: <1375777834-3374-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1375777834-3374-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Not complete, but a good start anyway. Intended to be used by me.

Change-Id: I64b7ef72ac8fe729f234f5d74847dc85225ccee0
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 make_a_release.sh | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)
 create mode 100755 make_a_release.sh

diff --git a/make_a_release.sh b/make_a_release.sh
new file mode 100755
index 0000000..5426f99
--- /dev/null
+++ b/make_a_release.sh
@@ -0,0 +1,118 @@
+#!/bin/sh -euf
+#
+# Copyright (c) 2012-2013 Intel, Inc.
+# License: GPLv2
+# Author: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
+# This script automates the process of releasing the bmap-tools project. The
+# idea is that it should be enough to run this script with few parameters and
+# the release is ready.
+#
+# This script is supposed to be executed in the root of the bmap-tools
+# project's source code tree. The checked-out branch has to be 'devel'.
+#
+# TODO:
+#   * support -rc releases;
+#   * update the version field in all places, the rpm/deb changelog and commit
+#     that.
+
+fatal() {
+        printf "Error: %s\n" "$1" >&2
+        exit 1
+}
+
+usage() {
+        cat <<EOF
+Usage: ${0##*/} <new_ver> <outdir>
+
+<new_ver>  - new bmap-tools version to make in X.Y format
+EOF
+        exit 0
+}
+
+[ $# -eq 0 ] && usage
+[ $# -eq 1 ] || fatal "insufficient or too many argumetns"
+
+new_ver="$1"; shift
+
+# Validate the new version
+printf "%s" "$new_ver" | egrep -q -x '[[:digit:]]+\.[[:digit:]]+' ||
+        fatal "please, provide new version in X.Y format"
+
+# Make sure the 'devel' branch is currently checked out
+current_branch="$(git branch | sed -n -e '/^*/ s/^* //p')"
+if [ "$current_branch" != "devel" ]; then
+	fatal "current branch is '$current_branch' but must be 'devel'"
+fi
+
+# Make sure the git index is up-to-date
+[ -z "$(git status --porcelain)" ] || fatal "git index is not up-to-date"
+
+outdir="."
+tag_name="v$new_ver"
+release_name="bmap-tools-$new_ver"
+
+# Create new signed tag
+echo "Signing tag $tag_name"
+git tag -m "$release_name" -s "$tag_name"
+
+# Prepare a signed tarball
+git archive --format=tar --prefix="$release_name/" "$tag_name" | \
+        gzip > "$outdir/$release_name.tgz"
+echo "Signing the tarball"
+gpg -o "$outdir/$release_name.tgz.asc" --detach-sign -a "$outdir/$release_name.tgz"
+
+# Get the name of the release branch corresponding to this version
+release_branch="release-$(printf "%s" "$new_ver" | sed -e 's/\(.*\)\..*/\1.0/')"
+
+cat <<EOF
+Make sure you have had updated the docs/RELEASE_NOTES file!
+Make sure you have had updated the version number and rpm/deb changelogs
+
+To finish the release:
+  1. push the $tag_name tag out
+  2. copy the tarball to ftp.infradead.org
+  3. push the changes to the $release_branch branch
+  4. point the master branch to the updated $release_branch branch
+  5. push the master, $release_branch, and devel branches out
+  4. announce the new release in the public mailing list
+
+The commands would be:
+
+#1
+git push origin $tag_name
+git push public $tag_name
+#2
+scp "$outdir/$release_name.tgz" "$outdir/$release_name.tgz.asc" casper.infradead.org:/var/ftp/pub/bmap-tools/
+#3
+git branch -f $release_branch devel
+#4
+git branch -f master $release_branch
+#5
+git push origin master:master
+git push origin $release_branch:$release_branch
+git push origin devel:devel
+git push public master:master
+git push public $release_branch:$release_branch
+git push public devel:devel
+#6
+mutt -H /proc/self/fd/0 -x <<END_OF_EMAIL
+From: Artem Bityutskiy <dedekind1 at gmail.com>
+To: Artem Bityutskiy <artem.bityutskiy at intel.com>
+Subject: Announcement: $release_name is out!
+
+Bmap-tools version $new_ver is out!
+
+Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/devel:/docs/RELEASE_NOTES
+Tarball: ftp://ftp.infradead.org/pub/bmap-tools/
+END_OF_EMAIL
+EOF
-- 
1.8.1.4



From dedekind1 at gmail.com  Sat Aug 10 08:09:41 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Sat, 10 Aug 2013 15:09:41 +0300
Subject: [bmap-tools] Announcement: bmap-tools-2.5 is out!
Message-ID: <1376136581-9256-1-git-send-email-dedekind1@gmail.com>

Bmap-tools version 2.5 is out!

Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/release-2.0:/docs/RELEASE_NOTES
Tarball: ftp://ftp.infradead.org/pub/bmap-tools/

Packages for various distributions are available here:
http://download.tizen.org/tools/pre-release/

At some later point they will be propagated to here:
http://download.tizen.org/tools/latest-release/

--
Artem Bityutskiy


From dedekind1 at gmail.com  Tue Aug 13 03:29:41 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue, 13 Aug 2013 10:29:41 +0300
Subject: [bmap-tools] [PATCH 3/4] packaging: add pyliblzma for Fedora
	dependencies
In-Reply-To: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
References: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376378982-26716-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I3fb1515c0ada923bc27a470979077f9c4e450f86
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 packaging/bmap-tools.spec | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index 40c58e4..94e851e 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -28,6 +28,8 @@ Requires: python-xml
 # In Fedora the xml.etree module is provided by the python-libs package
 %if 0%{?fedora_version}
 Requires: python-libs
+# Fedora has the pyliblzma which we use for decompressing .xz files
+Requires: pyliblzma
 %endif
 
 # Centos6 uses python 2.6, which does not have the argparse module. However,
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug 13 03:29:39 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue, 13 Aug 2013 10:29:39 +0300
Subject: [bmap-tools] [PATCH 1/4] make_a_release.sh: use git send-email
Message-ID: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Use git send-email instead of mutt, since the mutt command we had does not
really work, and I am unable to find out how to make it work.

Additinally, inform about where to find packages for various distributions.

Change-Id: Ia6408683a4794377f0ebf9f6b5cd1dc1f8559ca3
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 make_a_release.sh | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/make_a_release.sh b/make_a_release.sh
index 393d19d..d1e60af 100755
--- a/make_a_release.sh
+++ b/make_a_release.sh
@@ -104,14 +104,21 @@ git push origin $release_branch:$release_branch
 git push public master:master
 git push public $release_branch:$release_branch
 #5
-mutt -H /proc/self/fd/0 -x <<END_OF_EMAIL
-From: Artem Bityutskiy <dedekind1 at gmail.com>
-To: bmap-tools at lists.infradead.org
+git send-email --suppress-cc=all --from "Artem Bityutskiy <dedekind1 at gmail.com>" --to bmap-tools at lists.infradead.org /proc/self/fd/0 <<END_OF_EMAIL
 Subject: Announcement: $release_name is out!
 
 Bmap-tools version $new_ver is out!
 
 Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/$release_branch:/docs/RELEASE_NOTES
 Tarball: ftp://ftp.infradead.org/pub/bmap-tools/
+
+Packages for various distributions are available here:
+http://download.tizen.org/tools/pre-release/
+
+At some later point they will be propagated to here:
+http://download.tizen.org/tools/latest-release/
+
+--
+Artem Bityutskiy
 END_OF_EMAIL
 EOF
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug 13 03:29:40 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue, 13 Aug 2013 10:29:40 +0300
Subject: [bmap-tools] [PATCH 2/4] packaging: remove tabs and extra
	white-spaces
In-Reply-To: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
References: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376378982-26716-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Put one white-space after every keyword and do not try to align things, and
also do not use tabs. I think I saw this recommendation is the Fedore guide or
something, but not 100% sure. But at least this makes it easier to change the
spec file.

Change-Id: I6cee1b700287c9287637184978906af53862a04c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 packaging/bmap-tools.spec | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index 213a5bb..40c58e4 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -2,9 +2,9 @@
 # like '1.0-0.rc7' for release candidate rc7 and '1.0-1' for the '1.0' release.
 %define rc_str %{?rc_num:0.rc%{rc_num}}%{!?rc_num:1}
 
-Name:       bmap-tools
-Summary:    Tools to generate block map (AKA bmap) and flash images using bmap
-Version:    2.5
+Name: bmap-tools
+Summary: Tools to generate block map (AKA bmap) and flash images using bmap
+Version: 2.5
 
 %if 0%{?opensuse_bs}
 Release: %{rc_str}.<CI_CNT>.<B_CNT>
@@ -12,28 +12,28 @@ Release: %{rc_str}.<CI_CNT>.<B_CNT>
 Release: %{rc_str}.0.0
 %endif
 
-Group:      Development/Tools/Other
-License:    GPL-2.0
-BuildArch:  noarch
-URL:        http://www.tizen.org
-Source0:    %{name}_%{version}.tar.gz
+Group: Development/Tools/Other
+License: GPL-2.0
+BuildArch: noarch
+URL: http://www.tizen.org
+Source0: %{name}_%{version}.tar.gz
 
-BuildRequires:  python-distribute
+BuildRequires: python-distribute
 
 # In OpenSuse the xml.etree module is provided by the python-xml package
 %if 0%{?suse_version}
-Requires:	python-xml
+Requires: python-xml
 %endif
 
 # In Fedora the xml.etree module is provided by the python-libs package
 %if 0%{?fedora_version}
-Requires:	python-libs
+Requires: python-libs
 %endif
 
 # Centos6 uses python 2.6, which does not have the argparse module. However,
 # argparse is available as a separate package there.
 %if 0%{?centos_version} == 600
-Requires:	python-argparse
+Requires: python-argparse
 %endif
 
 %description
-- 
1.8.1.4



From dedekind1 at gmail.com  Tue Aug 13 03:29:42 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Tue, 13 Aug 2013 10:29:42 +0300
Subject: [bmap-tools] [PATCH 4/4] debian: add a dependency to python-lzma
In-Reply-To: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
References: <1376378982-26716-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376378982-26716-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Iafcae3a81f64934ff6d7fdede070fe39e75a58e1
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 debian/control | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/control b/debian/control
index eeed199..36bc7a9 100644
--- a/debian/control
+++ b/debian/control
@@ -11,6 +11,7 @@ XS-Python-Version: >= 2.7
 Package: bmap-tools
 Architecture: all
 Depends: python (>=2.7),
+         python-lzma,
          ${misc:Depends},
          ${python:Depends},
 Description: tool to flash image files to block devices using the block map
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:27 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:27 +0300
Subject: [bmap-tools] [PATCH 3/7] TransRead: close all the files
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Although CPython reference-counts objects and destroys them when they are no
longer used, it is still good practice to close all the opened files
explicitly, especially if we are talking about a library.

TransRead did not explicitely close the tar file object, and this patch fixes
this.

Additionally, add few useful commentaries.

Change-Id: Ib146be0a8e358bf208329932ca3a62ab2617f7bd
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 16d9525..ad38631 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -228,12 +228,27 @@ class TransRead:
         """
 
         self.name = filepath
+        # Size of the file (in uncompressed form), may be 'None' if the size is
+        # unknown
         self.size = None
+        # Whether the file is compressed
         self.is_compressed = True
+        # Whether the file is behind an URL
         self.is_url = False
+
+        # Wait for this child process in the destructor
         self._child_process = None
+        # There may be a chain of open files, and we save the intermediate file
+        # descriptors in 'self._file_objX', while the final file descriptor is
+        # stored in 'self._transfile_obj'. For example, when the path is an URL
+        # to a tar.bz2 file, the chain of opened file will be:
+        #   o self._file_obj is an liburl2 file-like object
+        #   o self._file_obj2 is a tarfile file-like object
+        #   o self._transfile_obj is a tarfile member file-like object
         self._file_obj = None
+        self._file_obj2 = None
         self._transfile_obj = None
+
         self._force_fake_seek = False
         self._pos = 0
 
@@ -257,6 +272,8 @@ class TransRead:
             self._transfile_obj.close()
         if self._file_obj:
             self._file_obj.close()
+        if self._file_obj2:
+            self._file_obj2.close()
         if self._child_process:
             self._child_process.wait()
 
@@ -273,9 +290,10 @@ class TransRead:
                or self.name.endswith('.tgz'):
                 import tarfile
 
-                tar = tarfile.open(fileobj=self._file_obj, mode='r|*')
-                member = tar.next()
-                self._transfile_obj = tar.extractfile(member)
+                self._file_obj2 = tarfile.open(fileobj=self._file_obj,
+                                               mode='r|*')
+                member = self._file_obj2.next()
+                self._transfile_obj = self._file_obj2.extractfile(member)
                 self.size = member.size
             elif self.name.endswith('.gz'):
                 import zlib
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:25 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:25 +0300
Subject: [bmap-tools] [PATCH 1/7] bmaptools: put __init__ first
Message-ID: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

It is just a common convention to put __init__ at the very beginning of the
class. And let's also put then the __del__ function to be the second.

So this patch does not do any functional modifications, just re-structuring.

Change-Id: I2ebdd8c4c54cf5bf3f9be21d7520b42865305dab
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py   | 256 ++++++++++++++++++++++++------------------------
 bmaptools/BmapCreate.py |  54 +++++-----
 bmaptools/Fiemap.py     |  20 ++--
 bmaptools/TransRead.py  |  86 ++++++++--------
 4 files changed, 208 insertions(+), 208 deletions(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index a025cb5..622fd49 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -117,6 +117,85 @@ class BmapCopy:
     instance.
     """
 
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes.
+            logger     - the logger object to use for printing messages.
+        """
+
+        self._logger = logger
+        if self._logger is None:
+            self._logger = logging.getLogger(__name__)
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
     def set_progress_indicator(self, file_obj, format_string):
         """
         Setup the progress indicator which shows how much data has been copied
@@ -226,85 +305,6 @@ class BmapCopy:
             # Bmap file checksum appeard in format 1.3
             self._verify_bmap_checksum()
 
-    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
-        """
-        The class constructor. The parameters are:
-            image      - file-like object of the image which should be copied,
-                         should only support 'read()' and 'seek()' methods,
-                         and only seeking forward has to be supported.
-            dest       - file object of the destination file to copy the image
-                         to.
-            bmap       - file object of the bmap file to use for copying.
-            image_size - size of the image in bytes.
-            logger     - the logger object to use for printing messages.
-        """
-
-        self._logger = logger
-        if self._logger is None:
-            self._logger = logging.getLogger(__name__)
-
-        self._xml = None
-
-        self._dest_fsync_watermark = None
-        self._batch_blocks = None
-        self._batch_queue = None
-        self._batch_bytes = 1024 * 1024
-        self._batch_queue_len = 2
-
-        self.bmap_version = None
-        self.bmap_version_major = None
-        self.bmap_version_minor = None
-        self.block_size = None
-        self.blocks_cnt = None
-        self.mapped_cnt = None
-        self.image_size = None
-        self.image_size_human = None
-        self.mapped_size = None
-        self.mapped_size_human = None
-        self.mapped_percent = None
-
-        self._f_bmap = None
-        self._f_bmap_path = None
-
-        self._progress_started = None
-        self._progress_index = None
-        self._progress_time = None
-        self._progress_file = None
-        self._progress_format = None
-        self.set_progress_indicator(None, None)
-
-        self._f_image = image
-        self._image_path = image.name
-
-        self._f_dest = dest
-        self._dest_path = dest.name
-        st_data = os.fstat(self._f_dest.fileno())
-        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
-
-        # Special quirk for /dev/null which does not support fsync()
-        if stat.S_ISCHR(st_data.st_mode) and \
-           os.major(st_data.st_rdev) == 1 and \
-           os.minor(st_data.st_rdev) == 3:
-            self._dest_supports_fsync = False
-        else:
-            self._dest_supports_fsync = True
-
-        if bmap:
-            self._f_bmap = bmap
-            self._bmap_path = bmap.name
-            self._parse_bmap()
-        else:
-            # There is no bmap. Initialize user-visible attributes to something
-            # sensible with an assumption that we just have all blocks mapped.
-            self.bmap_version = 0
-            self.block_size = 4096
-            self.mapped_percent = 100
-
-        if image_size:
-            self._set_image_size(image_size)
-
-        self._batch_blocks = self._batch_bytes / self.block_size
-
     def _update_progress(self, blocks_written):
         """
         Print the progress indicator if the mapped area size is known and if
@@ -584,6 +584,55 @@ class BmapBdevCopy(BmapCopy):
     scheduler.
     """
 
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices.
+        """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size, logger=logger)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not "
+                            "fit the block device '%s' which has %s capacity"
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
+
     def _tune_block_device(self):
         """
         Tune the block device for better performance:
@@ -673,52 +722,3 @@ class BmapBdevCopy(BmapCopy):
             raise
         finally:
             self._restore_bdev_settings()
-
-    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
-        """
-        The same as the constructor of the 'BmapCopy' base class, but adds
-        useful guard-checks specific to block devices.
-        """
-
-        # Call the base class constructor first
-        BmapCopy.__init__(self, image, dest, bmap, image_size, logger=logger)
-
-        self._batch_bytes = 1024 * 1024
-        self._batch_blocks = self._batch_bytes / self.block_size
-        self._batch_queue_len = 6
-        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
-
-        self._sysfs_base = None
-        self._sysfs_scheduler_path = None
-        self._sysfs_max_ratio_path = None
-        self._old_scheduler_value = None
-        self._old_max_ratio_value = None
-
-        # If the image size is known, check that it fits the block device
-        if self.image_size:
-            try:
-                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
-                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
-            except OSError as err:
-                raise Error("cannot seed block device '%s': %s "
-                            % (self._dest_path, err.strerror))
-
-            if bdev_size < self.image_size:
-                raise Error("the image file '%s' has size %s and it will not "
-                            "fit the block device '%s' which has %s capacity"
-                            % (self._image_path, self.image_size_human,
-                               self._dest_path, human_size(bdev_size)))
-
-        # Construct the path to the sysfs directory of our block device
-        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
-        self._sysfs_base = "/sys/dev/block/%s:%s/" \
-                           % (os.major(st_rdev), os.minor(st_rdev))
-
-        # Check if the 'queue' sub-directory exists. If yes, then our block
-        # device is entire disk. Otherwise, it is a partition, in which case we
-        # need to go one level up in the sysfs hierarchy.
-        if not os.path.exists(self._sysfs_base + "queue"):
-            self._sysfs_base = self._sysfs_base + "../"
-
-        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
-        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/bmaptools/BmapCreate.py b/bmaptools/BmapCreate.py
index cb45bc8..b96224a 100644
--- a/bmaptools/BmapCreate.py
+++ b/bmaptools/BmapCreate.py
@@ -106,26 +106,6 @@ class BmapCreate:
     the FIEMAP ioctl to generate the bmap.
     """
 
-    def _open_image_file(self):
-        """Open the image file."""
-        try:
-            self._f_image = open(self._image_path, 'rb')
-        except IOError as err:
-            raise Error("cannot open image file '%s': %s"
-                        % (self._image_path, err))
-
-        self._f_image_needs_close = True
-
-    def _open_bmap_file(self):
-        """Open the bmap file."""
-        try:
-            self._f_bmap = open(self._bmap_path, 'w+')
-        except IOError as err:
-            raise Error("cannot open bmap file '%s': %s"
-                        % (self._bmap_path, err))
-
-        self._f_bmap_needs_close = True
-
     def __init__(self, image, bmap):
         """
         Initialize a class instance:
@@ -176,6 +156,33 @@ class BmapCreate:
         self.block_size = self.fiemap.block_size
         self.blocks_cnt = self.fiemap.blocks_cnt
 
+    def __del__(self):
+        """The class destructor which closes the opened files."""
+        if self._f_image_needs_close:
+            self._f_image.close()
+        if self._f_bmap_needs_close:
+            self._f_bmap.close()
+
+    def _open_image_file(self):
+        """Open the image file."""
+        try:
+            self._f_image = open(self._image_path, 'rb')
+        except IOError as err:
+            raise Error("cannot open image file '%s': %s"
+                        % (self._image_path, err))
+
+        self._f_image_needs_close = True
+
+    def _open_bmap_file(self):
+        """Open the bmap file."""
+        try:
+            self._f_bmap = open(self._bmap_path, 'w+')
+        except IOError as err:
+            raise Error("cannot open bmap file '%s': %s"
+                        % (self._bmap_path, err))
+
+        self._f_bmap_needs_close = True
+
     def _bmap_file_start(self):
         """
         A helper function which generates the starting contents of the block
@@ -313,10 +320,3 @@ class BmapCreate:
                         % (self._bmap_path, err))
 
         self._f_image.seek(image_pos)
-
-    def __del__(self):
-        """The class destructor which closes the opened files."""
-        if self._f_image_needs_close:
-            self._f_image.close()
-        if self._f_bmap_needs_close:
-            self._f_bmap.close()
diff --git a/bmaptools/Fiemap.py b/bmaptools/Fiemap.py
index 76857b8..3066f18 100644
--- a/bmaptools/Fiemap.py
+++ b/bmaptools/Fiemap.py
@@ -59,16 +59,6 @@ class Fiemap:
     over all mapped blocks and over all holes.
     """
 
-    def _open_image_file(self):
-        """Open the image file."""
-        try:
-            self._f_image = open(self._image_path, 'rb')
-        except IOError as err:
-            raise Error("cannot open image file '%s': %s"
-                        % (self._image_path, err))
-
-        self._f_image_needs_close = True
-
     def __init__(self, image, buf_size=DEFAULT_BUFFER_SIZE):
         """
         Initialize a class instance. The 'image' argument is full path to the
@@ -134,6 +124,16 @@ class Fiemap:
         if self._f_image_needs_close:
             self._f_image.close()
 
+    def _open_image_file(self):
+        """Open the image file."""
+        try:
+            self._f_image = open(self._image_path, 'rb')
+        except IOError as err:
+            raise Error("cannot open image file '%s': %s"
+                        % (self._image_path, err))
+
+        self._f_image_needs_close = True
+
     def _invoke_fiemap(self, block, count):
         """
         Invoke the FIEMAP ioctl for 'count' blocks of the file starting from
diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 870b584..16d9525 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -217,6 +217,49 @@ class TransRead:
     this class are file-like objects which you can read and seek only forward.
     """
 
+    def __init__(self, filepath, local=False):
+        """
+        Class constructor. The 'filepath' argument is the full path to the file
+        to read transparently. If 'local' is True, then the file-like object is
+        guaranteed to be backed by a local file. This means that if the source
+        file is compressed or an URL, then it will first be copied to a
+        temporary local file, and then all the subsequent operations will be
+        done with the local copy.
+        """
+
+        self.name = filepath
+        self.size = None
+        self.is_compressed = True
+        self.is_url = False
+        self._child_process = None
+        self._file_obj = None
+        self._transfile_obj = None
+        self._force_fake_seek = False
+        self._pos = 0
+
+        try:
+            self._file_obj = open(self.name, "rb")
+        except IOError as err:
+            if err.errno == errno.ENOENT:
+                # This is probably an URL
+                self._open_url(filepath)
+            else:
+                raise Error("cannot open file '%s': %s" % (filepath, err))
+
+        self._open_compressed_file()
+
+        if local:
+            self._create_local_copy()
+
+    def __del__(self):
+        """The class destructor which closes opened files."""
+        if self._transfile_obj:
+            self._transfile_obj.close()
+        if self._file_obj:
+            self._file_obj.close()
+        if self._child_process:
+            self._child_process.wait()
+
     def _open_compressed_file(self):
         """
         Detect file compression type and open it with the corresponding
@@ -417,40 +460,6 @@ class TransRead:
             raise Error("cannot open own temporary file '%s': %s"
                         % (tmp_file_obj.name, err))
 
-    def __init__(self, filepath, local=False):
-        """
-        Class constructor. The 'filepath' argument is the full path to the file
-        to read transparently. If 'local' is True, then the file-like object is
-        guaranteed to be backed by a local file. This means that if the source
-        file is compressed or an URL, then it will first be copied to a
-        temporary local file, and then all the subsequent operations will be
-        done with the local copy.
-        """
-
-        self.name = filepath
-        self.size = None
-        self.is_compressed = True
-        self.is_url = False
-        self._child_process = None
-        self._file_obj = None
-        self._transfile_obj = None
-        self._force_fake_seek = False
-        self._pos = 0
-
-        try:
-            self._file_obj = open(self.name, "rb")
-        except IOError as err:
-            if err.errno == errno.ENOENT:
-                # This is probably an URL
-                self._open_url(filepath)
-            else:
-                raise Error("cannot open file '%s': %s" % (filepath, err))
-
-        self._open_compressed_file()
-
-        if local:
-            self._create_local_copy()
-
     def read(self, size=-1):
         """
         Read the data from the file or URL and and uncompress it on-the-fly if
@@ -465,15 +474,6 @@ class TransRead:
 
         return buf
 
-    def __del__(self):
-        """The class destructor which closes opened files."""
-        if self._transfile_obj:
-            self._transfile_obj.close()
-        if self._file_obj:
-            self._file_obj.close()
-        if self._child_process:
-            self._child_process.wait()
-
     def seek(self, offset, whence=os.SEEK_SET):
         """The 'seek()' method, similar to the one file objects have."""
         if self._force_fake_seek or not hasattr(self._transfile_obj, "seek"):
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:26 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:26 +0300
Subject: [bmap-tools] [PATCH 2/7] BmapCopy: fix a typo in error message
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

... add a missing whitespace.

Change-Id: I8d713263de9899ff314d83e69aae9f3a672c3285
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index 622fd49..4a9c8cc 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -541,7 +541,7 @@ class BmapCopy:
         # 'mapped_cnt' blocks.
         if blocks_written != self.mapped_cnt:
             raise Error("wrote %u blocks from image '%s' to '%s', but should "
-                        "have %u - bmap file '%s' does not belong to this"
+                        "have %u - bmap file '%s' does not belong to this "
                         "image"
                         % (blocks_written, self._image_path, self._dest_path,
                            self.mapped_cnt, self._bmap_path))
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:28 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:28 +0300
Subject: [bmap-tools] [PATCH 4/7] TransRead: uncompress .xz files on-the-fly
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I401e20224bc2d0050b7c6354e346e10f13e99024
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index ad38631..a3d34e8 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -13,7 +13,7 @@
 This module allows opening and reading local and remote files and decompress
 them on-the-fly if needed. Remote files are read using urllib2 (except of
 "ssh://" URLs, which are handled differently). Supported compression types are:
-'bz2', 'gz', 'tar.gz', 'tgz', 'tar.bz2'.
+'bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2'.
 """
 
 import os
@@ -29,7 +29,7 @@ import urlparse
 # pylint: disable=R0902
 
 # A list of supported compression types
-SUPPORTED_COMPRESSION_TYPES = ('bz2', 'gz', 'tar.gz', 'tgz', 'tar.bz2')
+SUPPORTED_COMPRESSION_TYPES = ('bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2')
 
 def _fake_seek_forward(file_obj, cur_pos, offset, whence=os.SEEK_SET):
     """
@@ -307,6 +307,19 @@ class TransRead:
                 self._transfile_obj = _CompressedFile(self._file_obj,
                                               bz2.BZ2Decompressor().decompress,
                                               128)
+            elif self.name.endswith('.xz'):
+                try:
+                    import lzma
+                except ImportError:
+                    try:
+                        from backports import lzma
+                    except ImportError:
+                        raise Error("cannot import the \"lzma\" python module, "
+                                    "it is required for decompressing .xz files")
+
+                self._transfile_obj = _CompressedFile(self._file_obj,
+                                              lzma.LZMADecompressor().decompress,
+                                              128)
             else:
                 self.is_compressed = False
                 self._transfile_obj = self._file_obj
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:30 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:30 +0300
Subject: [bmap-tools] [PATCH 6/7] RELEASE_NOTES: add changelog for release
	2.6
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-6-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Strictly speaking I've added a feature, so 2.6 is not just a bug-fix release,
but the feature is rather small and does not deserve a new major release.

Change-Id: Ice89d533a873687141412aaefe814eb3b221ac5f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 docs/RELEASE_NOTES | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/docs/RELEASE_NOTES b/docs/RELEASE_NOTES
index e2bf833..5cc1ae2 100644
--- a/docs/RELEASE_NOTES
+++ b/docs/RELEASE_NOTES
@@ -1,3 +1,8 @@
+Bug-fix release 2.6
+~~~~~~~~~~~~~~~~~~~
+
+1. Add on-the-fly decompression support for '.xz' and '.tar.xz' files.
+
 Bug-fix release 2.5
 ~~~~~~~~~~~~~~~~~~~
 
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:31 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:31 +0300
Subject: [bmap-tools] [PATCH 7/7] packaging: remove unneeded files
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-7-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

I cannot really explain why these files were needed, but they are not needed
anymore, according to Ed Bartosh. So removing them with pleasure.

Change-Id: I8239416e603ed86f2e0302cc1a7c14a9ba94d2dd
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 packaging/Makefile       | 21 ---------------------
 packaging/bmap-tools.dsc | 10 ----------
 2 files changed, 31 deletions(-)
 delete mode 100644 packaging/Makefile
 delete mode 100644 packaging/bmap-tools.dsc

diff --git a/packaging/Makefile b/packaging/Makefile
deleted file mode 100644
index 594e7e3..0000000
--- a/packaging/Makefile
+++ /dev/null
@@ -1,21 +0,0 @@
-PKG_NAME := bmap-tools
-SPECFILE = $(addsuffix .spec, $(PKG_NAME))
-PKG_VERSION := $(shell grep '^Version: ' $(SPECFILE)|awk '{print $$2}')
-
-TARBALL := $(PKG_NAME)_$(PKG_VERSION).tar.gz
-
-dsc: $(TARBALL)
-		$(eval MD5=$(shell md5sum $(TARBALL) | sed "s/  / $(shell stat -c '%s' $(TARBALL)) /"))
-			@sed  -i 's/^Version:.*/Version: $(PKG_VERSION)/' $(PKG_NAME).dsc
-			@sed  -i 's/ [a-f0-9]\+ [0-9]\+ $(PKG_NAME).*tar.*/ $(MD5)/' $(PKG_NAME).dsc
-
-tarball: $(TARBALL)
-$(TARBALL):
-		cd "$$(git rev-parse --show-toplevel)" \
-			&& git archive --prefix $(PKG_NAME)-$(PKG_VERSION)/ HEAD \
-			| gzip > "$(CURDIR)/$(TARBALL)"
-
-clean:
-		rm -f $(PKG_NAME)*.tar.gz
-
-all: clean tarball dsc
diff --git a/packaging/bmap-tools.dsc b/packaging/bmap-tools.dsc
deleted file mode 100644
index 09a925c..0000000
--- a/packaging/bmap-tools.dsc
+++ /dev/null
@@ -1,10 +0,0 @@
-Format: 1.0
-Source: bmap-tools
-Binary: bmap-tools
-Architecture: all
-Version: 2.5-1
-Maintainer: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
-Standards-Version: 3.8.4
-Build-Depends: debhelper (>= 7), python-all, python-distribute
-Files:
- 002b1492e8ebee0d01d58620a91c87cc 8145 bmap-tools_0.1.0.tar.gz
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 04:13:29 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 11:13:29 +0300
Subject: [bmap-tools] [PATCH 5/7] TransRead: uncompress 'tar.xz' files
	on-the-fly
In-Reply-To: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
References: <1376468011-4973-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376468011-4973-5-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ia8ed3c1c276510b3d63e831135f7ad6ce6bc1743
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index a3d34e8..1b9abff 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -13,7 +13,7 @@
 This module allows opening and reading local and remote files and decompress
 them on-the-fly if needed. Remote files are read using urllib2 (except of
 "ssh://" URLs, which are handled differently). Supported compression types are:
-'bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2'.
+'bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2', 'tar.xz'.
 """
 
 import os
@@ -29,7 +29,8 @@ import urlparse
 # pylint: disable=R0902
 
 # A list of supported compression types
-SUPPORTED_COMPRESSION_TYPES = ('bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2')
+SUPPORTED_COMPRESSION_TYPES = ('bz2', 'gz', 'xz', 'tar.gz', 'tgz', 'tar.bz2',
+                               'tar.xz')
 
 def _fake_seek_forward(file_obj, cur_pos, offset, whence=os.SEEK_SET):
     """
@@ -247,6 +248,7 @@ class TransRead:
         #   o self._transfile_obj is a tarfile member file-like object
         self._file_obj = None
         self._file_obj2 = None
+        self._file_obj3 = None
         self._transfile_obj = None
 
         self._force_fake_seek = False
@@ -270,10 +272,12 @@ class TransRead:
         """The class destructor which closes opened files."""
         if self._transfile_obj:
             self._transfile_obj.close()
-        if self._file_obj:
-            self._file_obj.close()
+        if self._file_obj3:
+            self._file_obj3.close()
         if self._file_obj2:
             self._file_obj2.close()
+        if self._file_obj:
+            self._file_obj.close()
         if self._child_process:
             self._child_process.wait()
 
@@ -320,6 +324,15 @@ class TransRead:
                 self._transfile_obj = _CompressedFile(self._file_obj,
                                               lzma.LZMADecompressor().decompress,
                                               128)
+                if self.name.endswith('.tar.xz'):
+                    import tarfile
+
+                    self._file_obj2 = self._transfile_obj
+                    self._file_obj3 = tarfile.open(fileobj=self._file_obj2,
+                                                   mode='r|*')
+                    member = self._file_obj3.next()
+                    self._transfile_obj = self._file_obj3.extractfile(member)
+                    self.size = member.size
             else:
                 self.is_compressed = False
                 self._transfile_obj = self._file_obj
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 07:50:43 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 14:50:43 +0300
Subject: [bmap-tools] [PATCH 2/4] tests: cover .xz files too
In-Reply-To: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
References: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376481045-16414-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I4e8318174a0a92f7c795c0c5cc61aeeff7237442
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 58f64b3..e814fa0 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -57,6 +57,15 @@ def _generate_compressed_files(file_obj, delete=True):
     import tarfile
     import shutil
 
+    lzma_present = True
+    try:
+        import lzma
+    except ImportError:
+        try:
+            from backports import lzma
+        except ImportError:
+            lzma_present = False
+
     # Make sure the temporary files start with the same name as 'file_obj' in
     # order to simplify debugging.
     prefix = os.path.splitext(os.path.basename(file_obj.name))[0] + '.'
@@ -95,6 +104,18 @@ def _generate_compressed_files(file_obj, delete=True):
     yield gzip_file_obj.name
     tmp_file_obj.close()
 
+    if lzma_present:
+        # Generate an .xz version of the file
+        tmp_file_obj = tempfile.NamedTemporaryFile('wb+', prefix=prefix,
+                                                   delete=delete, dir=directory,
+                                                   suffix='.xz')
+        lzma_file_obj = lzma.LZMAFile(tmp_file_obj.name, 'wb')
+        file_obj.seek(0)
+        shutil.copyfileobj(file_obj, lzma_file_obj)
+        lzma_file_obj.close()
+        yield lzma_file_obj.name
+        tmp_file_obj.close()
+
     # Generate a tar.gz version of the file
     tmp_file_obj = tempfile.NamedTemporaryFile('wb+', prefix=prefix,
                                                delete=delete, dir=directory,
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 07:50:44 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 14:50:44 +0300
Subject: [bmap-tools] [PATCH 3/4] TransRead: do not cache local uncompressed
	files
In-Reply-To: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
References: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376481045-16414-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

When the file is already local and uncompressed, do not create a temporary copy
of it when 'local' is True.

On top of this, re-use own __init__ function to re-open the local copy of the
remote/compressed file.

Change-Id: I2093419ed2cfc6d40f73a47657682436b8932402
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 23 ++++++++---------------
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 9d064e6..a16d0c8 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -222,10 +222,10 @@ class TransRead:
         """
         Class constructor. The 'filepath' argument is the full path to the file
         to read transparently. If 'local' is True, then the file-like object is
-        guaranteed to be backed by a local file. This means that if the source
-        file is compressed or an URL, then it will first be copied to a
-        temporary local file, and then all the subsequent operations will be
-        done with the local copy.
+        guaranteed to be backed by an uncompressed local file. This means that
+        if the source file is compressed and/or an URL, then it will first be
+        copied to an temporary local file, and then all the subsequent
+        operations will be done with the uncompresed local copy.
         """
 
         self.name = filepath
@@ -266,7 +266,7 @@ class TransRead:
 
         self._open_compressed_file()
 
-        if local:
+        if local and (self.is_url or self.is_compressed):
             self._create_local_copy()
 
     def __del__(self):
@@ -491,17 +491,10 @@ class TransRead:
             tmp_file_obj.write(chunk)
 
         tmp_file_obj.flush()
-        self.close()
-        self.is_compressed = False
-        self.is_url = False
-        self._f_objs.append(tmp_file_obj)
 
-        try:
-            f_obj = open(tmp_file_obj.name, "rb")
-            self._f_objs.appen(f_obj)
-        except IOError as err:
-            raise Error("cannot open own temporary file '%s': %s"
-                        % (tmp_file_obj.name, err))
+        self.close()
+        self.__init__(tmp_file_obj.name, local = False)
+        tmp_file_obj.close()
 
     def read(self, size=-1):
         """
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 07:50:45 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 14:50:45 +0300
Subject: [bmap-tools] [PATCH 4/4] Add the TODO list
In-Reply-To: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
References: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376481045-16414-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ib67bb47a25e3781ee6986693ed45fe22a418f343
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 TODO

diff --git a/TODO b/TODO
new file mode 100644
index 0000000..939ef3a
--- /dev/null
+++ b/TODO
@@ -0,0 +1,8 @@
+Current TODO list, any help with these is appreciated.
+
+1. Switch from SHA1 to SHA256
+2. Implement GPG signatures support, just like this is done in Fedora
+   https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
+3. In setup.py, fetch the version from 'bmaptool' file.
+4. Teach the 'make_a_release.sh' script to update the version and the
+   changelog.
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Aug 14 07:50:42 2013
From: dedekind1 at gmail.com (dedekind1 at gmail.com)
Date: Wed, 14 Aug 2013 14:50:42 +0300
Subject: [bmap-tools] [PATCH 1/4] TransRead: store all file descriptors in a
	list
Message-ID: <1376481045-16414-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This patch improves readability and does not do any functional changes.

In the TransRead module we have chains of file-like objects, every next element
of the chain is based on the previous one. And we store each element of the
chain in own variable like 'self._file_obj3', which is a bit ugly. Let's
introduce a 'self._f_objs' list, and just append there, and the last element is
always the final "transparent read" file descriptor.

Change-Id: I498130e9de35e65bddc7d3dbb4f47bdab795549e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 105 ++++++++++++++++++++++++-------------------------
 1 file changed, 52 insertions(+), 53 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 1b9abff..9d064e6 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -239,23 +239,24 @@ class TransRead:
 
         # Wait for this child process in the destructor
         self._child_process = None
+
         # There may be a chain of open files, and we save the intermediate file
-        # descriptors in 'self._file_objX', while the final file descriptor is
-        # stored in 'self._transfile_obj'. For example, when the path is an URL
-        # to a tar.bz2 file, the chain of opened file will be:
-        #   o self._file_obj is an liburl2 file-like object
-        #   o self._file_obj2 is a tarfile file-like object
-        #   o self._transfile_obj is a tarfile member file-like object
-        self._file_obj = None
-        self._file_obj2 = None
-        self._file_obj3 = None
-        self._transfile_obj = None
+        # objects in the 'self._f_objs' list. The final file object is stored
+        # in th elast element of the list.
+        #
+        # For example, when the path is an URL to a tar.xz file, the chain of
+        # opened file will be:
+        #   o self._f_objs[0] is the liburl2 file-like object
+        #   o self._f_objs[1] is the lzma file-like object
+        #   o self._f_objs[2] is the tarfile file-like object
+        #   o self._f_objs[3] is the tarfilemember file-like object
+        self._f_objs = []
 
         self._force_fake_seek = False
         self._pos = 0
 
         try:
-            self._file_obj = open(self.name, "rb")
+            self._f_objs.append(open(self.name, "rb"))
         except IOError as err:
             if err.errno == errno.ENOENT:
                 # This is probably an URL
@@ -270,16 +271,9 @@ class TransRead:
 
     def __del__(self):
         """The class destructor which closes opened files."""
-        if self._transfile_obj:
-            self._transfile_obj.close()
-        if self._file_obj3:
-            self._file_obj3.close()
-        if self._file_obj2:
-            self._file_obj2.close()
-        if self._file_obj:
-            self._file_obj.close()
-        if self._child_process:
-            self._child_process.wait()
+        for _file_obj in self._f_objs:
+            if _file_obj:
+                _file_obj.close()
 
     def _open_compressed_file(self):
         """
@@ -294,23 +288,27 @@ class TransRead:
                or self.name.endswith('.tgz'):
                 import tarfile
 
-                self._file_obj2 = tarfile.open(fileobj=self._file_obj,
-                                               mode='r|*')
-                member = self._file_obj2.next()
-                self._transfile_obj = self._file_obj2.extractfile(member)
+                f_obj = tarfile.open(fileobj=self._f_objs[-1], mode='r|*')
+                self._f_objs.append(f_obj)
+
+                member = self._f_objs[-1].next()
                 self.size = member.size
+                f_obj = self._f_objs[-1].extractfile(member)
+                self._f_objs.append(f_obj)
             elif self.name.endswith('.gz'):
                 import zlib
 
                 decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS)
-                self._transfile_obj = _CompressedFile(self._file_obj,
-                                                      decompressor.decompress)
+                f_obj = _CompressedFile(self._f_objs[-1],
+                                        decompressor.decompress)
+                self._f_objs.append(f_obj)
             elif self.name.endswith('.bz2'):
                 import bz2
 
-                self._transfile_obj = _CompressedFile(self._file_obj,
-                                              bz2.BZ2Decompressor().decompress,
-                                              128)
+
+                f_obj = _CompressedFile(self._f_objs[-1],
+                                        bz2.BZ2Decompressor().decompress, 128)
+                self._f_objs.append(f_obj)
             elif self.name.endswith('.xz'):
                 try:
                     import lzma
@@ -319,26 +317,26 @@ class TransRead:
                         from backports import lzma
                     except ImportError:
                         raise Error("cannot import the \"lzma\" python module, "
-                                    "it is required for decompressing .xz files")
+                                    "it's required for decompressing .xz files")
+
+                f_obj = _CompressedFile(self._f_objs[-1],
+                                        lzma.LZMADecompressor().decompress, 128)
+                self._f_objs.append(f_obj)
 
-                self._transfile_obj = _CompressedFile(self._file_obj,
-                                              lzma.LZMADecompressor().decompress,
-                                              128)
                 if self.name.endswith('.tar.xz'):
                     import tarfile
 
-                    self._file_obj2 = self._transfile_obj
-                    self._file_obj3 = tarfile.open(fileobj=self._file_obj2,
-                                                   mode='r|*')
-                    member = self._file_obj3.next()
-                    self._transfile_obj = self._file_obj3.extractfile(member)
+                    f_obj = tarfile.open(fileobj=self._f_objs[-1], mode='r|*')
+                    self._f_objs.append(f_obj)
+
+                    member = self._f_objs[-1].next()
                     self.size = member.size
+                    f_obj = self._f_objs[-1].extractfile(member)
+                    self._f_objs.append(f_obj)
             else:
                 self.is_compressed = False
-                self._transfile_obj = self._file_obj
                 if not self.is_url:
-                    self.size = os.fstat(self._file_obj.fileno()).st_size
-                self._file_obj = None
+                    self.size = os.fstat(self._f_objs[-1].fileno()).st_size
         except IOError as err:
             raise Error("cannot open file '%s': %s" % (self.name, err))
 
@@ -423,7 +421,7 @@ class TransRead:
 
         # Now the contents of the file should be available from sub-processes
         # stdout
-        self._file_obj = self._child_process.stdout
+        self._f_objs.append(self._child_process.stdout)
 
         self.is_url = True
         self._force_fake_seek = True
@@ -468,7 +466,7 @@ class TransRead:
         urllib2.install_opener(opener)
 
         try:
-            self._file_obj = opener.open(url)
+            self._f_objs.append(opener.open(url))
             self.is_url = True
         except (IOError, ValueError, httplib.InvalidURL) as err:
             raise Error("cannot open URL '%s': %s" % (url, err))
@@ -496,10 +494,11 @@ class TransRead:
         self.close()
         self.is_compressed = False
         self.is_url = False
-        self._file_obj = tmp_file_obj
+        self._f_objs.append(tmp_file_obj)
 
         try:
-            self._transfile_obj = open(tmp_file_obj.name, "rb")
+            f_obj = open(tmp_file_obj.name, "rb")
+            self._f_objs.appen(f_obj)
         except IOError as err:
             raise Error("cannot open own temporary file '%s': %s"
                         % (tmp_file_obj.name, err))
@@ -513,25 +512,25 @@ class TransRead:
         if size < 0:
             size = 0xFFFFFFFFFFFFFFFF
 
-        buf = self._transfile_obj.read(size)
+        buf = self._f_objs[-1].read(size)
         self._pos += len(buf)
 
         return buf
 
     def seek(self, offset, whence=os.SEEK_SET):
         """The 'seek()' method, similar to the one file objects have."""
-        if self._force_fake_seek or not hasattr(self._transfile_obj, "seek"):
-            self._pos = _fake_seek_forward(self._transfile_obj, self._pos,
+        if self._force_fake_seek or not hasattr(self._f_objs[-1], "seek"):
+            self._pos = _fake_seek_forward(self._f_objs[-1], self._pos,
                                            offset, whence)
         else:
-            self._transfile_obj.seek(offset, whence)
+            self._f_objs[-1].seek(offset, whence)
 
     def tell(self):
         """The 'tell()' method, similar to the one file objects have."""
-        if self._force_fake_seek or not hasattr(self._transfile_obj, "tell"):
+        if self._force_fake_seek or not hasattr(self._f_objs[-1], "tell"):
             return self._pos
         else:
-            return self._transfile_obj.tell()
+            return self._f_objs[-1].tell()
 
     def close(self):
         """Close the file-like object."""
@@ -544,6 +543,6 @@ class TransRead:
         """
 
         if not self.is_compressed and not self.is_url:
-            return getattr(self._transfile_obj, name)
+            return getattr(self._f_objs[-1], name)
         else:
             raise AttributeError
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Aug 16 05:49:34 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 16 Aug 2013 12:49:34 +0300
Subject: [bmap-tools] [PATCH 1/3] TODO: add more entries
Message-ID: <1376646576-6251-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I9cc5be56bb7f535fc20d50cc6ee1ffd5e708a138
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/TODO b/TODO
index 939ef3a..6e011e1 100644
--- a/TODO
+++ b/TODO
@@ -6,3 +6,10 @@ Current TODO list, any help with these is appreciated.
 3. In setup.py, fetch the version from 'bmaptool' file.
 4. Teach the 'make_a_release.sh' script to update the version and the
    changelog.
+5. Fsync before creating the bmap file
+6. Write in the man page, docs, and may be --help that it is very important
+   that the image is not changed while we create the bmep file, and after we
+   have created it too.
+7. Verify that unmapped areas have all zeroes to guarantee integrity.
+8. Update the man pages
+9. Teach bmaptool to update the alternate GPT partition
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Aug 16 05:49:35 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 16 Aug 2013 12:49:35 +0300
Subject: [bmap-tools] [PATCH 2/3] make_a_release.sh: add few more reminders
In-Reply-To: <1376646576-6251-1-git-send-email-dedekind1@gmail.com>
References: <1376646576-6251-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376646576-6251-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ifaf378a0b803a97ca3c15bb2b3adf8ffa73cdde9
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 make_a_release.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/make_a_release.sh b/make_a_release.sh
index d1e60af..1cdac49 100755
--- a/make_a_release.sh
+++ b/make_a_release.sh
@@ -79,8 +79,11 @@ gpg -o "$outdir/$release_name.tgz.asc" --detach-sign -a "$outdir/$release_name.t
 release_branch="release-$(printf "%s" "$new_ver" | sed -e 's/\(.*\)\..*/\1.0/')"
 
 cat <<EOF
-Make sure you have had updated the docs/RELEASE_NOTES file!
-Make sure you have had updated the version number and rpm/deb changelogs
+Make sure you updated the docs/RELEASE_NOTES file!
+Make sure you updated the docs/README file!
+Make sure you updated the version number and rpm/deb changelogs!
+Make sure you updated the man page!
+Make sure you updated the tizen.org documentation!
 
 To finish the release:
   1. push the $tag_name tag out
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Aug 16 05:49:36 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 16 Aug 2013 12:49:36 +0300
Subject: [bmap-tools] [PATCH 3/3] Release version 2.6
In-Reply-To: <1376646576-6251-1-git-send-email-dedekind1@gmail.com>
References: <1376646576-6251-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1376646576-6251-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ic0aba0b20847454b35c7dee9d50dccdbd2cf11fa
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool                     | 2 +-
 debian/changelog             | 6 ++++++
 packaging/bmap-tools.changes | 3 +++
 packaging/bmap-tools.spec    | 2 +-
 setup.py                     | 2 +-
 5 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/bmaptool b/bmaptool
index 10a35e9..50eeea1 100755
--- a/bmaptool
+++ b/bmaptool
@@ -46,7 +46,7 @@ also contribute to the mapped blocks and are also copied.
 #   * Too few public methods (R0903)
 # pylint: disable=R0903
 
-VERSION = "2.5"
+VERSION = "2.6"
 
 import argparse
 import sys
diff --git a/debian/changelog b/debian/changelog
index a1723ab..2b68ab4 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+bmap-tools (2.6) unstable; urgency=low
+
+  * Add support for on-the-fly decompression of '.xz' and '.tar.xz' files.
+
+ -- Artem Bityutskiy <artem.bityutskiy at linux.intel.com>  Tue, 13 Aug 2013 14:53:49 +0300
+
 bmap-tools (2.5) unstable; urgency=low
 
   * Do not fail when lacking permisssions for accessing block device's sysfs
diff --git a/packaging/bmap-tools.changes b/packaging/bmap-tools.changes
index 00d6ada..893fc7e 100644
--- a/packaging/bmap-tools.changes
+++ b/packaging/bmap-tools.changes
@@ -1,3 +1,6 @@
+Tue Aug 13 11:54:31 UTC 2013 - Artem Bityutskiy <artem.bityutskiy at linux.intel.com> 2.6-1
+- Add support for on-the-fly decompression of '.xz' and '.tar.xz' files.
+
 Mon Aug  5 07:05:59 UTC 2013 - Artem Bityutskiy <artem.bityutskiy at linux.intel.com> 2.5-1
 - Do not fail when lacking permisssions for accessing block device's sysfs
   files.
diff --git a/packaging/bmap-tools.spec b/packaging/bmap-tools.spec
index 94e851e..5d575f3 100644
--- a/packaging/bmap-tools.spec
+++ b/packaging/bmap-tools.spec
@@ -4,7 +4,7 @@
 
 Name: bmap-tools
 Summary: Tools to generate block map (AKA bmap) and flash images using bmap
-Version: 2.5
+Version: 2.6
 
 %if 0%{?opensuse_bs}
 Release: %{rc_str}.<CI_CNT>.<B_CNT>
diff --git a/setup.py b/setup.py
index 72fbd36..cc864a9 100644
--- a/setup.py
+++ b/setup.py
@@ -7,7 +7,7 @@ setup(
     description="Bmap tools",
     author="Artem Bityutskiy",
     author_email="artem.bityutskiy at linux.intel.com",
-    version="2.5",
+    version="2.6",
     scripts=['bmaptool'],
     packages=find_packages(exclude=["test*"]),
     license='GPLv2',
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 11 06:45:19 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 11 Sep 2013 13:45:19 +0300
Subject: [bmap-tools] Announcement: bmap-tools-2.6 is out!
Message-ID: <1378896319-13895-1-git-send-email-dedekind1@gmail.com>

Bmap-tools version 2.6 is out!

Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/release-2.0:/docs/RELEASE_NOTES
Tarball: ftp://ftp.infradead.org/pub/bmap-tools/

Packages for various distributions are available here:
http://download.tizen.org/tools/pre-release/

At some later point they will be propagated to here:
http://download.tizen.org/tools/latest-release/

--
Artem Bityutskiy


From dedekind1 at gmail.com  Fri Sep 13 06:48:12 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 13 Sep 2013 13:48:12 +0300
Subject: [bmap-tools] [PATCH 1/3] TransRead: handle the urllib2.URLError
	exception
Message-ID: <1379069294-17427-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

When opening an URL with urllib2, handle the URLError exceptions too.

This patch adds a new "except" statement instead of adding the exception object
to the existing array. The reason is that in the next commit we will need to
handle the urllib2.URLError exceptions a bit differently.

This patch also refactors the code a tiny bit as a preparation to the next
commit.

Change-Id: I69856aa8698b495c5a2450979db9384aad1b713a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index a16d0c8..45c243d 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -466,14 +466,18 @@ class TransRead:
         urllib2.install_opener(opener)
 
         try:
-            self._f_objs.append(opener.open(url))
-            self.is_url = True
+            f_obj = opener.open(url)
+        except urllib2.URLError as err:
+            raise Error("cannot open URL '%s': %s" % (url, err))
         except (IOError, ValueError, httplib.InvalidURL) as err:
             raise Error("cannot open URL '%s': %s" % (url, err))
         except httplib.BadStatusLine:
             raise Error("cannot open URL '%s': server responds with an HTTP "
                         "status code that we don't understand" % url)
 
+        self.is_url = True
+        self._f_objs.append(f_obj)
+
     def _create_local_copy(self):
         """Create a local copy of a remote or compressed file."""
         import tempfile
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Sep 13 06:48:14 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 13 Sep 2013 13:48:14 +0300
Subject: [bmap-tools] [PATCH 3/3] TransRead: improve user experience
In-Reply-To: <1379069294-17427-1-git-send-email-dedekind1@gmail.com>
References: <1379069294-17427-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379069294-17427-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This patch solves the following problem.

1. I forgot to define the proxy.
2. I run bmaptool, it blocks, and several minutes later it fails with an error
   like "Connection timed out"

I would instead like it to tell me that something is going wrong much earlier,
why should I wait for several minutes?

This patch improves the way we open URLs. Now instead of using the default
(usually long) timeout, we first try with a short timeout, and if we cannot
open the URL, we print user a warning, and then try to open with the default
timeout. The user may press Ctrl-C once he/she sees the warning, or start
checking the connectivity.

Change-Id: Ia5f387de62574d84d914c21794f782b1b41d530f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/TransRead.py | 47 ++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 38 insertions(+), 9 deletions(-)

diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 49c7b59..b8ab73b 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -439,8 +439,17 @@ class TransRead:
         Open an URL 'url' and return the file-like object of the opened URL.
         """
 
+        def _print_warning(logger, timeout):
+            """
+            This is a small helper function for printing a warning if we cannot
+            open the URL for some time.
+            """
+            logger.warning("failed to open the URL with %d sec timeout, is the "
+                           "proxy setup correctly? Keep trying..." % timeout)
+
         import urllib2
         import httplib
+        import socket
 
         parsed_url = urlparse.urlparse(url)
         username = parsed_url.username
@@ -473,15 +482,35 @@ class TransRead:
         opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
         urllib2.install_opener(opener)
 
-        try:
-            f_obj = opener.open(url)
-        except urllib2.URLError as err:
-            raise Error("cannot open URL '%s': %s" % (url, err))
-        except (IOError, ValueError, httplib.InvalidURL) as err:
-            raise Error("cannot open URL '%s': %s" % (url, err))
-        except httplib.BadStatusLine:
-            raise Error("cannot open URL '%s': server responds with an HTTP "
-                        "status code that we don't understand" % url)
+        # Open the URL. First try with a short timeout, and print a message
+        # which should supposedly give the a clue that something may be going
+        # wrong.
+        # The overall purpose of this is to improve user experience. For
+        # example, if one tries to open a file but did not setup the proxy
+        # environment variables propely, there will be a very long delay before
+        # the failure message. And it is much nicer to pre-warn the user early
+        # about something possibly being wrong.
+        for timeout in (10, None):
+            try:
+                f_obj = opener.open(url, timeout = timeout)
+            # Handling the timeout case in Python 2.7
+            except socket.timeout, err:
+                if timeout is not None:
+                    _print_warning(self._logger, timeout)
+                else:
+                    raise Error("cannot open URL '%s': %s" % (url, err))
+            except urllib2.URLError as err:
+                # Handling the timeout case in Python 2.7
+                if timeout is not None and \
+                   isinstance(err.reason, socket.timeout):
+                    _print_warning(self._logger, timeout)
+                else:
+                    raise Error("cannot open URL '%s': %s" % (url, err))
+            except (IOError, ValueError, httplib.InvalidURL) as err:
+                raise Error("cannot open URL '%s': %s" % (url, err))
+            except httplib.BadStatusLine:
+                raise Error("cannot open URL '%s': server responds with an "
+                            "HTTP status code that we don't understand" % url)
 
         self.is_url = True
         self._f_objs.append(f_obj)
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Sep 13 06:48:13 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri, 13 Sep 2013 13:48:13 +0300
Subject: [bmap-tools] [PATCH 2/3] TransRead: accept the logger object
In-Reply-To: <1379069294-17427-1-git-send-email-dedekind1@gmail.com>
References: <1379069294-17427-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379069294-17427-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

In order to improve user experience, I would like to print warnings when we
cannot open the URL for some time. This requires the TransRead object to accept
the logger object, or take the global one.

Change-Id: I4fc9f3d6544a2a699b6476dddf362aa127e81550
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool               | 10 +++++-----
 bmaptools/TransRead.py | 10 +++++++++-
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/bmaptool b/bmaptool
index 50eeea1..3c607e8 100755
--- a/bmaptool
+++ b/bmaptool
@@ -95,7 +95,7 @@ def copy_command_open_blkdev(path, log):
 
     return NamedFile(file_obj, path)
 
-def find_and_open_bmap(image_path):
+def find_and_open_bmap(image_path, log):
     """
     When the user does not specify the bmap file, we try to find it at the same
     place where the image file is located. We search for a file with the same
@@ -113,7 +113,7 @@ def find_and_open_bmap(image_path):
         bmap_path = image_path + ".bmap"
 
         try:
-            bmap_obj = TransRead.TransRead(bmap_path)
+            bmap_obj = TransRead.TransRead(bmap_path, logger=log)
             bmap_obj.close()
             return bmap_path
         except TransRead.Error:
@@ -140,7 +140,7 @@ def copy_command_open_all(args, log):
     # Open the image file using the TransRead module, which will automatically
     # recognize whether it is compressed or whether file path is an URL, etc.
     try:
-        image_obj = TransRead.TransRead(args.image)
+        image_obj = TransRead.TransRead(args.image, logger=log)
     except TransRead.Error as err:
         log.error("cannot open image: %s" % str(err))
         raise SystemExit(1)
@@ -149,7 +149,7 @@ def copy_command_open_all(args, log):
     # was not specified.
     bmap_path = args.bmap
     if not bmap_path and not args.nobmap:
-        bmap_path = find_and_open_bmap(args.image)
+        bmap_path = find_and_open_bmap(args.image, log)
         if bmap_path:
             log.info("discovered bmap file '%s'" % bmap_path)
 
@@ -158,7 +158,7 @@ def copy_command_open_all(args, log):
         try:
             # The current implementation of BmapCopy requires a local file for
             # the bmap file.
-            bmap_obj = TransRead.TransRead(bmap_path, local = True)
+            bmap_obj = TransRead.TransRead(bmap_path, local=True, logger=log)
         except TransRead.Error as err:
             log.error("cannot open bmap file '%s': %s" % (bmap_path, str(err)))
             raise SystemExit(1)
diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 45c243d..49c7b59 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -19,6 +19,7 @@ them on-the-fly if needed. Remote files are read using urllib2 (except of
 import os
 import errno
 import urlparse
+import logging
 
 # Disable the following pylint errors and recommendations:
 #   * Instance of X has no member Y (E1101), because it produces
@@ -218,7 +219,7 @@ class TransRead:
     this class are file-like objects which you can read and seek only forward.
     """
 
-    def __init__(self, filepath, local=False):
+    def __init__(self, filepath, local=False, logger=None):
         """
         Class constructor. The 'filepath' argument is the full path to the file
         to read transparently. If 'local' is True, then the file-like object is
@@ -226,8 +227,15 @@ class TransRead:
         if the source file is compressed and/or an URL, then it will first be
         copied to an temporary local file, and then all the subsequent
         operations will be done with the uncompresed local copy.
+
+        The "logger" argument is the logger object to use for printing
+        messages.
         """
 
+        self._logger = logger
+        if self._logger is None:
+            self._logger = logging.getLogger(__name__)
+
         self.name = filepath
         # Size of the file (in uncompressed form), may be 'None' if the size is
         # unknown
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:13 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:13 +0300
Subject: [bmap-tools] [PATCH 01/20] TODO: add another item
Message-ID: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I5cf46966858e3aa71f3cc6e55430499a9750730e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/TODO b/TODO
index 38431be..3e5f241 100644
--- a/TODO
+++ b/TODO
@@ -13,3 +13,5 @@ Current TODO list, any help with these is appreciated.
 7. Verify that unmapped areas have all zeroes to guarantee integrity.
 8. Update the man pages
 9. Teach bmaptool to update the alternate GPT partition
+10. Remove the whole "local=True" thing from TransRead completely and
+    do that in bmaptool directly.
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:14 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:14 +0300
Subject: [bmap-tools] [PATCH 02/20] TransRead: remove local caching
	functionality
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Remove the local caching functionality of the TransRead module. This
functionality does not really belong to the module, and it only makes things
more complex. It is better to either make users of TransRead to locally cache
remote files, or create a small wrapper over TransRead for these purposes.

Since currently there is only one user of this functionality, we just implement
local caching in bmaptool directly.

Change-Id: I649a5e761b9bc4b6b10d035bfd790357eaf6c373
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO                   |  2 --
 bmaptool               | 95 ++++++++++++++++++++++++++++++--------------------
 bmaptools/TransRead.py | 37 ++------------------
 3 files changed, 60 insertions(+), 74 deletions(-)

diff --git a/TODO b/TODO
index 3e5f241..38431be 100644
--- a/TODO
+++ b/TODO
@@ -13,5 +13,3 @@ Current TODO list, any help with these is appreciated.
 7. Verify that unmapped areas have all zeroes to guarantee integrity.
 8. Update the man pages
 9. Teach bmaptool to update the alternate GPT partition
-10. Remove the whole "local=True" thing from TransRead completely and
-    do that in bmaptool directly.
diff --git a/bmaptool b/bmaptool
index 3c607e8..bcc0b6b 100755
--- a/bmaptool
+++ b/bmaptool
@@ -56,6 +56,7 @@ import time
 import logging
 import tempfile
 import traceback
+import shutil
 from bmaptools import BmapCreate, BmapCopy, BmapHelpers, TransRead
 
 def copy_command_open_blkdev(path, log):
@@ -95,35 +96,66 @@ def copy_command_open_blkdev(path, log):
 
     return NamedFile(file_obj, path)
 
-def find_and_open_bmap(image_path, log):
+def find_and_open_bmap(args, log):
     """
-    When the user does not specify the bmap file, we try to find it at the same
-    place where the image file is located. We search for a file with the same
-    path and basename, but with a ".bmap" extension. Since the image may
-    contain more than one extension (e.g., image.raw.bz2), try to remove them
-    one-by-one.
-
-    This function returns a file-like object for the bmap file if it has been
-    found, and 'None' otherwise.
+    This function opens the bmap file and returns the corresponding file object
+    as the bmap file path.
+
+    If the user specified the bmap file explicitely, we just open the provided
+    path. Otherwise,  we try to find the bmap file at the same place where the
+    image file is located. We search for a file with the same path and
+    basename, but with a ".bmap" extension. Since the image may contain more
+    than one extension (e.g., image.raw.bz2), we try to remove them one-by-one.
+
+    Additionally, this function makes sure that the returned file object
+    corresponds to a local file, not a remote file. We do this by creating a
+    temporary local copy of the bmap file, if needed.
     """
 
-    bmap_path = None
-
-    while True:
-        bmap_path = image_path + ".bmap"
+    if args.nobmap:
+        return (None, None)
 
+    if args.bmap:
         try:
-            bmap_obj = TransRead.TransRead(bmap_path, logger=log)
-            bmap_obj.close()
-            return bmap_path
-        except TransRead.Error:
-            pass
+            bmap_obj = TransRead.TransRead(args.bmap, logger=log)
+        except TransRead.Error as err:
+            log.error("cannot open bmap file '%s': %s" % (args.bmap, str(err)))
+            raise SystemExit(1)
+        bmap_path = args.bmap
+    else:
+        # Automatically discover the bmap file
+        image_path = args.image
+        while True:
+            bmap_path = image_path + ".bmap"
+            try:
+                bmap_obj = TransRead.TransRead(bmap_path, logger=log)
+                log.info("discovered bmap file '%s'" % bmap_path)
+                break
+            except TransRead.Error:
+                pass
+
+            image_path, ext = os.path.splitext(image_path)
+            if ext == '':
+                return (None, None)
+
+    if not bmap_obj.is_compressed and not bmap_obj.is_url:
+        return (bmap_obj, bmap_path)
+
+    # Current implementation of BmapCopy requires a local file for block map,
+    # because it memory-maps it ('mmap()').
+    try:
+        # Create a temporary file for the bmap
+        tmp_obj = tempfile.TemporaryFile("w+")
+    except IOError as err:
+        log.error("cannot create a temporary file for bmap: %s" % err)
+        raise SystemExit(1)
 
-        image_path, ext = os.path.splitext(image_path)
-        if ext == '':
-            break
+    shutil.copyfileobj(bmap_obj, tmp_obj)
 
-    return None
+    tmp_obj.flush()
+    tmp_obj.seek(0)
+    bmap_obj.close()
+    return (tmp_obj, bmap_path)
 
 def copy_command_open_all(args, log):
     """
@@ -147,21 +179,7 @@ def copy_command_open_all(args, log):
 
     # Open the bmap file. Try to discover the bmap file automatically if it
     # was not specified.
-    bmap_path = args.bmap
-    if not bmap_path and not args.nobmap:
-        bmap_path = find_and_open_bmap(args.image, log)
-        if bmap_path:
-            log.info("discovered bmap file '%s'" % bmap_path)
-
-    bmap_obj = None
-    if bmap_path:
-        try:
-            # The current implementation of BmapCopy requires a local file for
-            # the bmap file.
-            bmap_obj = TransRead.TransRead(bmap_path, local=True, logger=log)
-        except TransRead.Error as err:
-            log.error("cannot open bmap file '%s': %s" % (bmap_path, str(err)))
-            raise SystemExit(1)
+    (bmap_obj, bmap_path) = find_and_open_bmap(args, log)
 
     # Try to open the destination file. If it does not exist, a new regular
     # file will be created. If it exists and it is a regular file - it'll be
@@ -210,7 +228,8 @@ def copy_command(args, log):
         if args.nobmap:
             log.info("no bmap given, copy entire image to '%s'" % args.dest)
         else:
-            log.error("please, use --nobmap option to flash without bmap")
+            log.error("bmap file not found, please, use --nobmap option to "
+                      "flash without bmap")
             raise SystemExit(1)
     else:
         log.info("block map format version %s" % writer.bmap_version)
diff --git a/bmaptools/TransRead.py b/bmaptools/TransRead.py
index 9bf4e34..171b31f 100644
--- a/bmaptools/TransRead.py
+++ b/bmaptools/TransRead.py
@@ -219,17 +219,11 @@ class TransRead:
     this class are file-like objects which you can read and seek only forward.
     """
 
-    def __init__(self, filepath, local=False, logger=None):
+    def __init__(self, filepath, logger=None):
         """
         Class constructor. The 'filepath' argument is the full path to the file
-        to read transparently. If 'local' is True, then the file-like object is
-        guaranteed to be backed by an uncompressed local file. This means that
-        if the source file is compressed and/or an URL, then it will first be
-        copied to an temporary local file, and then all the subsequent
-        operations will be done with the uncompresed local copy.
-
-        The "logger" argument is the logger object to use for printing
-        messages.
+        to read transparently. The "logger" argument is the logger object to
+        use for printing messages.
         """
 
         self._logger = logger
@@ -274,9 +268,6 @@ class TransRead:
 
         self._open_compressed_file()
 
-        if local and (self.is_url or self.is_compressed):
-            self._create_local_copy()
-
     def __del__(self):
         """The class destructor which closes opened files."""
         for _file_obj in self._f_objs:
@@ -515,28 +506,6 @@ class TransRead:
         self.is_url = True
         self._f_objs.append(f_obj)
 
-    def _create_local_copy(self):
-        """Create a local copy of a remote or compressed file."""
-        import tempfile
-
-        try:
-            tmp_file_obj = tempfile.NamedTemporaryFile("w+")
-        except IOError as err:
-            raise Error("cannot create a temporary file: %s" % err)
-
-        while True:
-            chunk = self.read(1024 * 1024)
-            if not chunk:
-                break
-
-            tmp_file_obj.write(chunk)
-
-        tmp_file_obj.flush()
-
-        self.close()
-        self.__init__(tmp_file_obj.name, local = False)
-        tmp_file_obj.close()
-
     def read(self, size=-1):
         """
         Read the data from the file or URL and and uncompress it on-the-fly if
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:17 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:17 +0300
Subject: [bmap-tools] [PATCH 05/20] BmapCopy: support arbitrary checksum
	types
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-5-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This patch makes BmapCopy support bmap format version 1.4 which allows for
and arbitrary checksum type, not only SHA1. We are, particularly, interested in
SHA256 support.

Change-Id: I9306907ac5d29b0ce524c2abcdbcafac60096958
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py | 45 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 36 insertions(+), 9 deletions(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index 0eb5c40..e209f60 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -172,6 +172,11 @@ class BmapCopy:
         st_data = os.fstat(self._f_dest.fileno())
         self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
 
+        # The bmap file checksum type and length
+        self._cs_type = None
+        self._cs_len = None
+        self._cs_attrib_name = None
+
         # Special quirk for /dev/null which does not support fsync()
         if stat.S_ISCHR(st_data.st_mode) and \
            os.major(st_data.st_rdev) == 1 and \
@@ -242,7 +247,10 @@ class BmapCopy:
 
         import mmap
 
-        correct_chksum = self._xml.find("BmapFileSHA1").text.strip()
+        if self.bmap_version_minor == 3:
+            correct_chksum = self._xml.find("BmapFileSHA1").text.strip()
+        else:
+            correct_chksum = self._xml.find("BmapFileChecksum").text.strip()
 
         # Before verifying the shecksum, we have to substitute the checksum
         # value stored in the file with all zeroes. For these purposes we
@@ -253,8 +261,11 @@ class BmapCopy:
         chksum_pos = mapped_bmap.find(correct_chksum)
         assert chksum_pos != -1
 
-        mapped_bmap[chksum_pos:chksum_pos + 40] = '0' * 40
-        calculated_chksum = hashlib.sha1(mapped_bmap).hexdigest()
+        mapped_bmap[chksum_pos:chksum_pos + self._cs_len] = '0' * self._cs_len
+
+        hash_obj = hashlib.new(self._cs_type)
+        hash_obj.update(mapped_bmap)
+        calculated_chksum = hash_obj.hexdigest()
 
         mapped_bmap.close()
 
@@ -302,7 +313,23 @@ class BmapCopy:
                         % (self.image_size, self.blocks_cnt, self.block_size))
 
         if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
-            # Bmap file checksum appeard in format 1.3
+            # Bmap file checksum appeard in format 1.3 and the only supported
+            # checksum type was SHA1. Version 1.4 started supporting arbitrary
+            # checksum types. A new "ChecksumType" tag was introduce to specify
+            # the checksum function name. And all XML tags which contained
+            # "sha1" in their name were renamed to something more neutral.
+            if self.bmap_version_minor == 3:
+                self._cs_type = "sha1"
+                self._cs_attrib_name = "sha1"
+            else:
+                self._cs_type = xml.find("ChecksumType").text.strip()
+                self._cs_attrib_name = "chksum"
+
+            try:
+                self._cs_len = len(hashlib.new(self._cs_type).hexdigest())
+            except ValueError as err:
+                raise Error("cannot initialize hash function \"%s\": %s" %
+                            (self._cs_type, err))
             self._verify_bmap_checksum()
 
     def _update_progress(self, blocks_written):
@@ -392,8 +419,8 @@ class BmapCopy:
             else:
                 last = first
 
-            if 'sha1' in xml_element.attrib:
-                chksum = xml_element.attrib['sha1']
+            if self._cs_attrib_name in xml_element.attrib:
+                chksum = xml_element.attrib[self._cs_attrib_name]
             else:
                 chksum = None
 
@@ -435,7 +462,7 @@ class BmapCopy:
         try:
             for (first, last, chksum) in self._get_block_ranges():
                 if verify and chksum:
-                    hash_obj = hashlib.new('sha1')
+                    hash_obj = hashlib.new(self._cs_type)
 
                 self._f_image.seek(first * self.block_size)
 
@@ -621,8 +648,8 @@ class BmapBdevCopy(BmapCopy):
 
         # Construct the path to the sysfs directory of our block device
         st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
-        self._sysfs_base = "/sys/dev/block/%s:%s/" \
-                           % (os.major(st_rdev), os.minor(st_rdev))
+        self._sysfs_base = "/sys/dev/block/%s:%s/" % \
+                           (os.major(st_rdev), os.minor(st_rdev))
 
         # Check if the 'queue' sub-directory exists. If yes, then our block
         # device is entire disk. Otherwise, it is a partition, in which case we
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:16 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:16 +0300
Subject: [bmap-tools] [PATCH 04/20] test_api_base: use sha256 instead of sha1
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Well, this does not matter in this case at all, but I am going to switch to
SHA256 in BmapCreate, so changing the tests just for consistency.

Change-Id: I930314a20655da88333b1d6e8e177324794c9ba5
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index a98b9b2..a1c1f02 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -142,7 +142,7 @@ def _calculate_chksum(file_obj):
     """
 
     file_obj.seek(0)
-    hash_obj = hashlib.new("sha1")
+    hash_obj = hashlib.new("sha256")
 
     chunk_size = 1024*1024
 
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:15 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:15 +0300
Subject: [bmap-tools] [PATCH 03/20] bmaptools: preparations to switch to
	sha256
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

In all places where it does not matter which exactly checksum we calculate,
substitute 'sha1' with 'chksum'. This is a preparation for the further change
where we will switch from using SHA1 checksums to using SHA256 checksums.

Change-Id: I51000ad7669d1459c5d6d5e8141e0a6d74f93a56
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py   | 56 ++++++++++++++++++++++++-------------------------
 bmaptools/BmapCreate.py | 28 ++++++++++++-------------
 tests/test_api_base.py  | 34 +++++++++++++++---------------
 3 files changed, 59 insertions(+), 59 deletions(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index 4a9c8cc..0eb5c40 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -99,9 +99,9 @@ class BmapCopy:
     Instead, they are initialized only by the 'copy()' method.
 
     The 'copy()' method implements image copying. You may choose whether to
-    verify the SHA1 checksum while copying or not. Note, this is done only in
-    case of bmap-based copying and only if bmap contains the SHA1 checksums
-    (e.g., bmap version 1.0 did not have SHA1 checksums).
+    verify the checksum while copying or not. Note, this is done only in case
+    of bmap-based copying and only if bmap contains checksums (e.g., bmap
+    version 1.0 did not have checksums support).
 
     You may choose whether to synchronize the destination file after writing or
     not. To explicitly synchronize it, use the 'sync()' method.
@@ -237,31 +237,31 @@ class BmapCopy:
 
     def _verify_bmap_checksum(self):
         """
-        This is a helper function which verifies SHA1 checksum of the bmap file.
+        This is a helper function which verifies the bmap file checksum.
         """
 
         import mmap
 
-        correct_sha1 = self._xml.find("BmapFileSHA1").text.strip()
+        correct_chksum = self._xml.find("BmapFileSHA1").text.strip()
 
-        # Before verifying the shecksum, we have to substitute the SHA1 value
-        # stored in the file with all zeroes. For these purposes we create
-        # private memory mapping of the bmap file.
+        # Before verifying the shecksum, we have to substitute the checksum
+        # value stored in the file with all zeroes. For these purposes we
+        # create private memory mapping of the bmap file.
         mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
                                 access = mmap.ACCESS_COPY)
 
-        sha1_pos = mapped_bmap.find(correct_sha1)
-        assert sha1_pos != -1
+        chksum_pos = mapped_bmap.find(correct_chksum)
+        assert chksum_pos != -1
 
-        mapped_bmap[sha1_pos:sha1_pos + 40] = '0' * 40
-        calculated_sha1 = hashlib.sha1(mapped_bmap).hexdigest()
+        mapped_bmap[chksum_pos:chksum_pos + 40] = '0' * 40
+        calculated_chksum = hashlib.sha1(mapped_bmap).hexdigest()
 
         mapped_bmap.close()
 
-        if calculated_sha1 != correct_sha1:
+        if calculated_chksum != correct_chksum:
             raise Error("checksum mismatch for bmap file '%s': calculated "
                         "'%s', should be '%s'"
-                        % (self._bmap_path, calculated_sha1, correct_sha1))
+                        % (self._bmap_path, calculated_chksum, correct_chksum))
 
     def _parse_bmap(self):
         """
@@ -348,12 +348,12 @@ class BmapCopy:
     def _get_block_ranges(self):
         """
         This is a helper generator that parses the bmap XML file and for each
-        block range in the XML file it yields ('first', 'last', 'sha1') tuples,
-        where:
+        block range in the XML file it yields ('first', 'last', 'chksum')
+        tuples, where:
           * 'first' is the first block of the range;
           * 'last' is the last block of the range;
-          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
-            missing.
+          * 'chksum' is the checksum of the range ('None' is used if it is
+            missing).
 
         If there is no bmap file, the generator just yields a single range
         for entire image file. If the image size is unknown, the generator
@@ -393,11 +393,11 @@ class BmapCopy:
                 last = first
 
             if 'sha1' in xml_element.attrib:
-                sha1 = xml_element.attrib['sha1']
+                chksum = xml_element.attrib['sha1']
             else:
-                sha1 = None
+                chksum = None
 
-            yield (first, last, sha1)
+            yield (first, last, chksum)
 
     def _get_batches(self, first, last):
         """
@@ -433,8 +433,8 @@ class BmapCopy:
         """
 
         try:
-            for (first, last, sha1) in self._get_block_ranges():
-                if verify and sha1:
+            for (first, last, chksum) in self._get_block_ranges():
+                if verify and chksum:
                     hash_obj = hashlib.new('sha1')
 
                 self._f_image.seek(first * self.block_size)
@@ -452,18 +452,18 @@ class BmapCopy:
                         self._batch_queue.put(None)
                         return
 
-                    if verify and sha1:
+                    if verify and chksum:
                         hash_obj.update(buf)
 
                     blocks = (len(buf) + self.block_size - 1) / self.block_size
                     self._batch_queue.put(("range", start, start + blocks - 1,
                                            buf))
 
-                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                if verify and chksum and hash_obj.hexdigest() != chksum:
                     raise Error("checksum mismatch for blocks range %d-%d: "
                                 "calculated %s, should be %s (image file %s)"
                                 % (first, last, hash_obj.hexdigest(),
-                                   sha1, self._image_path))
+                                   chksum, self._image_path))
         # Silence pylint warning about catching too general exception
         # pylint: disable=W0703
         except Exception:
@@ -478,8 +478,8 @@ class BmapCopy:
         """
         Copy the image to the destination file using bmap. The 'sync' argument
         defines whether the destination file has to be synchronized upon
-        return.  The 'verify' argument defines whether the SHA1 checksum has to
-        be verified while copying.
+        return.  The 'verify' argument defines whether the checksum has to be
+        verified while copying.
         """
 
         # Create the queue for block batches and start the reader thread, which
diff --git a/bmaptools/BmapCreate.py b/bmaptools/BmapCreate.py
index b96224a..2017a39 100644
--- a/bmaptools/BmapCreate.py
+++ b/bmaptools/BmapCreate.py
@@ -126,7 +126,7 @@ class BmapCreate:
 
         self._mapped_count_pos1 = None
         self._mapped_count_pos2 = None
-        self._sha1_pos = None
+        self._chksum_pos = None
 
         self._f_image_needs_close = False
         self._f_bmap_needs_close = False
@@ -218,7 +218,7 @@ class BmapCreate:
         xml += "    <BmapFileSHA1> "
 
         self._f_bmap.write(xml)
-        self._sha1_pos = self._f_bmap.tell()
+        self._chksum_pos = self._f_bmap.tell()
 
         xml = "0" * 40 + " </BmapFileSHA1>\n\n"
         xml += "    <!-- The block map which consists of elements which may either be a\n"
@@ -249,14 +249,14 @@ class BmapCreate:
         self._f_bmap.write("%u" % self.mapped_cnt)
 
         self._f_bmap.seek(0)
-        sha1 = hashlib.sha1(self._f_bmap.read()).hexdigest()
-        self._f_bmap.seek(self._sha1_pos)
-        self._f_bmap.write("%s" % sha1)
+        chksum = hashlib.sha1(self._f_bmap.read()).hexdigest()
+        self._f_bmap.seek(self._chksum_pos)
+        self._f_bmap.write("%s" % chksum)
 
-    def _calculate_sha1(self, first, last):
+    def _calculate_chksum(self, first, last):
         """
-        A helper function which calculates SHA1 checksum for the range of
-        blocks of the image file: from block 'first' to block 'last'.
+        A helper function which calculates checksum for the range of blocks of
+        the image file: from block 'first' to block 'last'.
         """
 
         start = first * self.block_size
@@ -281,7 +281,7 @@ class BmapCreate:
     def generate(self, include_checksums=True):
         """
         Generate bmap for the image file. If 'include_checksums' is 'True',
-        also generate SHA1 checksums for block ranges.
+        also generate checksums for block ranges.
         """
 
         # Save image file position in order to restore it at the end
@@ -295,17 +295,17 @@ class BmapCreate:
         for first, last in self.fiemap.get_mapped_ranges(0, self.blocks_cnt):
             self.mapped_cnt += last - first + 1
             if include_checksums:
-                sha1 = self._calculate_sha1(first, last)
-                sha1 = " sha1=\"%s\"" % sha1
+                chksum = self._calculate_chksum(first, last)
+                chksum = " sha1=\"%s\"" % chksum
             else:
-                sha1 = ""
+                chksum = ""
 
             if first != last:
                 self._f_bmap.write("        <Range%s> %s-%s </Range>\n"
-                                   % (sha1, first, last))
+                                   % (chksum, first, last))
             else:
                 self._f_bmap.write("        <Range%s> %s </Range>\n"
-                                   % (sha1, first))
+                                   % (chksum, first))
 
         self.mapped_size = self.mapped_cnt * self.block_size
         self.mapped_size_human = human_size(self.mapped_size)
diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index e814fa0..a98b9b2 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -136,9 +136,9 @@ def _generate_compressed_files(file_obj, delete=True):
     yield tbz2_file_obj.name
     tmp_file_obj.close()
 
-def _calculate_sha1(file_obj):
+def _calculate_chksum(file_obj):
     """
-    Calculates SHA1 checksum for the contents of file object 'file_obj'.
+    Calculates checksum for the contents of file object 'file_obj'.
     """
 
     file_obj.seek(0)
@@ -154,7 +154,7 @@ def _calculate_sha1(file_obj):
 
     return hash_obj.hexdigest()
 
-def _copy_image(image, f_dest, f_bmap, image_sha1, image_size):
+def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
     """
     Copy image 'image' using bmap 'f_bmap' to the destination file 'f_dest'.
     """
@@ -177,7 +177,7 @@ def _copy_image(image, f_dest, f_bmap, image_sha1, image_size):
 
     # Compare the original file and the copy are identical
     f_dest.seek(0)
-    assert _calculate_sha1(f_dest) == image_sha1
+    assert _calculate_chksum(f_dest) == image_chksum
 
     if not hasattr(image, "read"):
         f_image.close()
@@ -213,7 +213,7 @@ def _do_test(f_image, image_size, delete=True):
                                           delete=delete, dir=directory,
                                           suffix=".bmap2")
 
-    image_sha1 = _calculate_sha1(f_image)
+    image_chksum = _calculate_chksum(f_image)
 
     #
     # Pass 1: generate the bmap, copy and compare
@@ -223,7 +223,7 @@ def _do_test(f_image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(f_image.name, f_bmap1.name)
     creator.generate()
 
-    _copy_image(f_image, f_copy, f_bmap1, image_sha1, image_size)
+    _copy_image(f_image, f_copy, f_bmap1, image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -235,7 +235,7 @@ def _do_test(f_image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(f_image, f_bmap2)
     creator.generate()
-    _copy_image(f_image, f_copy, f_bmap2, image_sha1, image_size)
+    _copy_image(f_image, f_copy, f_bmap2, image_chksum, image_size)
     _compare_holes(f_image, f_copy)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
@@ -246,36 +246,36 @@ def _do_test(f_image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(f_image, delete=delete):
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, image_size)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, None)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
         compressed = "file:" + compressed
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, image_size)
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, None)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
 
     #
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_image(f_image, f_copy, None, image_sha1, image_size)
-    _copy_image(f_image, f_copy, None, image_sha1, None)
+    _copy_image(f_image, f_copy, None, image_chksum, image_size)
+    _copy_image(f_image, f_copy, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
     #
 
     for compressed in _generate_compressed_files(f_image, delete=delete):
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, image_size)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, None)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, image_size)
-        _copy_image(compressed, f_copy, f_bmap1, image_sha1, None)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
 
     # Close temporary files, which will also remove them
     f_copy.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:19 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:19 +0300
Subject: [bmap-tools] [PATCH 07/20] TODO: remove sha256 entry
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-7-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I5e36f03362ad3706501fbbd1c6e2dd78893b55e7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 1 -
 1 file changed, 1 deletion(-)

diff --git a/TODO b/TODO
index 38431be..2a10bd5 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,5 @@
 Current TODO list, any help with these is appreciated.
 
-1. Switch from SHA1 to SHA256
 2. Implement GPG signatures support, just like this is done in Fedora
    https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
 3. In setup.py, fetch the version from 'bmaptool' file.
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:18 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:18 +0300
Subject: [bmap-tools] [PATCH 06/20] BmapCreate: implement arbitrary hash
	type support
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-6-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Instead of just supporting SHA1 hash functions type, support arbitrary hash
function type. Well, actually "arbitrary" means any hash function supported by
the "hashlib" python module.

In particular, we are interested to switch from SHA1 to SHA256.

Unfortunately, this is a format change, since format version 1.3 guaranteed
that checksum is always SHA1. This is why this patch also increases the format
version to 1.4.

Change-Id: I096b47d6e4a413368d42e40ce8ebb06f612619c5
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCreate.py | 50 ++++++++++++++++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 15 deletions(-)

diff --git a/bmaptools/BmapCreate.py b/bmaptools/BmapCreate.py
index 2017a39..5dc46b1 100644
--- a/bmaptools/BmapCreate.py
+++ b/bmaptools/BmapCreate.py
@@ -46,8 +46,14 @@ import hashlib
 from bmaptools.BmapHelpers import human_size
 from bmaptools import Fiemap
 
-# The bmap format version we generate
-SUPPORTED_BMAP_VERSION = "1.3"
+# The bmap format version we generate.
+#
+# Changelog:
+# o 1.3 -> 1.4:
+#   Support SHA256 and SHA512 checksums, in 1.3 only SHA1 was supported.
+#   "BmapFileChecksum" is used instead of "BmapFileSHA1", and "chksum="
+#   attribute is used instead "sha1=". Introduced "ChecksumType" tag.
+SUPPORTED_BMAP_VERSION = "1.4"
 
 _BMAP_START_TEMPLATE = \
 """<?xml version="1.0" ?>
@@ -106,13 +112,15 @@ class BmapCreate:
     the FIEMAP ioctl to generate the bmap.
     """
 
-    def __init__(self, image, bmap):
+    def __init__(self, image, bmap, chksum_type="sha1"):
         """
         Initialize a class instance:
-        * image - full path or a file-like object of the image to create bmap
-                  for
-        * bmap  - full path or a file object to use for writing the resulting
-                  bmap to
+        * image  - full path or a file-like object of the image to create bmap
+                   for
+        * bmap   - full path or a file object to use for writing the resulting
+                   bmap to
+        * chksum - type of the check sum to use in the bmap file (all checksum
+                   types which python's "hashlib" module supports are allowed).
         """
 
         self.image_size = None
@@ -131,6 +139,13 @@ class BmapCreate:
         self._f_image_needs_close = False
         self._f_bmap_needs_close = False
 
+        self._cs_type = chksum_type.lower()
+        try:
+            self._cs_len = len(hashlib.new(self._cs_type).hexdigest())
+        except ValueError as err:
+            raise Error("cannot initialize hash function \"%s\": %s" %
+                        (self._cs_type, err))
+
         if hasattr(image, "read"):
             self._f_image = image
             self._image_path = image.name
@@ -213,17 +228,20 @@ class BmapCreate:
         xml  = "%s </MappedBlocksCount>\n\n" % mapped_count
 
         # pylint: disable=C0301
+        xml += "    <!-- Type of checksum used in this file. -->\n"
+        xml += "    <ChecksumType> %s </ChecksumType>\n\n" % self._cs_type
+
         xml += "    <!-- The checksum of this bmap file. When it is calculated, the value of\n"
-        xml += "         the SHA1 checksum has be zero (40 ASCII \"0\" symbols). -->\n"
-        xml += "    <BmapFileSHA1> "
+        xml += "         the checksum has be zero (all ASCII \"0\" symbols).  -->\n"
+        xml += "    <BmapFileChecksum> "
 
         self._f_bmap.write(xml)
         self._chksum_pos = self._f_bmap.tell()
 
-        xml = "0" * 40 + " </BmapFileSHA1>\n\n"
+        xml = "0" * self._cs_len + " </BmapFileChecksum>\n\n"
         xml += "    <!-- The block map which consists of elements which may either be a\n"
-        xml += "         range of blocks or a single block. The 'sha1' attribute (if present)\n"
-        xml += "         is the SHA1 checksum of this blocks range. -->\n"
+        xml += "         range of blocks or a single block. The 'chksum' attribute\n"
+        xml += "         (if present) is the checksum of this blocks range. -->\n"
         xml += "    <BlockMap>\n"
         # pylint: enable=C0301
 
@@ -249,7 +267,9 @@ class BmapCreate:
         self._f_bmap.write("%u" % self.mapped_cnt)
 
         self._f_bmap.seek(0)
-        chksum = hashlib.sha1(self._f_bmap.read()).hexdigest()
+        hash_obj = hashlib.new(self._cs_type)
+        hash_obj.update(self._f_bmap.read())
+        chksum = hash_obj.hexdigest()
         self._f_bmap.seek(self._chksum_pos)
         self._f_bmap.write("%s" % chksum)
 
@@ -263,7 +283,7 @@ class BmapCreate:
         end = (last + 1) * self.block_size
 
         self._f_image.seek(start)
-        hash_obj = hashlib.new("sha1")
+        hash_obj = hashlib.new(self._cs_type)
 
         chunk_size = 1024*1024
         to_read = end - start
@@ -296,7 +316,7 @@ class BmapCreate:
             self.mapped_cnt += last - first + 1
             if include_checksums:
                 chksum = self._calculate_chksum(first, last)
-                chksum = " sha1=\"%s\"" % chksum
+                chksum = " chksum=\"%s\"" % chksum
             else:
                 chksum = ""
 
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:21 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:21 +0300
Subject: [bmap-tools] [PATCH 09/20] Fiemap: fix failure for zero-sized files
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-9-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

The Fiemap module fails for zero-sized file with these symptoms:

Traceback (most recent call last):
  File "./bmaptool", line 561, in <module>
    sys.exit(main())
  File "./bmaptool", line 545, in main
    args.func(args, log)
  File "./bmaptool", line 419, in create_command
    creator = BmapCreate.BmapCreate(args.image, output, "sha256")
  File "/mnt/bigssd/dedekind/work/tizen/git/bmap-tools/bmaptools/BmapCreate.py", line 163, in __init__
    self.fiemap = Fiemap.Fiemap(self._f_image)
  File "/mnt/bigssd/dedekind/work/tizen/git/bmap-tools/bmaptools/Fiemap.py", line 120, in __init__
    self.block_is_mapped(0)
  File "/mnt/bigssd/dedekind/work/tizen/git/bmap-tools/bmaptools/Fiemap.py", line 175, in block_is_mapped
    struct_fiemap = self._invoke_fiemap(block, 1)
  File "/mnt/bigssd/dedekind/work/tizen/git/bmap-tools/bmaptools/Fiemap.py", line 149, in _invoke_fiemap
    % (block, self.blocks_cnt))

This patch fixes the issue.

Change-Id: I3eb2c6e69c892aea2a3d41f2b9bf830907d7658e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/Fiemap.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bmaptools/Fiemap.py b/bmaptools/Fiemap.py
index 3066f18..9413358 100644
--- a/bmaptools/Fiemap.py
+++ b/bmaptools/Fiemap.py
@@ -144,7 +144,7 @@ class Fiemap:
         list (just like 'struct.upack()').
         """
 
-        if block < 0 or block >= self.blocks_cnt:
+        if self.blocks_cnt != 0 and (block < 0 or block >= self.blocks_cnt):
             raise Error("bad block number %d, should be within [0, %d]"
                         % (block, self.blocks_cnt))
 
-- 
1.8.1.4



From dedekind1 at gmail.com  Wed Sep 18 03:38:20 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Wed, 18 Sep 2013 10:38:20 +0300
Subject: [bmap-tools] [PATCH 08/20] bmaptool: start using sha256 when
	creating bmap file
In-Reply-To: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
References: <1379489901-25083-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379489901-25083-8-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Switch from using SHA1 to SHA256 for the 'bmaptool create' command.

Basically, this is what the Fedora community requested, with a reference to
this:

http://csrc.nist.gov/groups/ST/hash/policy_2006.html

Change-Id: I3facbd8d5588903b1d7d28eac77d147d0390a1c4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bmaptool b/bmaptool
index bcc0b6b..6236810 100755
--- a/bmaptool
+++ b/bmaptool
@@ -313,7 +313,7 @@ def create_command(args, log):
             raise SystemExit(1)
 
     try:
-        creator = BmapCreate.BmapCreate(args.image, output)
+        creator = BmapCreate.BmapCreate(args.image, output, "sha256")
         creator.generate(not args.no_checksum)
     except BmapCreate.Error as err:
         log.error(str(err))
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:14 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:14 +0300
Subject: [bmap-tools] [PATCH 01/14] bmaptool: do not fail when copying a
	bmap file
Message-ID: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

It is possible that the user copies the bmapfile itself using 'bmaptool copy'.
Most probably this is just by a mistake. In this case bmaptool just fails,
since it discovers the bmap file, and it does not match the "image" (where the
image is the same bmap file).

This patch teaches bmaptool to handle this case by dropping the discoverd bmap
file if its path is the same as the image path.

Change-Id: Ib48842ed6157ba364173fcd7771e1b2ba3eea9e5
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/bmaptool b/bmaptool
index 6236810..95380e8 100755
--- a/bmaptool
+++ b/bmaptool
@@ -181,6 +181,15 @@ def copy_command_open_all(args, log):
     # was not specified.
     (bmap_obj, bmap_path) = find_and_open_bmap(args, log)
 
+    if bmap_path == args.image:
+        # Most probably the specified the bmap file instead of the image file
+        # by mistake.
+        log.warning("image has the same path as the bmap file, dropping bmap")
+        bmap_obj.close()
+        bmap_obj = None
+        bmap_path = None
+        args.nobmap = True
+
     # Try to open the destination file. If it does not exist, a new regular
     # file will be created. If it exists and it is a regular file - it'll be
     # truncated. If this is a block device, it'll just be opened.
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:16 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:16 +0300
Subject: [bmap-tools] [PATCH 03/14] bmaptool: disallow simultaneous --bmap
	and --nobmap
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Do not allow users to use --bmap and --nobmap at the same time since it makes
no sense.

Change-Id: I69e02c1e885f7877e6b6e1dd5218bab07f1c1f24
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/bmaptool b/bmaptool
index 6f3534a..77fee1c 100755
--- a/bmaptool
+++ b/bmaptool
@@ -221,6 +221,11 @@ def open_files(args, log):
 
 def copy_command(args, log):
     """Copy an image to a block device or a regular file using bmap."""
+
+    if args.nobmap and args.bmap:
+        log.error("--nobmap and --bmap cannot be used together")
+        raise SystemExit(1)
+
     image_obj, dest_obj, bmap_obj, bmap_path, image_size, dest_is_blkdev = \
                                                    open_files(args, log)
     try:
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:18 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:18 +0300
Subject: [bmap-tools] [PATCH 05/14] TODO: update the list
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-5-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Remove the item about GPG signatures support, add a reminder about updating the
man pages with the GPG and hash functions information.

Change-Id: Id4a0d0cb7ec907dfbebe9090d8b85ad7138e7bbf
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/TODO b/TODO
index 2a10bd5..c1d33ed 100644
--- a/TODO
+++ b/TODO
@@ -1,7 +1,5 @@
 Current TODO list, any help with these is appreciated.
 
-2. Implement GPG signatures support, just like this is done in Fedora
-   https://fedoraproject.org/static/checksums/Fedora-19-i386-CHECKSUM
 3. In setup.py, fetch the version from 'bmaptool' file.
 4. Teach the 'make_a_release.sh' script to update the version and the
    changelog.
@@ -12,3 +10,4 @@ Current TODO list, any help with these is appreciated.
 7. Verify that unmapped areas have all zeroes to guarantee integrity.
 8. Update the man pages
 9. Teach bmaptool to update the alternate GPT partition
+10. Document GPG singatures support and supported hash function types.
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:15 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:15 +0300
Subject: [bmap-tools] [PATCH 02/14] bmaptool: cleanups
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This function improves the commentaries and renames several functions. The
renames are about removing the 'copy_command_' prefix from function names,
because it is not very readable and we do not use it consistently anyway.

Change-Id: Id8030489820cb3d9b03830b7209d30a4c0452f23
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool | 43 ++++++++++++++++++++++++++-----------------
 1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/bmaptool b/bmaptool
index 95380e8..6f3534a 100755
--- a/bmaptool
+++ b/bmaptool
@@ -59,10 +59,18 @@ import traceback
 import shutil
 from bmaptools import BmapCreate, BmapCopy, BmapHelpers, TransRead
 
-def copy_command_open_blkdev(path, log):
+def open_block_device(path, log):
     """
-    Open a block device specified by 'path' in exclusive mode. Returns opened
-    file object.
+    This is a helper function for 'open_files()' which is called if the
+    destination file of the "copy" command is a block device. We handle block
+    devices a little bit different to regular files. Namely, we are trying to
+    make sure that we do not write to a mounted block device, otherwise the
+    user could corrupt, say, the root file system by a mistake. This is
+    achieved by opening the block device in exclusive mode, which guarantees
+    that we are the only users of the block device.
+
+    This function opens a block device specified by 'path' in exclusive mode.
+    Returns opened file object.
     """
 
     class NamedFile:
@@ -98,18 +106,21 @@ def copy_command_open_blkdev(path, log):
 
 def find_and_open_bmap(args, log):
     """
-    This function opens the bmap file and returns the corresponding file object
-    as the bmap file path.
+    This is a helper function for 'open_files()' which discovers and opens the
+    bmap file, then returns the corresponding file object and the bmap file
+    path.
 
     If the user specified the bmap file explicitely, we just open the provided
-    path. Otherwise,  we try to find the bmap file at the same place where the
-    image file is located. We search for a file with the same path and
-    basename, but with a ".bmap" extension. Since the image may contain more
-    than one extension (e.g., image.raw.bz2), we try to remove them one-by-one.
+    path. Otherwise, we try to discover the bmap file at the same place where
+    the image file is located. We search for a file with the same path and
+    basename, but with a ".bmap" extension.
 
     Additionally, this function makes sure that the returned file object
     corresponds to a local file, not a remote file. We do this by creating a
-    temporary local copy of the bmap file, if needed.
+    temporary local copy of the bmap file. The reason is that further on we may
+    need to check the GPG signature of the file, which requires it to be a
+    local file. On top of that, the BmapCopy class requires the bmap file to be
+    memory-mappable ('mmap()').
     """
 
     if args.nobmap:
@@ -141,8 +152,6 @@ def find_and_open_bmap(args, log):
     if not bmap_obj.is_compressed and not bmap_obj.is_url:
         return (bmap_obj, bmap_path)
 
-    # Current implementation of BmapCopy requires a local file for block map,
-    # because it memory-maps it ('mmap()').
     try:
         # Create a temporary file for the bmap
         tmp_obj = tempfile.TemporaryFile("w+")
@@ -157,10 +166,10 @@ def find_and_open_bmap(args, log):
     bmap_obj.close()
     return (tmp_obj, bmap_path)
 
-def copy_command_open_all(args, log):
+def open_files(args, log):
     """
-    Open the image/bmap/destination files for the "copy" command. Returns a
-    tuple of 5 elements:
+    This is a helper function for 'copy_command()' which the image, bmap, and
+    the destination files. Returns a tuple of 5 elements:
         1 file-like object for the image
         2 file object for the destination file
         3 file-like object for the bmap
@@ -204,7 +213,7 @@ def copy_command_open_all(args, log):
     dest_is_blkdev = stat.S_ISBLK(os.fstat(dest_obj.fileno()).st_mode)
     if dest_is_blkdev:
         dest_obj.close()
-        dest_obj = copy_command_open_blkdev(args.dest, log)
+        dest_obj = open_block_device(args.dest, log)
 
     return (image_obj, dest_obj, bmap_obj, bmap_path, image_obj.size,
             dest_is_blkdev)
@@ -213,7 +222,7 @@ def copy_command_open_all(args, log):
 def copy_command(args, log):
     """Copy an image to a block device or a regular file using bmap."""
     image_obj, dest_obj, bmap_obj, bmap_path, image_size, dest_is_blkdev = \
-                                          copy_command_open_all(args, log)
+                                                   open_files(args, log)
     try:
         if dest_is_blkdev:
             dest_str = "block device '%s'" % args.dest
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:20 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:20 +0300
Subject: [bmap-tools] [PATCH 07/14] TODO: remove an entry about checking
	zeroes
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-7-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

I think I will not implement this because it makes little sense. Drop the
unmapped areas anyway, why would we care checking their contents? What for?

Change-Id: I756c073ffef65715cf418a8f3f37bf35eef86218
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/TODO b/TODO
index b1925bf..dd622bc 100644
--- a/TODO
+++ b/TODO
@@ -6,7 +6,8 @@ Current TODO list, any help with these is appreciated.
 6. Write in the man page, docs, and may be --help that it is very important
    that the image is not changed while we create the bmep file, and after we
    have created it too.
-7. Verify that unmapped areas have all zeroes to guarantee integrity.
 8. Update the man pages
 9. Teach bmaptool to update the alternate GPT partition
 10. Document GPG singatures support and supported hash function types
+11. Write in the doc that bmaptool does not verify that unmapped areas do
+    contain zeroes.
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:19 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:19 +0300
Subject: [bmap-tools] [PATCH 06/14] Fiemap: synchronize the file before
	invoking the ioctl
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-6-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Early FIEMAP implementations had many bugs related to cached dirty data. And
this is why it is safer to synchronize the file before invoking FIEMAP for it.
Let's start using the 'FIEMAP_FLAG_SYNC' FIEMAP ioctl flag which does exactly
that.

Change-Id: I1698b88ed3978ffa632502ba72ad345ec8708ce0
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO                |  3 +--
 bmaptools/Fiemap.py | 14 ++++++++++++--
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/TODO b/TODO
index c1d33ed..b1925bf 100644
--- a/TODO
+++ b/TODO
@@ -3,11 +3,10 @@ Current TODO list, any help with these is appreciated.
 3. In setup.py, fetch the version from 'bmaptool' file.
 4. Teach the 'make_a_release.sh' script to update the version and the
    changelog.
-5. Fsync before creating the bmap file (probably use FIEMAP_FLAG_SYNC flag)
 6. Write in the man page, docs, and may be --help that it is very important
    that the image is not changed while we create the bmep file, and after we
    have created it too.
 7. Verify that unmapped areas have all zeroes to guarantee integrity.
 8. Update the man pages
 9. Teach bmaptool to update the alternate GPT partition
-10. Document GPG singatures support and supported hash function types.
+10. Document GPG singatures support and supported hash function types
diff --git a/bmaptools/Fiemap.py b/bmaptools/Fiemap.py
index 9413358..b5f7c2c 100644
--- a/bmaptools/Fiemap.py
+++ b/bmaptools/Fiemap.py
@@ -39,6 +39,9 @@ _FIEMAP_EXTENT_FORMAT = "=QQQQQLLLL"
 _FIEMAP_EXTENT_SIZE = struct.calcsize(_FIEMAP_EXTENT_FORMAT)
 # The FIEMAP ioctl number
 _FIEMAP_IOCTL = 0xC020660B
+# This FIEMAP ioctl flag which instructs the kernel to sync the file before
+# reading the block map
+_FIEMAP_FLAG_SYNC = 0x00000001
 
 # Minimum buffer which is required for 'class Fiemap' to operate
 MIN_BUFFER_SIZE = _FIEMAP_SIZE + _FIEMAP_EXTENT_SIZE
@@ -68,6 +71,9 @@ class Fiemap:
         fiemap_extent' elements which will be used when invoking the FIEMAP
         ioctl. The larger is the buffer, the less times the FIEMAP ioctl will
         be invoked.
+
+        This class synchronizes the image file every time it invokes the FIEMAP
+        ioctl in order to work-around early FIEMAP implementation kernel bugs.
         """
 
         self._f_image_needs_close = False
@@ -148,9 +154,13 @@ class Fiemap:
             raise Error("bad block number %d, should be within [0, %d]"
                         % (block, self.blocks_cnt))
 
-        # Initialize the 'struct fiemap' part of the buffer
+        # Initialize the 'struct fiemap' part of the buffer. We use the
+        # '_FIEMAP_FLAG_SYNC' flag in order to make sure the file is
+        # synchronized. The reason for this is that early FIEMAP
+        # implementations had many bugs related to cached dirty data, and
+        # synchronizing the file is a necessary work-around.
         struct.pack_into(_FIEMAP_FORMAT, self._buf, 0, block * self.block_size,
-                         count * self.block_size, 0, 0,
+                         count * self.block_size, _FIEMAP_FLAG_SYNC, 0,
                          self._fiemap_extent_cnt, 0)
 
         try:
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:21 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:21 +0300
Subject: [bmap-tools] [PATCH 08/14] setup.py: automatically detect version
	number
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-8-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

We currently duplicate the project version number in bmaptool and setup.py
files. Let's get rid of the duplication and have setup.py parse the bmaptool
file and fetch the version number.

Change-Id: I06ee68ca29b16fac2cbfa53cd9a61329b8606409
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO     |  1 -
 setup.py | 14 +++++++++++++-
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/TODO b/TODO
index dd622bc..7eacd8d 100644
--- a/TODO
+++ b/TODO
@@ -1,6 +1,5 @@
 Current TODO list, any help with these is appreciated.
 
-3. In setup.py, fetch the version from 'bmaptool' file.
 4. Teach the 'make_a_release.sh' script to update the version and the
    changelog.
 6. Write in the man page, docs, and may be --help that it is very important
diff --git a/setup.py b/setup.py
index cc864a9..1d38bf7 100644
--- a/setup.py
+++ b/setup.py
@@ -1,13 +1,25 @@
 #!/usr/bin/env python
 
+import re
 from setuptools import setup, find_packages
 
+def get_version():
+    """Fetch the project version number from the 'bmaptool' file."""
+
+    with open("bmaptool", "r") as fobj:
+        for line in fobj:
+            matchobj = re.match(r'^VERSION = "(\d+.\d+)"$', line)
+            if matchobj:
+                return matchobj.group(1)
+
+    return None
+
 setup(
     name="bmap-tools",
     description="Bmap tools",
     author="Artem Bityutskiy",
     author_email="artem.bityutskiy at linux.intel.com",
-    version="2.6",
+    version=get_version(),
     scripts=['bmaptool'],
     packages=find_packages(exclude=["test*"]),
     license='GPLv2',
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:17 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:17 +0300
Subject: [bmap-tools] [PATCH 04/14] bmaptool: implement GPG signature
	verification
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a feature which was requested by the Fedora communitiy.

Both clearsign and detached GPG signatures are supported. The signature file
auto-discovery is supported (similar to bmap file auto-discovery).

More user-facing information will be added to the man page a bit later.

We add the following command-line options:
    1. --bmap-sig option for specifying the detached signature file path
    2. --no-sig-verify option for making bmaptool avoid verifying the clearsign
       signature and avoid signature file auto-discovery

Change-Id: Id64194415759abdbbe4e2f54d997f0855f67c749
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptool | 207 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 205 insertions(+), 2 deletions(-)

diff --git a/bmaptool b/bmaptool
index 77fee1c..b0eb409 100755
--- a/bmaptool
+++ b/bmaptool
@@ -57,6 +57,7 @@ import logging
 import tempfile
 import traceback
 import shutil
+import io
 from bmaptools import BmapCreate, BmapCopy, BmapHelpers, TransRead
 
 def open_block_device(path, log):
@@ -104,13 +105,192 @@ def open_block_device(path, log):
 
     return NamedFile(file_obj, path)
 
+def report_verification_results(context, sigs, log):
+    """
+    This is a helper function which reports the GPG signature verification
+    results. The 'context' argument is the gpgme context object, and the 'sigs'
+    argument contains the results of the 'gpgme.verify()' function.
+    """
+
+    for sig in sigs:
+        if not sig.status:
+            key = context.get_key(sig.fpr)
+            author = "%s <%s>" % (key.uids[0].name, key.uids[0].email)
+            log.info("successfully verified bmap file signature of %s "
+                     "(fingerprint %s)" % (author, sig.fpr))
+        else:
+            log.error("signature verification failed (fingerprint %s): "
+                      "%s" % (sig.fpr, sig.status[2].lower()))
+            log.error("either fix the problem or use --no-sig-verify"
+                      "to disable signature verification")
+            raise SystemExit(1)
+
+def verify_detached_bmap_signature(args, bmap_obj, bmap_path, log):
+    """
+    This is a helper function for 'verify_bmap_signature()' which handles the
+    detached signature case.
+    """
+
+    if args.no_sig_verify:
+        return None
+
+    if args.bmap_sig:
+        try:
+            sig_obj = TransRead.TransRead(args.bmap_sig, logger=log)
+        except TransRead.Error as err:
+            log.error("cannot open bmap signature file '%s': %s" %
+                      (args.bmap_sig, str(err)))
+            raise SystemExit(1)
+        sig_path = args.bmap_sig
+    else:
+        # Check if there is a stand-alone signature file
+        try:
+            sig_path = bmap_path + ".asc"
+            sig_obj = TransRead.TransRead(sig_path, logger=log)
+        except TransRead.Error:
+            try:
+                sig_path = bmap_path + ".sig"
+                sig_obj = TransRead.TransRead(sig_path, logger=log)
+            except TransRead.Error:
+                # No signatures found
+                return None
+
+        log.info("discovered signature file for bmap '%s'" % sig_path)
+
+    # If the stand-alone signature file is not local, make a local copy
+    if sig_obj.is_compressed or sig_obj.is_url:
+        try:
+            tmp_obj = tempfile.NamedTemporaryFile("w+")
+        except IOError as err:
+            log.error("cannot create a temporary file for the "
+                      "signature: %s" % err)
+            raise SystemExit(1)
+
+        shutil.copyfileobj(sig_obj, tmp_obj)
+        tmp_obj.seek(0)
+        sig_obj.close()
+        sig_obj = tmp_obj
+
+    try:
+        import gpgme
+    except ImportError:
+        log.error("cannot verify the signature because the python \"gpgme\""
+                  "module is not installed on your system")
+        log.error("please, either install the module or use --no-sig-verify")
+        raise SystemExit(1)
+
+    try:
+        context = gpgme.Context()
+        signature = io.FileIO(sig_obj.name)
+        signed_data = io.FileIO(bmap_obj.name)
+        sigs = context.verify(signature, signed_data, None)
+    except gpgme.GpgmeError as err:
+        log.error("failure when trying to verify GPG signature: %s" %
+                  err[2].lower())
+        log.error("make sure file \"%s\" has proper GPG format" % sig_path)
+        raise SystemExit(1)
+
+    sig_obj.close()
+
+    if len(sigs) == 0:
+        log.warning("the \"%s\" signature file does not actually contain "
+                    "any valid signatures" % sig_path)
+    else:
+        report_verification_results(context, sigs, log)
+
+    return None
+
+def verify_clearsign_bmap_signature(args, bmap_obj, log):
+    """
+    This is a helper function for 'verify_bmap_signature()' which handles the
+    clarsign signature case.
+    """
+
+    if args.bmap_sig:
+        log.error("the bmap file has clearsign format and already contains "
+                  "the signature, so --bmap-sig option should not be used")
+        raise SystemExit(1)
+
+    try:
+        import gpgme
+    except ImportError:
+        log.error("cannot verify the signature because the python \"gpgme\""
+                  "module is not installed on your system")
+        log.error("cannot extract block map from the bmap file which has "
+                  "clearsign format, please, install the module")
+        raise SystemExit(1)
+
+    try:
+        context = gpgme.Context()
+        signature = io.FileIO(bmap_obj.name)
+        plaintext = io.BytesIO()
+        sigs = context.verify(signature, None, plaintext)
+    except gpgme.GpgmeError as err:
+        log.error("failure when trying to verify GPG signature: %s" %
+                  err[2].lower())
+        log.error("make sure the bmap file has proper GPG format ")
+        raise SystemExit(1)
+
+    if not args.no_sig_verify:
+        if len(sigs) == 0:
+            log.warning("the bmap file clearsign signature does not actually "
+                        "contain any valid signatures")
+        else:
+            report_verification_results(context, sigs, log)
+
+    try:
+        tmp_obj = tempfile.TemporaryFile("w+")
+    except IOError as err:
+        log.error("cannot create a temporary file for bmap: %s" % err)
+        raise SystemExit(1)
+
+    tmp_obj.write(plaintext.getvalue())
+    tmp_obj.seek(0)
+    return tmp_obj
+
+def verify_bmap_signature(args, bmap_obj, bmap_path, log):
+    """
+    Verify GPG signature of the bmap file if it is present. The signature may
+    be in a separate file (detached) or it may be inside the bmap file itself
+    (clearsign signature).
+
+    If user specifies the --bmap-sig option, the signature is assumed to be
+    detached and is taken from the user-specified file. Otherwise, this
+    function verifies whether the bmap file has clearsign signature, and if
+    not, it tries to automatically discover the detached signature by searching
+    for a ".sig" or ".asc" file at the same path and with the same basename as
+    the bmap file. This function then verifies the signature and reports the
+    results.
+
+    In case of the clearsign signature, the bmap file has "invalid" format,
+    meaning that the proper bmap XML contents is in the GPG clearsign
+    container. The XML contents has to be extracted from the container before
+    further processing. And this is be done even if user specified the
+    --no-sig-verify option. This function returns an open file object with the
+    extracted XML bmap file contents in this case. Otherwise, this function
+    returns None.
+    """
+
+    if not bmap_obj:
+        return None
+
+    clearsign_marker = "-----BEGIN PGP SIGNED MESSAGE-----"
+    buf = bmap_obj.read(len(clearsign_marker))
+    bmap_obj.seek(0)
+
+    if buf == clearsign_marker:
+        return verify_clearsign_bmap_signature(args, bmap_obj, log)
+    else:
+        return verify_detached_bmap_signature(args, bmap_obj, bmap_path, log)
+
+
 def find_and_open_bmap(args, log):
     """
     This is a helper function for 'open_files()' which discovers and opens the
     bmap file, then returns the corresponding file object and the bmap file
     path.
 
-    If the user specified the bmap file explicitely, we just open the provided
+    If the user specified the bmap file explicitly, we just open the provided
     path. Otherwise, we try to discover the bmap file at the same place where
     the image file is located. We search for a file with the same path and
     basename, but with a ".bmap" extension.
@@ -154,7 +334,7 @@ def find_and_open_bmap(args, log):
 
     try:
         # Create a temporary file for the bmap
-        tmp_obj = tempfile.TemporaryFile("w+")
+        tmp_obj = tempfile.NamedTemporaryFile("w+")
     except IOError as err:
         log.error("cannot create a temporary file for bmap: %s" % err)
         raise SystemExit(1)
@@ -226,8 +406,23 @@ def copy_command(args, log):
         log.error("--nobmap and --bmap cannot be used together")
         raise SystemExit(1)
 
+    if args.bmap_sig and args.no_sig_verify:
+        log.error("--bmap-sig and --no-sig-verify cannot be used together")
+        raise SystemExit(1)
+
     image_obj, dest_obj, bmap_obj, bmap_path, image_size, dest_is_blkdev = \
                                                    open_files(args, log)
+
+    if args.bmap_sig and not bmap_obj:
+        log.error("the bmap signature file was specified, but bmap file "
+                  "was not found")
+        raise SystemExit(1)
+
+    f_obj = verify_bmap_signature(args, bmap_obj, bmap_path, log)
+    if f_obj:
+        bmap_obj.close()
+        bmap_obj = f_obj
+
     try:
         if dest_is_blkdev:
             dest_str = "block device '%s'" % args.dest
@@ -413,6 +608,14 @@ def parse_arguments():
     text = "allow copying without a bmap file"
     parser_copy.add_argument("--nobmap", action="store_true", help=text)
 
+    # The --bmap-sig option
+    text = "the detached GPG signature for the bmap file"
+    parser_copy.add_argument("--bmap-sig", help=text)
+
+    # The --no-sig-verify option
+    text = "do not verify bmap file GPG signatrue"
+    parser_copy.add_argument("--no-sig-verify", action="store_true", help=text)
+
     # The --no-verify option
     text = "do not verify the data checksum while writing"
     parser_copy.add_argument("--no-verify", action="store_true", help=text)
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Sep 19 06:35:22 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu, 19 Sep 2013 13:35:22 +0300
Subject: [bmap-tools] [PATCH 09/14] docs: update bmaptool's man pages
In-Reply-To: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
References: <1379586922-12724-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1379586922-12724-9-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Add information about new options, and more.

Change-Id: Id1c9ffe97bbb22ebd2ee4dd645142a8bc7b6a476
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 docs/man1/bmaptool.1 | 89 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 72 insertions(+), 17 deletions(-)

diff --git a/docs/man1/bmaptool.1 b/docs/man1/bmaptool.1
index 912c9ca..dffcc08 100644
--- a/docs/man1/bmaptool.1
+++ b/docs/man1/bmaptool.1
@@ -1,4 +1,4 @@
-.TH BMAPTOOL "1" "February 2013" "bmap-tools 2.1" "User Commands"
+.TH BMAPTOOL "17" "September 2013" "bmap-tools 2.6" "User Commands"
 
 .SH NAME
 
@@ -9,7 +9,7 @@ bmaptool - create block map (bmap) for a file or copy a file using bmap
 
 .PP
 .B bmaptool
-[\-\-help] [\-\-version] [\-\-quiet] <subcommand> [<args>]
+[\-\-help] [\-\-version] [\-\-quiet] <subcommand> [<options>] <args>
 
 .SH DESCRIPTION
 
@@ -92,9 +92,9 @@ in the URL, for example "https://user:password at my.server.org/image.raw.bz2", or
 .PP
 IMAGE may be compressed, in which case \fIbmaptool\fR decompresses it on-the-fly.
 The compression type is detected by the file extension and the following
-compression types are supported: ".bz2", ".gz", ".tar.gz" or ".tgz", and
-".tar.bz2". If IMAGE has none of these extensions, it is assumed to be
-uncompressed.
+compression types are supported: ".bz2", ".gz", ".xz", ".gzip", ".tar.bz2",
+".tar.gz", ".tgz", ".tar.xz". If IMAGE has none of these extensions, it is
+assumed to be uncompressed.
 
 .PP
 If DEST is a block device node (e.g., "/dev/sdg"), \fIbmaptool\fR opens it in
@@ -102,6 +102,35 @@ exclusive mode. This means that it will fail if any other process has IMAGE
 block device node opened. This also means that no other processes will be able
 to open IMAGE until \fIbmaptool\fR finishes the copying. Please, see semantics
 of the "O_EXCL" flag of the "open()" syscall.
+
+.PP
+The bmap file typically contains SHA-256 checksum for itself as well as SHA-256
+checksum for all the mapped data regions, which makes it possible to guarantee
+data integrity. \fIbmaptool\fR verifies the checksums and exits with an error
+in case of a mismatch. Checksum verification can be disabled using the
+"--no-verify" option. \fIbmaptool\fR does not verify that unampped areas
+contain only zeroes, because these areas are anyway dropped and are not used for
+anything.
+
+.PP
+The bmap file may be signed with OpenPGP (gpg). The signature may be either
+detached (a separate file) or "built into" the bmap file (so called "clearsign"
+signature).
+
+.PP
+The detached signature can be specified with the "--bmap-sig" option, otherwise
+\fIbmaptool\fR tries to automatically discover it by looking for a file with
+the same basename as the bmap file but with the ".asc" or ".sig" extension.
+This is very similar to the bmap file auto-discovery. So if a ".asc" or ".sig"
+file exists, \fIbmaptool\fR will verify the signature.
+
+.PP
+The clearsign signature is part of the bmap file and \fIbmaptool\fR
+automatically detected and verifies it.
+
+.PP
+If the signature is bad, \fIbmaptool\fR exits with an error. Bmap file
+signature verification can be disabled using the "--no-sig-verify" option.
 .RE
 
 .\"
@@ -123,6 +152,14 @@ tries to automatically discover the bmap file.
 .RE
 
 .PP
+\-\-bmap-sig SIG
+.RS 2
+Use a detached OpenPGP signature file "SIG" for verifying the bmap file
+integrity and publisher. If this option is not specified, \fIbmaptool\fR
+tries to automatically discover the signature file.
+.RE
+
+.PP
 \-\-nobmap
 .RS 2
 Disable automatic bmap file discovery and force flashing entire IMAGE without bmap.
@@ -131,10 +168,16 @@ Disable automatic bmap file discovery and force flashing entire IMAGE without bm
 .PP
 \-\-no-verify
 .RS 2
-Do not verify SHA1 checksums when copying (not recommended). The checksums are
+Do not verify data checksums when copying (not recommended). The checksums are
 stored in the bmap file, and normally \fIbmaptool\fR verifies that the data in
 IMAGE matches the checksums.
 .RE
+
+.PP
+\-\-no-sig-verify
+.RS 2
+Do not verify the OpenPGP bmap file signature (not recommended).
+.RE
 .RE
 .RE
 
@@ -144,18 +187,12 @@ IMAGE matches the checksums.
 .RS 2
 \fBEXAMPLES\fR
 .RS 2
-\fIbmaptool\fR copy --bmap image.bmap image.raw /dev/sdg
-.RS 2
-Copy non-compressed local file "image.raw" to block device "/dev/sdg" using bmap file "image.bmap".
-.RE
-.RE
-
-.RS 2
 \fIbmaptool\fR copy image.raw.bz2 /dev/sdg
 .RS 2
 Copy bz2-compressed local file "image.raw.bz2" to block device "/dev/sdg". The
-image file is uncompressed on-the-fly. The bamap file is discovered
-automatically.
+image file is uncompressed on-the-fly. The bmap file is discovered
+automatically. The OpenPGP signature is detected/discovered automatically
+too.
 .RE
 .RE
 
@@ -163,11 +200,20 @@ automatically.
 \fIbmaptool\fR copy http://my-server.com/files/image.raw.bz2 $HOME/tmp/file
 .RS 2
 Copy bz2-compressed remote "image.raw.bz2" to regular file "$HOME/tmp/file".
-The image file is uncompressed on-the-fly. The bamap file is discovered
-automatically.
+The image file is uncompressed on-the-fly. The bmap file is discovered
+automatically. The OpenPGP signature is detected/discovered automatically
+too.
 .RE
 .RE
 
+.RS 2
+\fIbmaptool\fR copy --bmap image.bmap --bmap-sig image.bmap.asc image.raw /dev/sdg
+.RS 2
+Copy non-compressed local file "image.raw" to block device "/dev/sdg" using bmap file
+"image.bmap". Verify the bmap file signature using a detached OpenPGP signature
+from "imag.bmap.asc".
+.RE
+.RE
 
 .\"
 .\" The "create" command description
@@ -179,6 +225,15 @@ Generate bmap for a regular file IMAGE. Internally, this subcommand uses the
 Linux "FIEMAP" ioctl to find out which IMAGE blocks are mapped. By default, the
 resulting bmap file is printed to stdout, unless the "--output" option is used.
 
+.PP
+The IMAGE file is always synchronized before the block map is generated. And it
+is important to make sure that the IMAGE file is not modified when the bmap
+file is being generated, and after the bmap file has been generated. Otherwise
+the bmap file becomes invalid and checksum verification will fail.
+
+.PP
+The image file can further be signed using OpenPGP.
+
 .\"
 .\" The "create" command's options
 .\"
-- 
1.8.1.4



From dedekind1 at gmail.com  Fri Oct  4 07:52:34 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Fri,  4 Oct 2013 14:52:34 +0300
Subject: [bmap-tools] Announcement: bmap-tools-3.0 is out!
Message-ID: <1380887554-31460-1-git-send-email-dedekind1@gmail.com>

Bmap-tools version 3.0 is out!

Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/release-3.0:/docs/RELEASE_NOTES
Tarball: ftp://ftp.infradead.org/pub/bmap-tools/

Packages for various distributions are available here:
http://download.tizen.org/tools/pre-release/

At some later point they will be propagated to here:
http://download.tizen.org/tools/latest-release/

--
Artem Bityutskiy


From dedekind1 at gmail.com  Mon Nov  4 03:05:23 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:23 +0200
Subject: [bmap-tools] [PATCH 03/16] BmapCreate: fix bmap file format version
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

By a mistake, I've made bmap file format version to be 1.4, while is should
really have been 2.0, because there were incompatible changes. Fix this.

Change-Id: If2618db2c12b82f64e1f813e72a2786efdf4d3d5
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCreate.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/bmaptools/BmapCreate.py b/bmaptools/BmapCreate.py
index e1a84c1..2cb7c06 100644
--- a/bmaptools/BmapCreate.py
+++ b/bmaptools/BmapCreate.py
@@ -49,11 +49,16 @@ from bmaptools import Fiemap
 # The bmap format version we generate.
 #
 # Changelog:
-# o 1.3 -> 1.4:
+# o 1.3 -> 2.0:
 #   Support SHA256 and SHA512 checksums, in 1.3 only SHA1 was supported.
 #   "BmapFileChecksum" is used instead of "BmapFileSHA1", and "chksum="
-#   attribute is used instead "sha1=". Introduced "ChecksumType" tag.
-SUPPORTED_BMAP_VERSION = "1.4"
+#   attribute is used instead "sha1=". Introduced "ChecksumType" tag. This is
+#   an incompatible change.
+#   Note, bmap format 1.4 is identical to 2.0. Version 1.4 was a mistake,
+#   instead of incrementing the major version number, we incremented minor
+#   version number. Unfortunately, the mistake slipped into bmap-tools version
+#   3.0, and was only fixed in bmap-tools v3.1.
+SUPPORTED_BMAP_VERSION = "2.0"
 
 _BMAP_START_TEMPLATE = \
 """<?xml version="1.0" ?>
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:21 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:21 +0200
Subject: [bmap-tools] [PATCH 01/16] TODO: add some more entries
Message-ID: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Ief83f6e42a306946826f89794c64981f0450baf3
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/TODO b/TODO
index 145e305..1c049fe 100644
--- a/TODO
+++ b/TODO
@@ -1,3 +1,6 @@
 Current TODO list, any help with these is appreciated.
 
 1. Teach bmaptool to update the alternate GPT partition
+2. Add nosetests for all bmap format versions
+3. Add a test for version with incompatible changes
+4. Add a test for bmap with invalid checksums
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:22 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:22 +0200
Subject: [bmap-tools] [PATCH 02/16] BmapCopy: verify versions in a single
	function
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This patch is a minor clean-up and a preparation to the upcoming changes. It
makes sure that all the differences between various bmap format version are
handled in the '_parse_bmap()' method, and we do not have to worry about the
format differences later on.

Change-Id: I00b0cc7b5f9c3372f85fb7519c3ce768f5db3bf4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index fe8897c..357b45d 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -176,6 +176,7 @@ class BmapCopy:
         self._cs_type = None
         self._cs_len = None
         self._cs_attrib_name = None
+        self._bmap_cs_attrib_name = None
 
         # Special quirk for /dev/null which does not support fsync()
         if stat.S_ISCHR(st_data.st_mode) and \
@@ -247,10 +248,7 @@ class BmapCopy:
 
         import mmap
 
-        if self.bmap_version_minor == 3:
-            correct_chksum = self._xml.find("BmapFileSHA1").text.strip()
-        else:
-            correct_chksum = self._xml.find("BmapFileChecksum").text.strip()
+        correct_chksum = self._xml.find(self._bmap_cs_attrib_name).text.strip()
 
         # Before verifying the shecksum, we have to substitute the checksum
         # value stored in the file with all zeroes. For these purposes we
@@ -329,9 +327,11 @@ class BmapCopy:
             if self.bmap_version_minor == 3:
                 self._cs_type = "sha1"
                 self._cs_attrib_name = "sha1"
+                self._bmap_cs_attrib_name = "BmapFileSHA1"
             else:
                 self._cs_type = xml.find("ChecksumType").text.strip()
                 self._cs_attrib_name = "chksum"
+                self._bmap_cs_attrib_name = "BmapFileChecksum"
 
             try:
                 self._cs_len = len(hashlib.new(self._cs_type).hexdigest())
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:24 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:24 +0200
Subject: [bmap-tools] [PATCH 04/16] BmapCopy: handle bmap format version 2.0
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This patch makes BmapCopy handle bmap format version 2.0, which is identical to
format 1.4. Format 1.4 was released by a mistake. Format 1.4 makes bmaptool
v2.x crash because it has incompatible changes. This is unfurtunate, sorry for
this.

Change-Id: Id26fd9fc807474004dddef0eee0b2028f75c3c41
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 bmaptools/BmapCopy.py | 42 ++++++++++++++++++++++++++----------------
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/bmaptools/BmapCopy.py b/bmaptools/BmapCopy.py
index 357b45d..5c9e5dc 100644
--- a/bmaptools/BmapCopy.py
+++ b/bmaptools/BmapCopy.py
@@ -59,7 +59,7 @@ from xml.etree import ElementTree
 from bmaptools.BmapHelpers import human_size
 
 # The highest supported bmap format version
-SUPPORTED_BMAP_VERSION = 1
+SUPPORTED_BMAP_VERSION = 2
 
 class Error(Exception):
     """
@@ -318,21 +318,31 @@ class BmapCopy:
                         "blocks count (%d bytes != %d blocks * %d bytes)"
                         % (self.image_size, self.blocks_cnt, self.block_size))
 
-        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
-            # Bmap file checksum appeard in format 1.3 and the only supported
-            # checksum type was SHA1. Version 1.4 started supporting arbitrary
-            # checksum types. A new "ChecksumType" tag was introduce to specify
-            # the checksum function name. And all XML tags which contained
-            # "sha1" in their name were renamed to something more neutral.
-            if self.bmap_version_minor == 3:
-                self._cs_type = "sha1"
-                self._cs_attrib_name = "sha1"
-                self._bmap_cs_attrib_name = "BmapFileSHA1"
-            else:
-                self._cs_type = xml.find("ChecksumType").text.strip()
-                self._cs_attrib_name = "chksum"
-                self._bmap_cs_attrib_name = "BmapFileChecksum"
-
+        if self.bmap_version_major > 1 or \
+           (self.bmap_version_major == 1 and self.bmap_version_minor == 4):
+            # In bmap format version 1.0-1.3 the only supported checksum type
+            # was SHA1. Version 2.0 started supporting arbitrary checksum
+            # types. A new "ChecksumType" tag was introduce to specify the
+            # checksum function name. And all XML tags which contained "sha1"
+            # in their name were renamed to something more neutral. This was an
+            # change incompatible with previous formats.
+            #
+            # There is a special format version 1.4, which should not have been
+            # ever issued, but was released by a mistake. The mistake was that
+            # when implementing version 2.0 support we mistakenly gave it
+            # version number 1.4. This was later on fixed and format version
+            # 1.4 became version 2.0. So 1.4 and 2.0 formats are identical.
+            #
+            # Note, bmap files did not contain checksums prior to version 1.3.
+            self._cs_type = xml.find("ChecksumType").text.strip()
+            self._cs_attrib_name = "chksum"
+            self._bmap_cs_attrib_name = "BmapFileChecksum"
+        elif self.bmap_version_minor == 3:
+            self._cs_type = "sha1"
+            self._cs_attrib_name = "sha1"
+            self._bmap_cs_attrib_name = "BmapFileSHA1"
+
+        if self._cs_type:
             try:
                 self._cs_len = len(hashlib.new(self._cs_type).hexdigest())
             except ValueError as err:
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:26 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:26 +0200
Subject: [bmap-tools] [PATCH 06/16] test_api_base: use image name in
	_copy_image
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-6-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_copy_image()' so that it expects
the image to be a path and does not allow for file-like objects. This is
cleaner than seeking the file-like object in this function, which changes the
object state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: I81e7e2d15676794f9d5e0e93ce62e0945b95a6d0
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index a73f19d..0164542 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -177,11 +177,7 @@ def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
     Copy image 'image' using bmap 'f_bmap' to the destination file 'f_dest'.
     """
 
-    if hasattr(image, "read"):
-        f_image = image
-        image.seek(0)
-    else:
-        f_image = TransRead.TransRead(image)
+    f_image = TransRead.TransRead(image)
 
     f_dest.seek(0)
     if f_bmap:
@@ -197,8 +193,7 @@ def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
     f_dest.seek(0)
     assert _calculate_chksum(f_dest) == image_chksum
 
-    if not hasattr(image, "read"):
-        f_image.close()
+    f_image.close()
 
 def _do_test(f_image, image_size, delete=True):
     """
@@ -241,7 +236,7 @@ def _do_test(f_image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(f_image.name, f_bmap1.name)
     creator.generate()
 
-    _copy_image(f_image, f_copy, f_bmap1, image_chksum, image_size)
+    _copy_image(f_image.name, f_copy, f_bmap1, image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -253,7 +248,7 @@ def _do_test(f_image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(f_image, f_bmap2)
     creator.generate()
-    _copy_image(f_image, f_copy, f_bmap2, image_chksum, image_size)
+    _copy_image(f_image.name, f_copy, f_bmap2, image_chksum, image_size)
     _compare_holes(f_image, f_copy)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
@@ -278,8 +273,8 @@ def _do_test(f_image, image_size, delete=True):
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_image(f_image, f_copy, None, image_chksum, image_size)
-    _copy_image(f_image, f_copy, None, image_chksum, None)
+    _copy_image(f_image.name, f_copy, None, image_chksum, image_size)
+    _copy_image(f_image.name, f_copy, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:25 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:25 +0200
Subject: [bmap-tools] [PATCH 05/16] tests: add proper license header to all
	tests
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-5-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I95195b227301119486c967cea2a338391ff26123
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/helpers.py       | 13 +++++++++++++
 tests/test_api_base.py | 13 +++++++++++++
 tests/test_fiemap.py   | 13 +++++++++++++
 3 files changed, 39 insertions(+)

diff --git a/tests/helpers.py b/tests/helpers.py
index deeb5ee..81c24e3 100644
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -1,3 +1,16 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+# License: GPLv2
+# Author: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
 """
 This module contains independent functions shared between various
 tests.
diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index ca10488..a73f19d 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -1,3 +1,16 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+# License: GPLv2
+# Author: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
 """
 This test verifies the base bmap creation and copying API functionality. It
 generates a random sparse file, then creates a bmap fir this file and copies it
diff --git a/tests/test_fiemap.py b/tests/test_fiemap.py
index 47fe0fe..d9ab85c 100644
--- a/tests/test_fiemap.py
+++ b/tests/test_fiemap.py
@@ -1,3 +1,16 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+# License: GPLv2
+# Author: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
 """
 This test verifies Fiemap module functionality. It generates random sparse
 files and makes sure FIEMAP returns correct information about the holes.
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:27 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:27 +0200
Subject: [bmap-tools] [PATCH 07/16] test_api_base: use file name in
	_calculate_chksum
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-7-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_calculate_chksum()' so that it
expects the image to be a path and does not allow for file-like objects. This
is cleaner than seeking the file-like object in this function, which changes
the object state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: Iec472bd5f02eb9c2a1162f43fb41defe87ff3ed6
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 0164542..18ba1a3 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -154,12 +154,10 @@ def _generate_compressed_files(file_obj, delete=True):
     yield tbz2_file_obj.name
     tmp_file_obj.close()
 
-def _calculate_chksum(file_obj):
-    """
-    Calculates checksum for the contents of file object 'file_obj'.
-    """
+def _calculate_chksum(file_path):
+    """Calculates checksum for the contents of file 'file_path'."""
 
-    file_obj.seek(0)
+    file_obj = TransRead.TransRead(file_path)
     hash_obj = hashlib.new("sha256")
 
     chunk_size = 1024*1024
@@ -170,6 +168,7 @@ def _calculate_chksum(file_obj):
             break
         hash_obj.update(chunk)
 
+    file_obj.close()
     return hash_obj.hexdigest()
 
 def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
@@ -191,7 +190,7 @@ def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
 
     # Compare the original file and the copy are identical
     f_dest.seek(0)
-    assert _calculate_chksum(f_dest) == image_chksum
+    assert _calculate_chksum(f_dest.name) == image_chksum
 
     f_image.close()
 
@@ -226,7 +225,7 @@ def _do_test(f_image, image_size, delete=True):
                                           delete=delete, dir=directory,
                                           suffix=".bmap2")
 
-    image_chksum = _calculate_chksum(f_image)
+    image_chksum = _calculate_chksum(f_image.name)
 
     #
     # Pass 1: generate the bmap, copy and compare
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:28 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:28 +0200
Subject: [bmap-tools] [PATCH 08/16] test_api_base: use file name in
	_generate_compressed_files
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-8-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_generate_compressed_files()' so
that it expects the image to be a path and does not allow for file-like
objects. This is cleaner than seeking the file-like object in this function,
which changes the object state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: I2c9004fa66b50a73d310f5ab649e3c1b1a3835c1
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 23 +++++++++++++----------
 1 file changed, 13 insertions(+), 10 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 18ba1a3..15edead 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -61,10 +61,10 @@ def _compare_holes(file1, file2):
             raise Error("mismatch for hole %d-%d, it is %d-%d in file2"
                         % (range1[0], range1[1], range2[0], range2[1]))
 
-def _generate_compressed_files(file_obj, delete=True):
+def _generate_compressed_files(file_path, delete=True):
     """
-    This is a generator which yields compressed versions of a file represented
-    by a file object 'file_obj'.
+    This is a generator which yields compressed versions of a file
+    'file_path'.
 
     The 'delete' argument specifies whether the compressed files that this
     generator yields have to be automatically deleted.
@@ -84,17 +84,18 @@ def _generate_compressed_files(file_obj, delete=True):
         except ImportError:
             lzma_present = False
 
+    file_obj = TransRead.TransRead(file_path)
+
     # Make sure the temporary files start with the same name as 'file_obj' in
     # order to simplify debugging.
-    prefix = os.path.splitext(os.path.basename(file_obj.name))[0] + '.'
+    prefix = os.path.splitext(os.path.basename(file_path))[0] + '.'
     # Put the temporary files in the directory with 'file_obj'
-    directory = os.path.dirname(file_obj.name)
+    directory = os.path.dirname(file_path)
 
     # Generate an uncompressed version of the file
     tmp_file_obj = tempfile.NamedTemporaryFile('wb+', prefix=prefix,
                                                delete=delete, dir=directory,
                                                suffix='.uncompressed')
-    file_obj.seek(0)
     shutil.copyfileobj(file_obj, tmp_file_obj)
     tmp_file_obj.flush()
     yield tmp_file_obj.name
@@ -139,7 +140,7 @@ def _generate_compressed_files(file_obj, delete=True):
                                                delete=delete, dir=directory,
                                                suffix='.tar.gz')
     tgz_file_obj = tarfile.open(tmp_file_obj.name, "w:gz")
-    tgz_file_obj.add(file_obj.name)
+    tgz_file_obj.add(file_path)
     tgz_file_obj.close()
     yield tgz_file_obj.name
     tmp_file_obj.close()
@@ -149,11 +150,13 @@ def _generate_compressed_files(file_obj, delete=True):
                                                delete=delete, dir=directory,
                                                suffix='.tar.bz2')
     tbz2_file_obj = tarfile.open(tmp_file_obj.name, "w:bz2")
-    tbz2_file_obj.add(file_obj.name)
+    tbz2_file_obj.add(file_path)
     tbz2_file_obj.close()
     yield tbz2_file_obj.name
     tmp_file_obj.close()
 
+    file_obj.close()
+
 def _calculate_chksum(file_path):
     """Calculates checksum for the contents of file 'file_path'."""
 
@@ -257,7 +260,7 @@ def _do_test(f_image, image_size, delete=True):
     # Pass 3: test compressed files copying with bmap
     #
 
-    for compressed in _generate_compressed_files(f_image, delete=delete):
+    for compressed in _generate_compressed_files(f_image.name, delete=delete):
         _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
@@ -279,7 +282,7 @@ def _do_test(f_image, image_size, delete=True):
     # Pass 6: test compressed files copying without bmap
     #
 
-    for compressed in _generate_compressed_files(f_image, delete=delete):
+    for compressed in _generate_compressed_files(f_image.name, delete=delete):
         _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:30 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:30 +0200
Subject: [bmap-tools] [PATCH 10/16] test_api_base: use file name in
	_copy_image (2)
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-10-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_copy_image()' so that it expects
the image to be a path and does not allow for file-like objects. This is
cleaner than seeking the file-like object in this function, which changes the
object state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: I4a74431d10e2c174c4e21dd11a8adc69603a717f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 6073906..17f5191 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -174,12 +174,13 @@ def _calculate_chksum(file_path):
     file_obj.close()
     return hash_obj.hexdigest()
 
-def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
+def _copy_image(image, dest, f_bmap, image_chksum, image_size):
     """
-    Copy image 'image' using bmap 'f_bmap' to the destination file 'f_dest'.
+    Copy image 'image' using bmap 'f_bmap' to the destination file 'dest'.
     """
 
     f_image = TransRead.TransRead(image)
+    f_dest = open(dest, "w+")
 
     f_dest.seek(0)
     if f_bmap:
@@ -195,6 +196,7 @@ def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
     f_dest.seek(0)
     assert _calculate_chksum(f_dest.name) == image_chksum
 
+    f_dest.close()
     f_image.close()
 
 def _do_test(image, image_size, delete=True):
@@ -238,7 +240,7 @@ def _do_test(image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(image, f_bmap1.name)
     creator.generate()
 
-    _copy_image(image, f_copy, f_bmap1, image_chksum, image_size)
+    _copy_image(image, f_copy.name, f_bmap1, image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -250,8 +252,8 @@ def _do_test(image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(image, f_bmap2)
     creator.generate()
-    _copy_image(image, f_copy, f_bmap2, image_chksum, image_size)
-    _compare_holes(image, f_copy)
+    _copy_image(image, f_copy.name, f_bmap2, image_chksum, image_size)
+    _compare_holes(image, f_copy.name)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
     assert filecmp.cmp(f_bmap1.name, f_bmap2.name, False)
@@ -261,36 +263,36 @@ def _do_test(image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
         compressed = "file:" + compressed
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
 
     #
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_image(image, f_copy, None, image_chksum, image_size)
-    _copy_image(image, f_copy, None, image_chksum, None)
+    _copy_image(image, f_copy.name, None, image_chksum, image_size)
+    _copy_image(image, f_copy.name, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
-        _copy_image(compressed, f_copy, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
 
     # Close temporary files, which will also remove them
     f_copy.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:29 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:29 +0200
Subject: [bmap-tools] [PATCH 09/16] test_api_base: use file name in _do_test
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-9-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_do_test()' so that it expects the
image to be a path and does not allow for file-like objects. This is cleaner
than seeking the file-like object in this function, which changes the object
state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: I884b54db2875bfbe68b15172ccc26767c40ff971
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 38 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 15edead..6073906 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -197,23 +197,23 @@ def _copy_image(image, f_dest, f_bmap, image_chksum, image_size):
 
     f_image.close()
 
-def _do_test(f_image, image_size, delete=True):
+def _do_test(image, image_size, delete=True):
     """
     A basic test for the bmap creation and copying functionality. It first
-    generates a bmap for file object 'f_image', and then copies the sparse file
-    to a different file, and then checks that the original file and the copy
-    are identical.
+    generates a bmap for file 'image', and then copies the sparse file to a
+    different file, and then checks that the original file and the copy are
+    identical.
 
     The 'image_size' argument is size of the image in bytes. The 'delete'
     argument specifies whether the temporary files that this function creates
     have to be automatically deleted.
     """
 
-    # Make sure the temporary files start with the same name as 'f_image' in
+    # Make sure the temporary files start with the same name as 'image' in
     # order to simplify debugging.
-    prefix = os.path.splitext(os.path.basename(f_image.name))[0] + '.'
+    prefix = os.path.splitext(os.path.basename(image))[0] + '.'
     # Put the temporary files in the directory with the image
-    directory = os.path.dirname(f_image.name)
+    directory = os.path.dirname(image)
 
     # Create and open a temporary file for a copy of the copy
     f_copy = tempfile.NamedTemporaryFile("wb+", prefix=prefix,
@@ -228,30 +228,30 @@ def _do_test(f_image, image_size, delete=True):
                                           delete=delete, dir=directory,
                                           suffix=".bmap2")
 
-    image_chksum = _calculate_chksum(f_image.name)
+    image_chksum = _calculate_chksum(image)
 
     #
     # Pass 1: generate the bmap, copy and compare
     #
 
     # Create bmap for the random sparse file
-    creator = BmapCreate.BmapCreate(f_image.name, f_bmap1.name)
+    creator = BmapCreate.BmapCreate(image, f_bmap1.name)
     creator.generate()
 
-    _copy_image(f_image.name, f_copy, f_bmap1, image_chksum, image_size)
+    _copy_image(image, f_copy, f_bmap1, image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
-    _compare_holes(f_image.name, f_copy.name)
+    _compare_holes(image, f_copy.name)
 
     #
     # Pass 2: same as pass 1, but use file objects instead of paths
     #
 
-    creator = BmapCreate.BmapCreate(f_image, f_bmap2)
+    creator = BmapCreate.BmapCreate(image, f_bmap2)
     creator.generate()
-    _copy_image(f_image.name, f_copy, f_bmap2, image_chksum, image_size)
-    _compare_holes(f_image, f_copy)
+    _copy_image(image, f_copy, f_bmap2, image_chksum, image_size)
+    _compare_holes(image, f_copy)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
     assert filecmp.cmp(f_bmap1.name, f_bmap2.name, False)
@@ -260,7 +260,7 @@ def _do_test(f_image, image_size, delete=True):
     # Pass 3: test compressed files copying with bmap
     #
 
-    for compressed in _generate_compressed_files(f_image.name, delete=delete):
+    for compressed in _generate_compressed_files(image, delete=delete):
         _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
@@ -275,14 +275,14 @@ def _do_test(f_image, image_size, delete=True):
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_image(f_image.name, f_copy, None, image_chksum, image_size)
-    _copy_image(f_image.name, f_copy, None, image_chksum, None)
+    _copy_image(image, f_copy, None, image_chksum, image_size)
+    _copy_image(image, f_copy, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
     #
 
-    for compressed in _generate_compressed_files(f_image.name, delete=delete):
+    for compressed in _generate_compressed_files(image, delete=delete):
         _copy_image(compressed, f_copy, f_bmap1, image_chksum, image_size)
 
         # Test without setting the size
@@ -320,4 +320,4 @@ class TestCreateCopy(unittest.TestCase):
                                                      directory=directory)
         for f_image, image_size, _, _ in iterator:
             assert image_size == os.path.getsize(f_image.name)
-            _do_test(f_image, image_size, delete=delete)
+            _do_test(f_image.name, image_size, delete=delete)
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:32 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:32 +0200
Subject: [bmap-tools] [PATCH 12/16] test_api_base: rename _copy_image
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-12-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This function copies and verifies, so let's call it '_copy_and_verify_image()'.

Change-Id: Ifd67af5a3eee73dc755097462410bcc1835a145c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 47 +++++++++++++++++++++++++++--------------------
 1 file changed, 27 insertions(+), 20 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 6e94a32..2a9ce10 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -174,9 +174,10 @@ def _calculate_chksum(file_path):
     file_obj.close()
     return hash_obj.hexdigest()
 
-def _copy_image(image, dest, bmap, image_chksum, image_size):
+def _copy_and_verify_image(image, dest, bmap, image_chksum, image_size):
     """
-    Copy image 'image' using bmap file 'bmap' to the destination file 'dest'.
+    Copy image 'image' using bmap file 'bmap' to the destination file 'dest'
+    and verify the resulting image checksum.
     """
 
     f_image = TransRead.TransRead(image)
@@ -241,7 +242,8 @@ def _do_test(image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(image, f_bmap1.name)
     creator.generate()
 
-    _copy_image(image, f_copy.name, f_bmap1.name, image_chksum, image_size)
+    _copy_and_verify_image(image, f_copy.name, f_bmap1.name, image_chksum,
+                           image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -253,7 +255,8 @@ def _do_test(image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(image, f_bmap2)
     creator.generate()
-    _copy_image(image, f_copy.name, f_bmap2.name, image_chksum, image_size)
+    _copy_and_verify_image(image, f_copy.name, f_bmap2.name, image_chksum,
+                           image_size)
     _compare_holes(image, f_copy.name)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
@@ -264,40 +267,44 @@ def _do_test(image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
-                    image_size)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
         compressed = "file:" + compressed
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
-                    image_size)
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, image_size)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, None)
 
     #
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_image(image, f_copy.name, None, image_chksum, image_size)
-    _copy_image(image, f_copy.name, None, image_chksum, None)
+    _copy_and_verify_image(image, f_copy.name, None, image_chksum, image_size)
+    _copy_and_verify_image(image, f_copy.name, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
-                    image_size)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
-
-        # Append a "file:" prefixe to make BmapCopy use urllib
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
-                    image_size)
-        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, None)
+
+        # Append a "file:" prefix to make BmapCopy use urllib
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, image_size)
+        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                               image_chksum, None)
 
     # Close temporary files, which will also remove them
     f_copy.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:31 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:31 +0200
Subject: [bmap-tools] [PATCH 11/16] test_api_base: use file name in
	_copy_image (3)
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-11-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This is a minor clean-up which changes the '_copy_image()' so that it expects
the bmap file to be a path and does not allow for file-like objects. This is
cleaner than seeking the file-like object in this function, which changes the
object state.

This change is rather mechanical, and this is actually a preparation for the
upcoming changes, where I am going to make '_copy_image()' and some other
function generic, and then use them in a new test which I am going to add.

Change-Id: I9929a4bbbc496156db9e0fb821f998bcf7279904
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_api_base.py | 43 ++++++++++++++++++++++++-------------------
 1 file changed, 24 insertions(+), 19 deletions(-)

diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 17f5191..6e94a32 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -174,17 +174,17 @@ def _calculate_chksum(file_path):
     file_obj.close()
     return hash_obj.hexdigest()
 
-def _copy_image(image, dest, f_bmap, image_chksum, image_size):
+def _copy_image(image, dest, bmap, image_chksum, image_size):
     """
-    Copy image 'image' using bmap 'f_bmap' to the destination file 'dest'.
+    Copy image 'image' using bmap file 'bmap' to the destination file 'dest'.
     """
 
     f_image = TransRead.TransRead(image)
     f_dest = open(dest, "w+")
-
-    f_dest.seek(0)
-    if f_bmap:
-        f_bmap.seek(0)
+    if (bmap):
+        f_bmap = open(bmap, "r")
+    else:
+        f_bmap = None
 
     writer = BmapCopy.BmapCopy(f_image, f_dest, f_bmap, image_size)
     # Randomly decide whether we want the progress bar or not
@@ -193,9 +193,10 @@ def _copy_image(image, dest, f_bmap, image_chksum, image_size):
     writer.copy(bool(random.getrandbits(1)), bool(random.getrandbits(1)))
 
     # Compare the original file and the copy are identical
-    f_dest.seek(0)
-    assert _calculate_chksum(f_dest.name) == image_chksum
+    assert _calculate_chksum(dest) == image_chksum
 
+    if f_bmap:
+        f_bmap.close()
     f_dest.close()
     f_image.close()
 
@@ -217,7 +218,7 @@ def _do_test(image, image_size, delete=True):
     # Put the temporary files in the directory with the image
     directory = os.path.dirname(image)
 
-    # Create and open a temporary file for a copy of the copy
+    # Create and open a temporary file for a copy of the image
     f_copy = tempfile.NamedTemporaryFile("wb+", prefix=prefix,
                                         delete=delete, dir=directory,
                                         suffix=".copy")
@@ -240,7 +241,7 @@ def _do_test(image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(image, f_bmap1.name)
     creator.generate()
 
-    _copy_image(image, f_copy.name, f_bmap1, image_chksum, image_size)
+    _copy_image(image, f_copy.name, f_bmap1.name, image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -252,7 +253,7 @@ def _do_test(image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(image, f_bmap2)
     creator.generate()
-    _copy_image(image, f_copy.name, f_bmap2, image_chksum, image_size)
+    _copy_image(image, f_copy.name, f_bmap2.name, image_chksum, image_size)
     _compare_holes(image, f_copy.name)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
@@ -263,15 +264,17 @@ def _do_test(image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
+                    image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
         compressed = "file:" + compressed
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
+                    image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
 
     #
     # Pass 5: copy without bmap and make sure it is identical to the original
@@ -285,14 +288,16 @@ def _do_test(image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
+                    image_size)
 
         # Test without setting the size
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, image_size)
-        _copy_image(compressed, f_copy.name, f_bmap1, image_chksum, None)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum,
+                    image_size)
+        _copy_image(compressed, f_copy.name, f_bmap1.name, image_chksum, None)
 
     # Close temporary files, which will also remove them
     f_copy.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:36 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:36 +0200
Subject: [bmap-tools] [PATCH 16/16] TODO: update the todo list
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-16-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: Iae87d0a54aab6425dc35e30f18a01ffa1e7ca59c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/TODO b/TODO
index 1c049fe..2b8df99 100644
--- a/TODO
+++ b/TODO
@@ -2,5 +2,6 @@ Current TODO list, any help with these is appreciated.
 
 1. Teach bmaptool to update the alternate GPT partition
 2. Add nosetests for all bmap format versions
-3. Add a test for version with incompatible changes
-4. Add a test for bmap with invalid checksums
+3. Add a test for bmap with invalid checksums
+4. Add a test for checking that old BmapCopy works fine with newer compatible
+   bmap formats
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:35 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:35 +0200
Subject: [bmap-tools] [PATCH 15/16] test_compat: new test for checking
	backward-compatibility
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-15-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

This test makes sure that BmapCopy works fine with all the previous bmap file
formats.

Change-Id: Ia3da88a99b32497f8ca782630d698ca54cfafc09
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_compat.py | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
 create mode 100644 tests/test_compat.py

diff --git a/tests/test_compat.py b/tests/test_compat.py
new file mode 100644
index 0000000..d0ba703
--- /dev/null
+++ b/tests/test_compat.py
@@ -0,0 +1,78 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+# License: GPLv2
+# Author: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
+"""
+This unit test verifies that BmapCopy can handle all the bmap file formats and
+have no backward-compatibility problems.
+"""
+
+import os
+import shutil
+import tempfile
+from tests import helpers
+from bmaptools import TransRead
+
+# This is a work-around for Centos 6
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
+
+class TestCreateCopy(unittest.TestCase):
+    """
+    The test class for this unit tests which executes the '_do_test()' function
+    for various bmap file formats.
+    """
+
+    def test(self):
+        """
+        The test entry point. Executes the '_do_test()' function for various
+        bmap file formats.
+        """
+
+        image_name = "test.image.gz"
+        bmap_name  = "test.image.bmap.v"
+        test_data_dir = "test-data"
+
+        test_data_dir = os.path.join(os.path.dirname(__file__), test_data_dir)
+        image_path = os.path.join(test_data_dir, image_name)
+
+
+        # Get the list of bmap files to test
+        bmap_paths = []
+        for direntry in os.listdir(test_data_dir):
+            direntry_path = os.path.join(test_data_dir, direntry)
+            if os.path.isfile(direntry_path) and direntry.startswith(bmap_name):
+                bmap_paths.append(direntry_path)
+
+        # Create and open a temporary file for uncompressed image and its copy
+        f_image = tempfile.NamedTemporaryFile("wb+", prefix=image_name,
+                                              suffix=".image")
+        f_copy = tempfile.NamedTemporaryFile("wb+", prefix=image_name,
+                                             suffix=".copy")
+
+        # Create an ucompressed version of the image file
+        f_tmp_img = TransRead.TransRead(image_path)
+        shutil.copyfileobj(f_tmp_img, f_image)
+        f_tmp_img.close()
+        f_image.flush()
+
+        image_chksum = helpers.calculate_chksum(f_image.name)
+        image_size = os.path.getsize(f_image.name)
+
+        for bmap_path in bmap_paths:
+            helpers.copy_and_verify_image(image_path, f_copy.name, bmap_path,
+                                          image_chksum, image_size)
+
+        f_copy.close()
+        f_image.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:33 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:33 +0200
Subject: [bmap-tools] [PATCH 13/16] test_api_base: move a couple of
	functions to helpers.py
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-13-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Move '_copy_and_verify_image()' and  '_calculate_chksum()' to the helpers.py
file since I am going to use them in the new unit test which I am about to add.
Remove the leading underscore since these functions become usable from outside.
Move some necessary module imports to helpers.py too.

Change-Id: Ifca399497f2d7b7bf046f916c1e744a4422cc7cc
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/helpers.py       |  48 ++++++++++++++++++++++-
 tests/test_api_base.py | 101 ++++++++++++++-----------------------------------
 2 files changed, 75 insertions(+), 74 deletions(-)

diff --git a/tests/helpers.py b/tests/helpers.py
index 81c24e3..c0d1607 100644
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -23,7 +23,9 @@ tests.
 import tempfile
 import random
 import itertools
-from bmaptools import BmapHelpers
+import hashlib
+import sys
+from bmaptools import BmapHelpers, BmapCopy, TransRead
 
 def _create_random_sparse_file(file_obj, size):
     """
@@ -248,3 +250,47 @@ def generate_test_files(max_size=4*1024*1024, directory=None, delete=True):
         blocks_cnt = (size + block_size - 1) / block_size
         yield (file_obj, size, [(0, blocks_cnt - 1)], [])
         file_obj.close()
+
+def calculate_chksum(file_path):
+    """Calculates checksum for the contents of file 'file_path'."""
+
+    file_obj = TransRead.TransRead(file_path)
+    hash_obj = hashlib.new("sha256")
+
+    chunk_size = 1024*1024
+
+    while True:
+        chunk = file_obj.read(chunk_size)
+        if not chunk:
+            break
+        hash_obj.update(chunk)
+
+    file_obj.close()
+    return hash_obj.hexdigest()
+
+def copy_and_verify_image(image, dest, bmap, image_chksum, image_size):
+    """
+    Copy image 'image' using bmap file 'bmap' to the destination file 'dest'
+    and verify the resulting image checksum.
+    """
+
+    f_image = TransRead.TransRead(image)
+    f_dest = open(dest, "w+")
+    if (bmap):
+        f_bmap = open(bmap, "r")
+    else:
+        f_bmap = None
+
+    writer = BmapCopy.BmapCopy(f_image, f_dest, f_bmap, image_size)
+    # Randomly decide whether we want the progress bar or not
+    if bool(random.getrandbits(1)):
+        writer.set_progress_indicator(sys.stdout, None)
+    writer.copy(bool(random.getrandbits(1)), bool(random.getrandbits(1)))
+
+    # Compare the original file and the copy are identical
+    assert calculate_chksum(dest) == image_chksum
+
+    if f_bmap:
+        f_bmap.close()
+    f_dest.close()
+    f_image.close()
diff --git a/tests/test_api_base.py b/tests/test_api_base.py
index 2a9ce10..1379e12 100644
--- a/tests/test_api_base.py
+++ b/tests/test_api_base.py
@@ -23,10 +23,8 @@ file and the copy and verifies that they are identical.
 # pylint: disable=R0904
 
 import os
-import sys
 import tempfile
 import filecmp
-import hashlib
 import itertools
 import random
 
@@ -36,8 +34,8 @@ try:
 except ImportError:
     import unittest
 
-import tests.helpers
-from bmaptools import BmapCreate, BmapCopy, Fiemap, TransRead
+from tests import helpers
+from bmaptools import BmapCreate, Fiemap, TransRead
 
 class Error(Exception):
     """A class for exceptions generated by this test."""
@@ -157,50 +155,6 @@ def _generate_compressed_files(file_path, delete=True):
 
     file_obj.close()
 
-def _calculate_chksum(file_path):
-    """Calculates checksum for the contents of file 'file_path'."""
-
-    file_obj = TransRead.TransRead(file_path)
-    hash_obj = hashlib.new("sha256")
-
-    chunk_size = 1024*1024
-
-    while True:
-        chunk = file_obj.read(chunk_size)
-        if not chunk:
-            break
-        hash_obj.update(chunk)
-
-    file_obj.close()
-    return hash_obj.hexdigest()
-
-def _copy_and_verify_image(image, dest, bmap, image_chksum, image_size):
-    """
-    Copy image 'image' using bmap file 'bmap' to the destination file 'dest'
-    and verify the resulting image checksum.
-    """
-
-    f_image = TransRead.TransRead(image)
-    f_dest = open(dest, "w+")
-    if (bmap):
-        f_bmap = open(bmap, "r")
-    else:
-        f_bmap = None
-
-    writer = BmapCopy.BmapCopy(f_image, f_dest, f_bmap, image_size)
-    # Randomly decide whether we want the progress bar or not
-    if bool(random.getrandbits(1)):
-        writer.set_progress_indicator(sys.stdout, None)
-    writer.copy(bool(random.getrandbits(1)), bool(random.getrandbits(1)))
-
-    # Compare the original file and the copy are identical
-    assert _calculate_chksum(dest) == image_chksum
-
-    if f_bmap:
-        f_bmap.close()
-    f_dest.close()
-    f_image.close()
-
 def _do_test(image, image_size, delete=True):
     """
     A basic test for the bmap creation and copying functionality. It first
@@ -232,7 +186,7 @@ def _do_test(image, image_size, delete=True):
                                           delete=delete, dir=directory,
                                           suffix=".bmap2")
 
-    image_chksum = _calculate_chksum(image)
+    image_chksum = helpers.calculate_chksum(image)
 
     #
     # Pass 1: generate the bmap, copy and compare
@@ -242,8 +196,8 @@ def _do_test(image, image_size, delete=True):
     creator = BmapCreate.BmapCreate(image, f_bmap1.name)
     creator.generate()
 
-    _copy_and_verify_image(image, f_copy.name, f_bmap1.name, image_chksum,
-                           image_size)
+    helpers.copy_and_verify_image(image, f_copy.name, f_bmap1.name,
+                                  image_chksum, image_size)
 
     # Make sure that holes in the copy are identical to holes in the random
     # sparse file.
@@ -255,8 +209,8 @@ def _do_test(image, image_size, delete=True):
 
     creator = BmapCreate.BmapCreate(image, f_bmap2)
     creator.generate()
-    _copy_and_verify_image(image, f_copy.name, f_bmap2.name, image_chksum,
-                           image_size)
+    helpers.copy_and_verify_image(image, f_copy.name, f_bmap2.name,
+                                  image_chksum, image_size)
     _compare_holes(image, f_copy.name)
 
     # Make sure the bmap files generated at pass 1 and pass 2 are identical
@@ -267,44 +221,45 @@ def _do_test(image, image_size, delete=True):
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, image_size)
+        helpers.copy_and_verify_image(compressed, f_copy.name,
+                                      f_bmap1.name, image_chksum, image_size)
 
         # Test without setting the size
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, None)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, None)
 
         # Append a "file:" prefixe to make BmapCopy use urllib
         compressed = "file:" + compressed
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, image_size)
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, None)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, image_size)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, None)
 
     #
     # Pass 5: copy without bmap and make sure it is identical to the original
     # file.
 
-    _copy_and_verify_image(image, f_copy.name, None, image_chksum, image_size)
-    _copy_and_verify_image(image, f_copy.name, None, image_chksum, None)
+    helpers.copy_and_verify_image(image, f_copy.name, None, image_chksum,
+                                  image_size)
+    helpers.copy_and_verify_image(image, f_copy.name, None, image_chksum, None)
 
     #
     # Pass 6: test compressed files copying without bmap
     #
 
     for compressed in _generate_compressed_files(image, delete=delete):
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, image_size)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, image_size)
 
         # Test without setting the size
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, None)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, None)
 
         # Append a "file:" prefix to make BmapCopy use urllib
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, image_size)
-        _copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
-                               image_chksum, None)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, image_size)
+        helpers.copy_and_verify_image(compressed, f_copy.name, f_bmap1.name,
+                                      image_chksum, None)
 
     # Close temporary files, which will also remove them
     f_copy.close()
@@ -330,8 +285,8 @@ class TestCreateCopy(unittest.TestCase):
         # FIEMAP).
         directory = '.'
 
-        iterator = tests.helpers.generate_test_files(delete=delete,
-                                                     directory=directory)
+        iterator = helpers.generate_test_files(delete=delete,
+                                               directory=directory)
         for f_image, image_size, _, _ in iterator:
             assert image_size == os.path.getsize(f_image.name)
             _do_test(f_image.name, image_size, delete=delete)
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 03:05:34 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 10:05:34 +0200
Subject: [bmap-tools] [PATCH 14/16] tests: add test data
In-Reply-To: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
References: <1383552336-9704-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383552336-9704-14-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Add a randomly-generated test image and bmap files of different formats. The
intention is to add a test case which verifies backward-compatibility.

Change-Id: I4cfbb5b16b252bbde3292aabcc7b432c3ee23e9e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test-data/test.image.bmap.v1.2 |  90 ++++++++++++++++++++++++++++++++
 tests/test-data/test.image.bmap.v1.3 |  94 +++++++++++++++++++++++++++++++++
 tests/test-data/test.image.bmap.v1.4 |  97 +++++++++++++++++++++++++++++++++++
 tests/test-data/test.image.bmap.v2.0 |  97 +++++++++++++++++++++++++++++++++++
 tests/test-data/test.image.gz        | Bin 0 -> 1612 bytes
 5 files changed, 378 insertions(+)
 create mode 100644 tests/test-data/test.image.bmap.v1.2
 create mode 100644 tests/test-data/test.image.bmap.v1.3
 create mode 100644 tests/test-data/test.image.bmap.v1.4
 create mode 100644 tests/test-data/test.image.bmap.v2.0
 create mode 100644 tests/test-data/test.image.gz

diff --git a/tests/test-data/test.image.bmap.v1.2 b/tests/test-data/test.image.bmap.v1.2
new file mode 100644
index 0000000..9764bbb
--- /dev/null
+++ b/tests/test-data/test.image.bmap.v1.2
@@ -0,0 +1,90 @@
+<?xml version="1.0" ?>
+<!-- This file contains the block map for an image file, which is basically
+     a list of useful (mapped) block numbers in the image file. In other words,
+     it lists only those blocks which contain data (boot sector, partition
+     table, file-system metadata, files, directories, extents, etc). These
+     blocks have to be copied to the target device. The other blocks do not
+     contain any useful data and do not have to be copied to the target
+     device.
+
+     The block map an optimization which allows to copy or flash the image to
+     the image quicker than copying of flashing the entire image. This is
+     because with bmap less data is copied: <MappedBlocksCount> blocks instead
+     of <BlocksCount> blocks.
+
+     Besides the machine-readable data, this file contains useful commentaries
+     which contain human-readable information like image size, percentage of
+     mapped data, etc.
+
+     The 'version' attribute is the block map file format version in the
+     'major.minor' format. The version major number is increased whenever an
+     incompatible block map format change is made. The minor number changes
+     in case of minor backward-compatible changes. -->
+
+<bmap version="1.2">
+    <!-- Image size in bytes (802.5 KiB) -->
+    <ImageSize> 821752 </ImageSize>
+
+    <!-- Size of a block in bytes -->
+    <BlockSize> 4096 </BlockSize>
+
+    <!-- Count of blocks in the image file -->
+    <BlocksCount> 201 </BlocksCount>
+
+    <!-- The block map which consists of elements which may either be a
+         range of blocks or a single block. The 'sha1' attribute (if present)
+         is the SHA1 checksum of this blocks range. -->
+    <BlockMap>
+        <Range sha1="94789636db14cdb8929133e7b8d3a158837e2e5a"> 0-1 </Range>
+        <Range sha1="4ad09a593ed833ece30d808477cb2f43fe7e96fe"> 3-5 </Range>
+        <Range sha1="35bea6bfaa394f4cf030dc0060befed0ab04c55c"> 9-10 </Range>
+        <Range sha1="21338baaedb0976efbb075736820f6bd6ae43b97"> 12 </Range>
+        <Range sha1="80086eaaf78f8eb52c80ea3a559be9958b4dee82"> 15-18 </Range>
+        <Range sha1="3e7c00626f9db9451fe0b55fa2a8cf2a81d9830a"> 20 </Range>
+        <Range sha1="458e9562df17d15ab3aa055ae681eec8f2d4c0fd"> 22 </Range>
+        <Range sha1="624066e4cb2838443ab2b1178bd4179d9d62acb8"> 24 </Range>
+        <Range sha1="7aa2876168020797c7060e241436c82889d66da9"> 30-32 </Range>
+        <Range sha1="5ae8333f60e55f8891e21c90e89ea603be421ca3"> 34-35 </Range>
+        <Range sha1="d5af843e9f9a1154ae610a30c71698089a298215"> 40 </Range>
+        <Range sha1="a98956e9a1801e612a1c07a0aa0f0c57be86c671"> 42-43 </Range>
+        <Range sha1="a121482d334a2bd9a42c444f99fe365ddf83a758"> 45 </Range>
+        <Range sha1="f619721549e15de1d6d5f7be85f9942140ae0d16"> 47 </Range>
+        <Range sha1="196df31f78d0e3cb623d4496e6237c4e9683cd81"> 49-50 </Range>
+        <Range sha1="f4cba9e7822dfb5206874e2a952681c53c513a71"> 52-53 </Range>
+        <Range sha1="7794743dc660f6ef29577b01af7146f6ad682a89"> 55-56 </Range>
+        <Range sha1="557fac47c5ae2476c04054b73680e531eeba116e"> 60-63 </Range>
+        <Range sha1="dba433052c7009aa4cbfdf81dabc17b285b8d087"> 65-67 </Range>
+        <Range sha1="55ab7ef8c797e942ea110aadcd22b55b19734ad3"> 70 </Range>
+        <Range sha1="2797a838edf16114b587d349da5b6c999290b02f"> 72 </Range>
+        <Range sha1="594cc8b274a0a92f2489765ca35545e1118d2375"> 78-80 </Range>
+        <Range sha1="e5f421fc2bdcdddd1f1f7081355091d5ec116930"> 82-83 </Range>
+        <Range sha1="c152f2a6e5eb08eae7122eabe8de71abb67fa810"> 85 </Range>
+        <Range sha1="67de1912c30546a2a0debca8ae41a0605a6eaa1a"> 88 </Range>
+        <Range sha1="b87c912352c16d33f056be47e26ece50375e560e"> 90-91 </Range>
+        <Range sha1="2976768755f1b5dd3a6386f8ade4887022024660"> 96 </Range>
+        <Range sha1="045ae769f1de3d99b35362b7e309609b6ff39886"> 98-105 </Range>
+        <Range sha1="dc6ba49fd6639a416c515575cd3c1a18bbbd77d7"> 111 </Range>
+        <Range sha1="b489f230fa3d578aeeb64d15d3ceba9c3f7fac18"> 114-116 </Range>
+        <Range sha1="29645cb904ec22f017fd24a8d5a188329cca1bdf"> 119-133 </Range>
+        <Range sha1="b25d6cd7311fd4b132e085b4649ca6af663f290b"> 135 </Range>
+        <Range sha1="8cba06dbce2c88a2d8370ab28c696e5871a3e80f"> 137 </Range>
+        <Range sha1="c5143c106c7cdb1324e55b862d8f3a7c7773fbd2"> 140 </Range>
+        <Range sha1="386e2a53aff173f7c8432a33ba9f115cb0f3f2dd"> 142-144 </Range>
+        <Range sha1="9536bce6180a813bf09b555bef8cc385c2250155"> 146-147 </Range>
+        <Range sha1="0b615f57ce2368bc58a03adab323d1fc40182ada"> 150-151 </Range>
+        <Range sha1="fa2982eb55d8302d534719e2099405e3cf10ab20"> 155 </Range>
+        <Range sha1="2867488d3abd8f16805c19e0ec639aebff9ff5b2"> 157 </Range>
+        <Range sha1="98cbb098a082e8c9c42945c38f057cc911b2e892"> 159-160 </Range>
+        <Range sha1="2627075946035748c9e341adc915edfdb1a97bc8"> 163-174 </Range>
+        <Range sha1="1a34d6fc1e86e38d5e5c4e3fb50e7f9b2e07870d"> 177 </Range>
+        <Range sha1="e170e81e69f8301d331382378823b5b2a577b670"> 181-186 </Range>
+        <Range sha1="1d463c99d29fb207fc7fd36bb297218294b851af"> 188-189 </Range>
+        <Range sha1="39652579027a70352fb8ee9ba94fe5bff486de6e"> 191 </Range>
+        <Range sha1="bee29c4d8f84c068a1be05a22aaf20c2e1f5fb92"> 193 </Range>
+        <Range sha1="959bba0c61e2e25d87af92e882d2d7d851376e72"> 195 </Range>
+        <Range sha1="dbca3770aa48e9e05629f8890d1b2330be709b8a"> 198-199 </Range>
+    </BlockMap>
+
+    <!-- Count of mapped blocks (468.0 KiB or 58.2% mapped) -->
+    <MappedBlocksCount> 117 </MappedBlocksCount>
+</bmap>
diff --git a/tests/test-data/test.image.bmap.v1.3 b/tests/test-data/test.image.bmap.v1.3
new file mode 100644
index 0000000..5c20964
--- /dev/null
+++ b/tests/test-data/test.image.bmap.v1.3
@@ -0,0 +1,94 @@
+<?xml version="1.0" ?>
+<!-- This file contains the block map for an image file, which is basically
+     a list of useful (mapped) block numbers in the image file. In other words,
+     it lists only those blocks which contain data (boot sector, partition
+     table, file-system metadata, files, directories, extents, etc). These
+     blocks have to be copied to the target device. The other blocks do not
+     contain any useful data and do not have to be copied to the target
+     device.
+
+     The block map an optimization which allows to copy or flash the image to
+     the image quicker than copying of flashing the entire image. This is
+     because with bmap less data is copied: <MappedBlocksCount> blocks instead
+     of <BlocksCount> blocks.
+
+     Besides the machine-readable data, this file contains useful commentaries
+     which contain human-readable information like image size, percentage of
+     mapped data, etc.
+
+     The 'version' attribute is the block map file format version in the
+     'major.minor' format. The version major number is increased whenever an
+     incompatible block map format change is made. The minor number changes
+     in case of minor backward-compatible changes. -->
+
+<bmap version="1.3">
+    <!-- Image size in bytes: 802.5 KiB -->
+    <ImageSize> 821752 </ImageSize>
+
+    <!-- Size of a block in bytes -->
+    <BlockSize> 4096 </BlockSize>
+
+    <!-- Count of blocks in the image file -->
+    <BlocksCount> 201 </BlocksCount>
+
+    <!-- Count of mapped blocks: 468.0 KiB or 58.2%    -->
+    <MappedBlocksCount> 117    </MappedBlocksCount>
+
+    <!-- The checksum of this bmap file. When it is calculated, the value of
+         the SHA1 checksum has be zero (40 ASCII "0" symbols). -->
+    <BmapFileSHA1> e235f7cd0c6b8c07a2e6f2538510fb763e7790a6 </BmapFileSHA1>
+
+    <!-- The block map which consists of elements which may either be a
+         range of blocks or a single block. The 'sha1' attribute (if present)
+         is the SHA1 checksum of this blocks range. -->
+    <BlockMap>
+        <Range sha1="94789636db14cdb8929133e7b8d3a158837e2e5a"> 0-1 </Range>
+        <Range sha1="4ad09a593ed833ece30d808477cb2f43fe7e96fe"> 3-5 </Range>
+        <Range sha1="35bea6bfaa394f4cf030dc0060befed0ab04c55c"> 9-10 </Range>
+        <Range sha1="21338baaedb0976efbb075736820f6bd6ae43b97"> 12 </Range>
+        <Range sha1="80086eaaf78f8eb52c80ea3a559be9958b4dee82"> 15-18 </Range>
+        <Range sha1="3e7c00626f9db9451fe0b55fa2a8cf2a81d9830a"> 20 </Range>
+        <Range sha1="458e9562df17d15ab3aa055ae681eec8f2d4c0fd"> 22 </Range>
+        <Range sha1="624066e4cb2838443ab2b1178bd4179d9d62acb8"> 24 </Range>
+        <Range sha1="7aa2876168020797c7060e241436c82889d66da9"> 30-32 </Range>
+        <Range sha1="5ae8333f60e55f8891e21c90e89ea603be421ca3"> 34-35 </Range>
+        <Range sha1="d5af843e9f9a1154ae610a30c71698089a298215"> 40 </Range>
+        <Range sha1="a98956e9a1801e612a1c07a0aa0f0c57be86c671"> 42-43 </Range>
+        <Range sha1="a121482d334a2bd9a42c444f99fe365ddf83a758"> 45 </Range>
+        <Range sha1="f619721549e15de1d6d5f7be85f9942140ae0d16"> 47 </Range>
+        <Range sha1="196df31f78d0e3cb623d4496e6237c4e9683cd81"> 49-50 </Range>
+        <Range sha1="f4cba9e7822dfb5206874e2a952681c53c513a71"> 52-53 </Range>
+        <Range sha1="7794743dc660f6ef29577b01af7146f6ad682a89"> 55-56 </Range>
+        <Range sha1="557fac47c5ae2476c04054b73680e531eeba116e"> 60-63 </Range>
+        <Range sha1="dba433052c7009aa4cbfdf81dabc17b285b8d087"> 65-67 </Range>
+        <Range sha1="55ab7ef8c797e942ea110aadcd22b55b19734ad3"> 70 </Range>
+        <Range sha1="2797a838edf16114b587d349da5b6c999290b02f"> 72 </Range>
+        <Range sha1="594cc8b274a0a92f2489765ca35545e1118d2375"> 78-80 </Range>
+        <Range sha1="e5f421fc2bdcdddd1f1f7081355091d5ec116930"> 82-83 </Range>
+        <Range sha1="c152f2a6e5eb08eae7122eabe8de71abb67fa810"> 85 </Range>
+        <Range sha1="67de1912c30546a2a0debca8ae41a0605a6eaa1a"> 88 </Range>
+        <Range sha1="b87c912352c16d33f056be47e26ece50375e560e"> 90-91 </Range>
+        <Range sha1="2976768755f1b5dd3a6386f8ade4887022024660"> 96 </Range>
+        <Range sha1="045ae769f1de3d99b35362b7e309609b6ff39886"> 98-105 </Range>
+        <Range sha1="dc6ba49fd6639a416c515575cd3c1a18bbbd77d7"> 111 </Range>
+        <Range sha1="b489f230fa3d578aeeb64d15d3ceba9c3f7fac18"> 114-116 </Range>
+        <Range sha1="29645cb904ec22f017fd24a8d5a188329cca1bdf"> 119-133 </Range>
+        <Range sha1="b25d6cd7311fd4b132e085b4649ca6af663f290b"> 135 </Range>
+        <Range sha1="8cba06dbce2c88a2d8370ab28c696e5871a3e80f"> 137 </Range>
+        <Range sha1="c5143c106c7cdb1324e55b862d8f3a7c7773fbd2"> 140 </Range>
+        <Range sha1="386e2a53aff173f7c8432a33ba9f115cb0f3f2dd"> 142-144 </Range>
+        <Range sha1="9536bce6180a813bf09b555bef8cc385c2250155"> 146-147 </Range>
+        <Range sha1="0b615f57ce2368bc58a03adab323d1fc40182ada"> 150-151 </Range>
+        <Range sha1="fa2982eb55d8302d534719e2099405e3cf10ab20"> 155 </Range>
+        <Range sha1="2867488d3abd8f16805c19e0ec639aebff9ff5b2"> 157 </Range>
+        <Range sha1="98cbb098a082e8c9c42945c38f057cc911b2e892"> 159-160 </Range>
+        <Range sha1="2627075946035748c9e341adc915edfdb1a97bc8"> 163-174 </Range>
+        <Range sha1="1a34d6fc1e86e38d5e5c4e3fb50e7f9b2e07870d"> 177 </Range>
+        <Range sha1="e170e81e69f8301d331382378823b5b2a577b670"> 181-186 </Range>
+        <Range sha1="1d463c99d29fb207fc7fd36bb297218294b851af"> 188-189 </Range>
+        <Range sha1="39652579027a70352fb8ee9ba94fe5bff486de6e"> 191 </Range>
+        <Range sha1="bee29c4d8f84c068a1be05a22aaf20c2e1f5fb92"> 193 </Range>
+        <Range sha1="959bba0c61e2e25d87af92e882d2d7d851376e72"> 195 </Range>
+        <Range sha1="dbca3770aa48e9e05629f8890d1b2330be709b8a"> 198-199 </Range>
+    </BlockMap>
+</bmap>
diff --git a/tests/test-data/test.image.bmap.v1.4 b/tests/test-data/test.image.bmap.v1.4
new file mode 100644
index 0000000..b1efaea
--- /dev/null
+++ b/tests/test-data/test.image.bmap.v1.4
@@ -0,0 +1,97 @@
+<?xml version="1.0" ?>
+<!-- This file contains the block map for an image file, which is basically
+     a list of useful (mapped) block numbers in the image file. In other words,
+     it lists only those blocks which contain data (boot sector, partition
+     table, file-system metadata, files, directories, extents, etc). These
+     blocks have to be copied to the target device. The other blocks do not
+     contain any useful data and do not have to be copied to the target
+     device.
+
+     The block map an optimization which allows to copy or flash the image to
+     the image quicker than copying of flashing the entire image. This is
+     because with bmap less data is copied: <MappedBlocksCount> blocks instead
+     of <BlocksCount> blocks.
+
+     Besides the machine-readable data, this file contains useful commentaries
+     which contain human-readable information like image size, percentage of
+     mapped data, etc.
+
+     The 'version' attribute is the block map file format version in the
+     'major.minor' format. The version major number is increased whenever an
+     incompatible block map format change is made. The minor number changes
+     in case of minor backward-compatible changes. -->
+
+<bmap version="1.4">
+    <!-- Image size in bytes: 802.5 KiB -->
+    <ImageSize> 821752 </ImageSize>
+
+    <!-- Size of a block in bytes -->
+    <BlockSize> 4096 </BlockSize>
+
+    <!-- Count of blocks in the image file -->
+    <BlocksCount> 201 </BlocksCount>
+
+    <!-- Count of mapped blocks: 468.0 KiB or 58.2%    -->
+    <MappedBlocksCount> 117    </MappedBlocksCount>
+
+    <!-- Type of checksum used in this file -->
+    <ChecksumType> sha256 </ChecksumType>
+
+    <!-- The checksum of this bmap file. When it is calculated, the value of
+         the checksum has be zero (all ASCII "0" symbols).  -->
+    <BmapFileChecksum> 4310fd457a88d307abeeb593a7888e1fa3cae0cfc01d905158967c904c5375e5 </BmapFileChecksum>
+
+    <!-- The block map which consists of elements which may either be a
+         range of blocks or a single block. The 'chksum' attribute
+         (if present) is the checksum of this blocks range. -->
+    <BlockMap>
+        <Range chksum="9eaf19215d55d23de1be1fe4bed4a95bfe620a404352fd06e782738fff58e500"> 0-1 </Range>
+        <Range chksum="e8a26f49a71262870f8294a73f40f122d622fd70fb82bef01c0322785e9fd6b2"> 3-5 </Range>
+        <Range chksum="9251ec146af47d2db881b4beabf39e8a09f400e9fdd587080ba872be250bfa66"> 9-10 </Range>
+        <Range chksum="1f82b9594a9385934f7d62ad2c95b680adb0a715311da24a592353f4eedc9196"> 12 </Range>
+        <Range chksum="f9abfd404b01ee6cc3aae89c0b2d32595d174067414d4c7ed347e01027e5198a"> 15-18 </Range>
+        <Range chksum="7a6304c0e75f1db501e01ceaf03f925adb5046de4d8f21b4453ac1f34ad9a04a"> 20 </Range>
+        <Range chksum="7efcfe912e7a0d603823ecd387040816d718dbe356abc33d75c7a4074b5166e6"> 22 </Range>
+        <Range chksum="fa89f8bfa8647f253a65547f2fec7da8220beb2abf3e83387cfde4a5adc2e371"> 24 </Range>
+        <Range chksum="b9b49e2335300e6c9cbc41f023b95262f1e8997a328cefac4e2ee98dbffe2876"> 30-32 </Range>
+        <Range chksum="73d8a6f96db05475c61905707405707d881f975c6a20c4b3860e5da9cba7d0d6"> 34-35 </Range>
+        <Range chksum="72ecd33adba1ce49626179d66500581e96d56b26882d1882fa73b0a8973a54ae"> 40 </Range>
+        <Range chksum="343b15a5df354fa45c3e3a0755c2c0c65e1e8b6b8566c96ee018e3fdf22b7fea"> 42-43 </Range>
+        <Range chksum="67632b2bd1704a7d3535fcd1224d1cc2e5defe5f8503416aba108deb0a8c5195"> 45 </Range>
+        <Range chksum="567853b6db8ba8fc6b74b68bd054c4570c94c563421c791d4282271b218d1970"> 47 </Range>
+        <Range chksum="4c6bea2040e058ac183e4f89740b6519e0acf7690cc8c1e3a92d4cadde18afc1"> 49-50 </Range>
+        <Range chksum="a6923635cd40a502321dad8127db631eaae77af1adbd7c3af28e4c47a6cc17b4"> 52-53 </Range>
+        <Range chksum="754d4a59d22aa4e635c12da22437a2980a47cac1ffd3e9319a16e935517a3d28"> 55-56 </Range>
+        <Range chksum="d03670c239e2f30ef065da9848cef9f5386543d1d25bdce3b6b060f846c02336"> 60-63 </Range>
+        <Range chksum="7c1092388c70a7f20eb73245cb3253c8a3e94fe4727c89ccc8175a83931f30c6"> 65-67 </Range>
+        <Range chksum="69f6917acf67c408c6f904a0b4d1e4731fc63fcebf96b863c46e1f1345b89d5f"> 70 </Range>
+        <Range chksum="584951a12f0a64d605cf5c40ecc5c35fea09320215b078ff1409a6c8cb53d779"> 72 </Range>
+        <Range chksum="983775dfe44cf57c9c16f388fb0a0feb70b1fe1df5c15ccadf06fb52c687e090"> 78-80 </Range>
+        <Range chksum="7792004ce7de28e55cee7ad365760e2717ed10ba85bd732fee821eada3ed851f"> 82-83 </Range>
+        <Range chksum="9422e34ab590d8396203744c680411526d5ef1a9e75815a4c5f26fb45233c654"> 85 </Range>
+        <Range chksum="bb8fbdd5ee992341e0344284f9ef2fd60f7fcc66022d5ddf5167cb4dbf4d77e0"> 88 </Range>
+        <Range chksum="d7c2fe248a1f4faba0beb7ed7982721af09e43d39ba031adc47b92b7a512c2a6"> 90-91 </Range>
+        <Range chksum="462a930c88a2e3c5319559feb39389c64b6c21dc2cb68552e7499208bd7103aa"> 96 </Range>
+        <Range chksum="16dab0a1420bcf3fb47676ad3cf63d37018359bf8d74f69bda6deb584defe017"> 98-105 </Range>
+        <Range chksum="dde77bda930499f1ca1f5926f3bbce02628312744035031ac985953c2b992fd6"> 111 </Range>
+        <Range chksum="e53e4350a4a6f24d8535fe1889fd51620eb767ba9edf1f54e4716075024b4930"> 114-116 </Range>
+        <Range chksum="483e868aa797d861b15623755e99e195b9bd2ae21cf65e13cbc3a31196ff1399"> 119-133 </Range>
+        <Range chksum="c246d08bfb4c637c8ba4bc36de01c6d0a0fa1ccb6ce5073f36bee03553930dec"> 135 </Range>
+        <Range chksum="112bf5a172bddfdaa3547ec75cec2e06f53469092877d3235f4875a5f2d27a6a"> 137 </Range>
+        <Range chksum="cb0bd3e46ba71a0107e6565fcf9dd74e77890ef8e44233c2e622083264e32ee5"> 140 </Range>
+        <Range chksum="84cd195fb6ba711efaab2c726a4b022b59c35411288ea195ac8de7c55770b8a0"> 142-144 </Range>
+        <Range chksum="4581295e35fcc3192c67516f296833a0d7efb60c682ec5eb41d478fd471c7d37"> 146-147 </Range>
+        <Range chksum="265121e906434c8f824116152a774d6f29d64dd9d824742437a1b43337376913"> 150-151 </Range>
+        <Range chksum="751a8b29e92c1af64c0298c14e823b36c2e595d2b8f40e88a74a9cfec7969178"> 155 </Range>
+        <Range chksum="f5323d152b5fa270485f8405d14697a3c85e7ebd81191b4d63472537d4e12820"> 157 </Range>
+        <Range chksum="bfb2148640de1bfe102cd005978ccc06ee4d1da40b4f9ba62c0ed1ce62c08370"> 159-160 </Range>
+        <Range chksum="dafa3fbd9fb00807231520ed080d412b16e7a64f2f22e5252ba435bba13f3094"> 163-174 </Range>
+        <Range chksum="75678fa3017705a5c4015668beb2f69a88c7ad1ca981468ad9bac0322db90c98"> 177 </Range>
+        <Range chksum="1e11b14396428302f3e6a6b770a5d55bd4211fd952a9acf5acad8996257615a9"> 181-186 </Range>
+        <Range chksum="c9bb5fa33c4dee6cb6f846017dfca8ab33e5b211cb1dfa1a92fc26ce21ebf258"> 188-189 </Range>
+        <Range chksum="229690ebe7f88e243b5b1ba3a87b20fa21cb63c52c31315c65c9075074be9384"> 191 </Range>
+        <Range chksum="38997ca7da7129a382db6138e8f42102a22a482f859ddb9bdc2fba8a44fe1965"> 193 </Range>
+        <Range chksum="af30597da547c8c0cf124b33159644f089ba2e172bddc1fe2864d80183565900"> 195 </Range>
+        <Range chksum="cb732fc3f3a0f81f6a761a534201c05549c8efe4a92630ccd24241f72d7d618c"> 198-199 </Range>
+    </BlockMap>
+</bmap>
diff --git a/tests/test-data/test.image.bmap.v2.0 b/tests/test-data/test.image.bmap.v2.0
new file mode 100644
index 0000000..a9df54d
--- /dev/null
+++ b/tests/test-data/test.image.bmap.v2.0
@@ -0,0 +1,97 @@
+<?xml version="1.0" ?>
+<!-- This file contains the block map for an image file, which is basically
+     a list of useful (mapped) block numbers in the image file. In other words,
+     it lists only those blocks which contain data (boot sector, partition
+     table, file-system metadata, files, directories, extents, etc). These
+     blocks have to be copied to the target device. The other blocks do not
+     contain any useful data and do not have to be copied to the target
+     device.
+
+     The block map an optimization which allows to copy or flash the image to
+     the image quicker than copying of flashing the entire image. This is
+     because with bmap less data is copied: <MappedBlocksCount> blocks instead
+     of <BlocksCount> blocks.
+
+     Besides the machine-readable data, this file contains useful commentaries
+     which contain human-readable information like image size, percentage of
+     mapped data, etc.
+
+     The 'version' attribute is the block map file format version in the
+     'major.minor' format. The version major number is increased whenever an
+     incompatible block map format change is made. The minor number changes
+     in case of minor backward-compatible changes. -->
+
+<bmap version="2.0">
+    <!-- Image size in bytes: 802.5 KiB -->
+    <ImageSize> 821752 </ImageSize>
+
+    <!-- Size of a block in bytes -->
+    <BlockSize> 4096 </BlockSize>
+
+    <!-- Count of blocks in the image file -->
+    <BlocksCount> 201 </BlocksCount>
+
+    <!-- Count of mapped blocks: 468.0 KiB or 58.2%    -->
+    <MappedBlocksCount> 117    </MappedBlocksCount>
+
+    <!-- Type of checksum used in this file -->
+    <ChecksumType> sha256 </ChecksumType>
+
+    <!-- The checksum of this bmap file. When it is calculated, the value of
+         the checksum has be zero (all ASCII "0" symbols).  -->
+    <BmapFileChecksum> d9cf7d44790d04fcbb089c5eeec7700e9233439ab6e4bd759035906e20f90070 </BmapFileChecksum>
+
+    <!-- The block map which consists of elements which may either be a
+         range of blocks or a single block. The 'chksum' attribute
+         (if present) is the checksum of this blocks range. -->
+    <BlockMap>
+        <Range chksum="9eaf19215d55d23de1be1fe4bed4a95bfe620a404352fd06e782738fff58e500"> 0-1 </Range>
+        <Range chksum="e8a26f49a71262870f8294a73f40f122d622fd70fb82bef01c0322785e9fd6b2"> 3-5 </Range>
+        <Range chksum="9251ec146af47d2db881b4beabf39e8a09f400e9fdd587080ba872be250bfa66"> 9-10 </Range>
+        <Range chksum="1f82b9594a9385934f7d62ad2c95b680adb0a715311da24a592353f4eedc9196"> 12 </Range>
+        <Range chksum="f9abfd404b01ee6cc3aae89c0b2d32595d174067414d4c7ed347e01027e5198a"> 15-18 </Range>
+        <Range chksum="7a6304c0e75f1db501e01ceaf03f925adb5046de4d8f21b4453ac1f34ad9a04a"> 20 </Range>
+        <Range chksum="7efcfe912e7a0d603823ecd387040816d718dbe356abc33d75c7a4074b5166e6"> 22 </Range>
+        <Range chksum="fa89f8bfa8647f253a65547f2fec7da8220beb2abf3e83387cfde4a5adc2e371"> 24 </Range>
+        <Range chksum="b9b49e2335300e6c9cbc41f023b95262f1e8997a328cefac4e2ee98dbffe2876"> 30-32 </Range>
+        <Range chksum="73d8a6f96db05475c61905707405707d881f975c6a20c4b3860e5da9cba7d0d6"> 34-35 </Range>
+        <Range chksum="72ecd33adba1ce49626179d66500581e96d56b26882d1882fa73b0a8973a54ae"> 40 </Range>
+        <Range chksum="343b15a5df354fa45c3e3a0755c2c0c65e1e8b6b8566c96ee018e3fdf22b7fea"> 42-43 </Range>
+        <Range chksum="67632b2bd1704a7d3535fcd1224d1cc2e5defe5f8503416aba108deb0a8c5195"> 45 </Range>
+        <Range chksum="567853b6db8ba8fc6b74b68bd054c4570c94c563421c791d4282271b218d1970"> 47 </Range>
+        <Range chksum="4c6bea2040e058ac183e4f89740b6519e0acf7690cc8c1e3a92d4cadde18afc1"> 49-50 </Range>
+        <Range chksum="a6923635cd40a502321dad8127db631eaae77af1adbd7c3af28e4c47a6cc17b4"> 52-53 </Range>
+        <Range chksum="754d4a59d22aa4e635c12da22437a2980a47cac1ffd3e9319a16e935517a3d28"> 55-56 </Range>
+        <Range chksum="d03670c239e2f30ef065da9848cef9f5386543d1d25bdce3b6b060f846c02336"> 60-63 </Range>
+        <Range chksum="7c1092388c70a7f20eb73245cb3253c8a3e94fe4727c89ccc8175a83931f30c6"> 65-67 </Range>
+        <Range chksum="69f6917acf67c408c6f904a0b4d1e4731fc63fcebf96b863c46e1f1345b89d5f"> 70 </Range>
+        <Range chksum="584951a12f0a64d605cf5c40ecc5c35fea09320215b078ff1409a6c8cb53d779"> 72 </Range>
+        <Range chksum="983775dfe44cf57c9c16f388fb0a0feb70b1fe1df5c15ccadf06fb52c687e090"> 78-80 </Range>
+        <Range chksum="7792004ce7de28e55cee7ad365760e2717ed10ba85bd732fee821eada3ed851f"> 82-83 </Range>
+        <Range chksum="9422e34ab590d8396203744c680411526d5ef1a9e75815a4c5f26fb45233c654"> 85 </Range>
+        <Range chksum="bb8fbdd5ee992341e0344284f9ef2fd60f7fcc66022d5ddf5167cb4dbf4d77e0"> 88 </Range>
+        <Range chksum="d7c2fe248a1f4faba0beb7ed7982721af09e43d39ba031adc47b92b7a512c2a6"> 90-91 </Range>
+        <Range chksum="462a930c88a2e3c5319559feb39389c64b6c21dc2cb68552e7499208bd7103aa"> 96 </Range>
+        <Range chksum="16dab0a1420bcf3fb47676ad3cf63d37018359bf8d74f69bda6deb584defe017"> 98-105 </Range>
+        <Range chksum="dde77bda930499f1ca1f5926f3bbce02628312744035031ac985953c2b992fd6"> 111 </Range>
+        <Range chksum="e53e4350a4a6f24d8535fe1889fd51620eb767ba9edf1f54e4716075024b4930"> 114-116 </Range>
+        <Range chksum="483e868aa797d861b15623755e99e195b9bd2ae21cf65e13cbc3a31196ff1399"> 119-133 </Range>
+        <Range chksum="c246d08bfb4c637c8ba4bc36de01c6d0a0fa1ccb6ce5073f36bee03553930dec"> 135 </Range>
+        <Range chksum="112bf5a172bddfdaa3547ec75cec2e06f53469092877d3235f4875a5f2d27a6a"> 137 </Range>
+        <Range chksum="cb0bd3e46ba71a0107e6565fcf9dd74e77890ef8e44233c2e622083264e32ee5"> 140 </Range>
+        <Range chksum="84cd195fb6ba711efaab2c726a4b022b59c35411288ea195ac8de7c55770b8a0"> 142-144 </Range>
+        <Range chksum="4581295e35fcc3192c67516f296833a0d7efb60c682ec5eb41d478fd471c7d37"> 146-147 </Range>
+        <Range chksum="265121e906434c8f824116152a774d6f29d64dd9d824742437a1b43337376913"> 150-151 </Range>
+        <Range chksum="751a8b29e92c1af64c0298c14e823b36c2e595d2b8f40e88a74a9cfec7969178"> 155 </Range>
+        <Range chksum="f5323d152b5fa270485f8405d14697a3c85e7ebd81191b4d63472537d4e12820"> 157 </Range>
+        <Range chksum="bfb2148640de1bfe102cd005978ccc06ee4d1da40b4f9ba62c0ed1ce62c08370"> 159-160 </Range>
+        <Range chksum="dafa3fbd9fb00807231520ed080d412b16e7a64f2f22e5252ba435bba13f3094"> 163-174 </Range>
+        <Range chksum="75678fa3017705a5c4015668beb2f69a88c7ad1ca981468ad9bac0322db90c98"> 177 </Range>
+        <Range chksum="1e11b14396428302f3e6a6b770a5d55bd4211fd952a9acf5acad8996257615a9"> 181-186 </Range>
+        <Range chksum="c9bb5fa33c4dee6cb6f846017dfca8ab33e5b211cb1dfa1a92fc26ce21ebf258"> 188-189 </Range>
+        <Range chksum="229690ebe7f88e243b5b1ba3a87b20fa21cb63c52c31315c65c9075074be9384"> 191 </Range>
+        <Range chksum="38997ca7da7129a382db6138e8f42102a22a482f859ddb9bdc2fba8a44fe1965"> 193 </Range>
+        <Range chksum="af30597da547c8c0cf124b33159644f089ba2e172bddc1fe2864d80183565900"> 195 </Range>
+        <Range chksum="cb732fc3f3a0f81f6a761a534201c05549c8efe4a92630ccd24241f72d7d618c"> 198-199 </Range>
+    </BlockMap>
+</bmap>
diff --git a/tests/test-data/test.image.gz b/tests/test-data/test.image.gz
new file mode 100644
index 0000000000000000000000000000000000000000..66d734108ab7c9549783ba259a6b80a7c2b85e84
GIT binary patch
literal 1612
zcmZuxYfuwc6h3*Fpz at F+g7Oeign}X!2O%0<P!WO(RX~a&OIrmMC};$XrtDbM3P^b>
z4ydUT1q97hEXd2WPNkgziA#kJG=MOpETE2Mc}Q*T-3 at WH?fG%{Jic?zckj7(7iJn5
ze187WHsDG39Fn^3i;s at k1JAVenI)l?!oK)8ey(DkeXMdxWVJz2h{xH7et|xG(`CmM
z{WJJKY&FVAb*b!-s2*<JrZni1_*a|x%xSgXSr(ppx@#oomyzc!DNVh1*0x0 at Hcd3t
zxVOf|a{$!XRwFx`&j2KzT^UjZFoN4V67x!T&Oqu{ujT;kE$kr>RFJsBtz5c=$6}_J
z;1(kQ2jw{%CapYwyMG#AAD*3 at lG46OZ-HWdif<M969-y`mIE=QT)mVDDTLi_-%7oC
zb6<LfI7M3M(80x5TAbwFZ&NXCPjormsJB&v&7tI}_h{%t{vsi`U7?>dLCyDOCra<m
zCMtd>mC_KzA4yqjFRWT7-pCpdHmMAvR9v0$&Kfe{K&&L6p`Ec}_r!mg<nX_)72->%
ztFu|3`d}Kssa#vF4x*{O3q(?>g^A82_V0oszq$*kDK_xnS)sQf<6nG402O%`vyh72
z?BNgPBnTL{`mr^lOB^VTx(=}U$xPgk+WYU#sqJhRY!E#Fb?7>noBKQg)T@#!3bWa=
zb4I;jEYL*93$D^4>;&0pDm#o0Gx#=L%)w6&Ku&>eI at 3!piLu>SCn;w8gCZ=*Cxr}U
z#{EAco>~BQ&W%P7mA4S@@TpJ+qaOLvb<tlRvvBOfZUJ$b%r^$h&TFb~q}jRy#pD;$
zx#SW3A+k`_+2TP=l*#JozBoVKClN(4)?hnD>n1cEV?P9M3c_=41n7LY{~FaM(Lx3$
zxx)>vkrwlT*z(#~%_p_isM>*H`HkErGk^%%<8Yv(0m8XeUMdQk@?}zFiRW|ZR_0u-
zYU`7z$^UjkA!xkw!MSWk(TS at 0CKxUff1$I&`^VP at T$(pNx?EE!eWS-PUq2PE+EMEX
z#Dfx{(-Lhj8{0p=f5X&u)F<|S@<-GP)Ot``57yyzF+8Xqi3VIuEiy(&T9we`EN3UN
zl-_xO<nV~r5&>jL+GMCq#JKh{odiUKMRXXIB?LG*asMRw6e&oi4$c%5-VnoY&FJ~L
z2fPpiJUUwEK;gR0b-r|foV?xoyF|#2ac)Us*0QPWf(7(~<~UtD?ZFpc8gsFUKrhvV
zi<y&VrpQmoBNHT|2nL}V^a6q^-C>3FvU at x@6u8=(@lQ+H{iB!-*~(bB^2M8c?|1K$
zCf^Ei>I3q{4hBF}qPxVQY?v7Vgg>@KueF3;!v)-6f}`48C)RUDQPb{pr8|oodQ#(s
zay`V{?f#A{0c&Z778fm=WS=ZyMv+flCu&iUv at Xjz@)+EEHH-K}`6nXydYB8r*XTT^
z#F_dt0_80M5dGd<k9^lpk*hCl6A+KcdkdiKS4Rl0q6^;sz~H;T`4GEa7XSVq!m?*H
z%jrTgz-`%FP>3n>S%_45!;do3LZ~^Lk0#ni1{p)y&mZExR0cEHgJ=dF1sD4n3gDJW
zoBRO~XWzI55vysobBWZ~*HmRzu8<#LVW3XktuUoFv!ldQ+Yn}BX^LDIV^<uf(h_&M
b+L$9Nwtap_Zj%Rq^O#A4k$KeQX+!u2D38jy

literal 0
HcmV?d00001

-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 10:28:54 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 17:28:54 +0200
Subject: [bmap-tools] [PATCH 3/4] TODO: remove the complete items
In-Reply-To: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
References: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383578935-21224-3-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I56e94874e871ece9454359be15bb91c7ec203a02
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 TODO | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/TODO b/TODO
index 2b8df99..74c4733 100644
--- a/TODO
+++ b/TODO
@@ -1,7 +1,4 @@
 Current TODO list, any help with these is appreciated.
 
 1. Teach bmaptool to update the alternate GPT partition
-2. Add nosetests for all bmap format versions
-3. Add a test for bmap with invalid checksums
-4. Add a test for checking that old BmapCopy works fine with newer compatible
-   bmap formats
+2. Add a test for bmap with invalid checksums
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 10:28:53 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 17:28:53 +0200
Subject: [bmap-tools] [PATCH 2/4] test_compat: improve backward
	compatibility test
In-Reply-To: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
References: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383578935-21224-2-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Currently 'test_compat' verifies that BmapCopy handles all the older bmap file
formats. This patch adds a check that older BmapCopy implementations are able
to handle all the compatible bmap file formats.

The reason for this test is that I once screwed it up with version 1.4: it had
incompatible changes, and older versions of BmapCopy crash with 1.4. If I had
this test-suite, I would add the new bmap format file to 'tests/test-data', and
the mistake would be caought right away.

Change-Id: I62ac407fb9cfe3b7158853ce09d5402cf85785de
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/test_compat.py | 147 ++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 109 insertions(+), 38 deletions(-)

diff --git a/tests/test_compat.py b/tests/test_compat.py
index d0ba703..ea8e606 100644
--- a/tests/test_compat.py
+++ b/tests/test_compat.py
@@ -12,15 +12,29 @@
 # General Public License for more details.
 
 """
-This unit test verifies that BmapCopy can handle all the bmap file formats and
-have no backward-compatibility problems.
+This unit test verifies various compatibility aspects of the BmapCopy module:
+    * current BmapCopy has to handle all the older bmap formats
+    * older BmapCopy have to handle all the newer compatible bmap formats
 """
 
+# Disable the following pylint recommendations:
+#   *  Too many public methods - R0904
+# pylint: disable=R0904
+
 import os
 import shutil
 import tempfile
 from tests import helpers
-from bmaptools import TransRead
+from bmaptools import TransRead, BmapCopy
+
+# Test image file name
+_IMAGE_NAME = "test.image.gz"
+# Test bmap file names template
+_BMAP_TEMPL = "test.image.bmap.v"
+# Name of the subdirectory where test data are stored
+_TEST_DATA_SUBDIR = "test-data"
+# Name of the subdirectory where old BmapCopy modules are stored
+_OLDCODEBASE_SUBDIR = "oldcodebase"
 
 # This is a work-around for Centos 6
 try:
@@ -29,50 +43,107 @@ except ImportError:
     import unittest
 
 class TestCreateCopy(unittest.TestCase):
-    """
-    The test class for this unit tests which executes the '_do_test()' function
-    for various bmap file formats.
-    """
+    """The test class for this unit test."""
 
     def test(self):
-        """
-        The test entry point. Executes the '_do_test()' function for various
-        bmap file formats.
-        """
-
-        image_name = "test.image.gz"
-        bmap_name  = "test.image.bmap.v"
-        test_data_dir = "test-data"
-
-        test_data_dir = os.path.join(os.path.dirname(__file__), test_data_dir)
-        image_path = os.path.join(test_data_dir, image_name)
+        """The test entry point."""
 
+        test_data_dir = os.path.join(os.path.dirname(__file__),
+                                     _TEST_DATA_SUBDIR)
+        image_path = os.path.join(test_data_dir, _IMAGE_NAME)
 
-        # Get the list of bmap files to test
-        bmap_paths = []
-        for direntry in os.listdir(test_data_dir):
-            direntry_path = os.path.join(test_data_dir, direntry)
-            if os.path.isfile(direntry_path) and direntry.startswith(bmap_name):
-                bmap_paths.append(direntry_path)
+        # Construct the list of bmap files to test
+        self._bmap_paths = []
+        for dentry in os.listdir(test_data_dir):
+            dentry_path = os.path.join(test_data_dir, dentry)
+            if os.path.isfile(dentry_path) and dentry.startswith(_BMAP_TEMPL):
+                self._bmap_paths.append(dentry_path)
 
         # Create and open a temporary file for uncompressed image and its copy
-        f_image = tempfile.NamedTemporaryFile("wb+", prefix=image_name,
-                                              suffix=".image")
-        f_copy = tempfile.NamedTemporaryFile("wb+", prefix=image_name,
-                                             suffix=".copy")
+        self._f_image = tempfile.NamedTemporaryFile("wb+", prefix=_IMAGE_NAME,
+                                                   suffix=".image")
+        self._f_copy = tempfile.NamedTemporaryFile("wb+", prefix=_IMAGE_NAME,
+                                                  suffix=".copy")
 
-        # Create an ucompressed version of the image file
+        # Uncompress the test image into 'self._f_image'
         f_tmp_img = TransRead.TransRead(image_path)
-        shutil.copyfileobj(f_tmp_img, f_image)
+        shutil.copyfileobj(f_tmp_img, self._f_image)
         f_tmp_img.close()
-        f_image.flush()
+        self._f_image.flush()
+
+        image_chksum = helpers.calculate_chksum(self._f_image.name)
+        image_size = os.path.getsize(self._f_image.name)
+
+        # Test the current version of BmapCopy
+        for bmap_path in self._bmap_paths:
+            helpers.copy_and_verify_image(image_path, self._f_copy.name,
+                                          bmap_path, image_chksum,
+                                          image_size)
+
+        # Test the older versions of BmapCopy
+        self._test_older_bmapcopy()
+
+        self._f_copy.close()
+        self._f_image.close()
+
+    def _test_older_bmapcopy(self):
+        """Test older than the current versions of the BmapCopy class."""
+
+        def import_module(searched_module):
+            """Search and import a module by its name."""
+
+            modref = __import__(searched_module)
+            for name in searched_module.split(".")[1:]:
+                modref = getattr(modref, name)
+            return modref
+
+        oldcodebase_dir = os.path.join(os.path.dirname(__file__),
+                                       _OLDCODEBASE_SUBDIR)
+
+        # Construct the list of old BmapCopy modules
+        old_modules = []
+        for dentry in os.listdir(oldcodebase_dir):
+            if dentry.startswith("BmapCopy") and dentry.endswith(".py"):
+                old_modules.append("tests." + _OLDCODEBASE_SUBDIR + "."
+                                   + dentry[:-3])
+
+        for old_module in old_modules:
+            modref = import_module(old_module)
+
+            for bmap_path in self._bmap_paths:
+                self._do_test_older_bmapcopy(bmap_path, modref)
+
+    def _do_test_older_bmapcopy(self, bmap_path, modref):
+        """
+        Test an older version of BmapCopy class, referenced to by the 'modref'
+        argument. The 'bmap_path' argument is the bmap file path to test with.
+        """
+
+        # Get a reference to the older BmapCopy class object to test with
+        old_bmapcopy_class = getattr(modref, "BmapCopy")
+        supported_ver = getattr(modref, "SUPPORTED_BMAP_VERSION")
+
+        f_bmap = open(bmap_path, "r")
 
-        image_chksum = helpers.calculate_chksum(f_image.name)
-        image_size = os.path.getsize(f_image.name)
+        # Find the version of the bmap file. The easiest is to simply use the
+        # latest BmapCopy.
+        bmapcopy = BmapCopy.BmapCopy(self._f_image, self._f_copy, f_bmap)
+        bmap_version = bmapcopy.bmap_version
+        bmap_version_major = bmapcopy.bmap_version_major
 
-        for bmap_path in bmap_paths:
-            helpers.copy_and_verify_image(image_path, f_copy.name, bmap_path,
-                                          image_chksum, image_size)
+        try:
+            if supported_ver >= bmap_version:
+                writer = old_bmapcopy_class(self._f_image, self._f_copy, f_bmap)
+                writer.copy(True, True)
+        except:
+            if supported_ver >= bmap_version_major:
+                # The BmapCopy which we are testing is supposed to support this
+                # version of bmap file format. However, bmap format version 1.4
+                # was a screw-up, because it actually had incompatible changes,
+                # so old versions of BmapCopy are supposed to fail.
+                if not (supported_ver == 1 and bmap_version == "1.4"):
+                    print("Module \"%s\" failed to handle \"%s\"" %
+                          (modref.__name__, bmap_path))
+                    raise
 
-        f_copy.close()
-        f_image.close()
+        f_bmap.close()
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 10:28:55 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 17:28:55 +0200
Subject: [bmap-tools] [PATCH 4/4] make_a_release: remind about updating the
	compat test
In-Reply-To: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
References: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>
Message-ID: <1383578935-21224-4-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

Change-Id: I06d0bdea86c391ffc5cba8ff862a791b8d3f678d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 make_a_release.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/make_a_release.sh b/make_a_release.sh
index bb070ee..1694ce6 100755
--- a/make_a_release.sh
+++ b/make_a_release.sh
@@ -94,6 +94,7 @@ ask_question "Did you update the docs/RELEASE_NOTES file"
 ask_question "Did you update the docs/README file"
 ask_question "Did you update the man page"
 ask_question "Did you update documentation on tizen.org"
+ask_question "Did you update tests: test-data and oldcodebase"
 
 # Make sure the git index is up-to-date
 [ -z "$(git status --porcelain)" ] || fatal "git index is not up-to-date"
-- 
1.8.1.4



From dedekind1 at gmail.com  Mon Nov  4 10:28:52 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Mon,  4 Nov 2013 17:28:52 +0200
Subject: [bmap-tools] [PATCH 1/4] tests: add old code-base
Message-ID: <1383578935-21224-1-git-send-email-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy at intel.com>

I am going to add a test which verifies that older BmapCopy work fine with
newer compatible bmap formats, as well as newer BmapCopy works fine with all
the older bmap formats.

This patch simply add a copy of various BmapCopy versions.

Change-Id: I37941f343b56511a85a47cda239dd702b7d2afc0
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at intel.com>
---
 tests/oldcodebase/BmapCopy1_0.py | 707 ++++++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_0.py | 631 ++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_1.py | 631 ++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_2.py | 632 ++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_3.py | 667 ++++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_4.py | 667 ++++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_5.py | 724 +++++++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy2_6.py | 724 +++++++++++++++++++++++++++++++++++++
 tests/oldcodebase/BmapCopy3_0.py | 764 +++++++++++++++++++++++++++++++++++++++
 tests/oldcodebase/__init__.py    |   0
 10 files changed, 6147 insertions(+)
 create mode 100644 tests/oldcodebase/BmapCopy1_0.py
 create mode 100644 tests/oldcodebase/BmapCopy2_0.py
 create mode 100644 tests/oldcodebase/BmapCopy2_1.py
 create mode 100644 tests/oldcodebase/BmapCopy2_2.py
 create mode 100644 tests/oldcodebase/BmapCopy2_3.py
 create mode 100644 tests/oldcodebase/BmapCopy2_4.py
 create mode 100644 tests/oldcodebase/BmapCopy2_5.py
 create mode 100644 tests/oldcodebase/BmapCopy2_6.py
 create mode 100644 tests/oldcodebase/BmapCopy3_0.py
 create mode 100644 tests/oldcodebase/__init__.py

diff --git a/tests/oldcodebase/BmapCopy1_0.py b/tests/oldcodebase/BmapCopy1_0.py
new file mode 100644
index 0000000..9722a1f
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy1_0.py
@@ -0,0 +1,707 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+#   * Too many statements (R0915)
+#   * Too many branches (R0912)
+# pylint: disable=R0902
+# pylint: disable=R0915
+# pylint: disable=R0912
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# A list of supported image formats
+SUPPORTED_IMAGE_FORMATS = ('bz2', 'gz', 'tar.gz', 'tgz', 'tar.bz2')
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file-like object of the destination file copy the image to
+    * full path or a file-like object of the bmap file (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    The image file may either be an uncompressed raw image or a compressed
+    image. Compression type is defined by the image file extension.  Supported
+    types are listed by 'SUPPORTED_IMAGE_FORMATS'.
+
+    IMPORTANT: if the image is given as a file-like object, the compression
+    type recognition is not performed - the file-like object's 'read()' method
+    is used directly instead.
+
+    Once an instance of 'BmapCopy' is created, all the 'bmap_*' attributes are
+    initialized and available. They are read from the bmap.
+
+    However, if bmap was not provided, this is not always the case and some of
+    the 'bmap_*' attributes are not initialize by the class constructor.
+    Instead, they are initialized only in the 'copy()' method. The reason for
+    this is that when bmap is absent, 'BmapCopy' uses sensible fall-back values
+    for the 'bmap_*' attributes assuming the entire image is "mapped". And if
+    the image is compressed, it cannot easily find out the image size. Thus,
+    this is postponed until the 'copy()' method decompresses the image for the
+    first time.
+
+    The 'copy()' method implements the copying. You may choose whether to
+    verify the SHA1 checksum while copying or not.  Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'. """
+
+    def _initialize_sizes(self, image_size):
+        """ This function is only used when the there is no bmap. It
+        initializes attributes like 'blocks_cnt', 'mapped_cnt', etc. Normally,
+        the values are read from the bmap file, but in this case they are just
+        set to something reasonable. """
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+        self.mapped_cnt = self.blocks_cnt
+        self.mapped_size = self.image_size
+        self.mapped_size_human = self.image_size_human
+
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize the 'bmap_*' attributes. """
+
+        bmap_pos = self._f_bmap.tell()
+        self._f_bmap.seek(0)
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        major = int(self.bmap_version.split('.', 1)[0])
+        if major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        self._f_bmap.seek(bmap_pos)
+
+    def _open_image_file(self):
+        """ Open the image file which may be compressed or not. The compression
+        type is recognized by the file extension. Supported types are defined
+        by 'SUPPORTED_IMAGE_FORMATS'. """
+
+        try:
+            is_regular_file = stat.S_ISREG(os.stat(self._image_path).st_mode)
+        except OSError as err:
+            raise Error("cannot access image file '%s': %s" \
+                        % (self._image_path, err.strerror))
+
+        if not is_regular_file:
+            raise Error("image file '%s' is not a regular file" \
+                        % self._image_path)
+
+        try:
+            if self._image_path.endswith('.tar.gz') \
+               or self._image_path.endswith('.tar.bz2') \
+               or self._image_path.endswith('.tgz'):
+                import tarfile
+
+                tar = tarfile.open(self._image_path, 'r')
+                # The tarball is supposed to contain only one single member
+                members = tar.getnames()
+                if len(members) > 1:
+                    raise Error("the image tarball '%s' contains more than " \
+                                "one file" % self._image_path)
+                elif len(members) == 0:
+                    raise Error("the image tarball '%s' is empty (no files)" \
+                                % self._image_path)
+                self._f_image = tar.extractfile(members[0])
+            elif self._image_path.endswith('.gz'):
+                import gzip
+                self._f_image = gzip.GzipFile(self._image_path, 'rb')
+            elif self._image_path.endswith('.bz2'):
+                import bz2
+                self._f_image = bz2.BZ2File(self._image_path, 'rb')
+            else:
+                self._image_is_compressed = False
+                self._f_image = open(self._image_path, 'rb')
+        except IOError as err:
+            raise Error("cannot open image file '%s': %s" \
+                        % (self._image_path, err))
+
+        self._f_image_needs_close = True
+
+    def _validate_image_size(self):
+        """ Make sure that image size from bmap matches real image size. """
+
+        image_size = os.fstat(self._f_image.fileno()).st_size
+        if image_size != self.image_size:
+            raise Error("Size mismatch, bmap '%s' was created for an image " \
+                        "of size %d bytes, but image '%s' has size %d bytes" \
+                        % (self._bmap_path, self.image_size,
+                           self._image_path, image_size))
+
+    def _open_destination_file(self):
+        """ Open the destination file. """
+
+        try:
+            self._f_dest = open(self._dest_path, 'w')
+        except IOError as err:
+            raise Error("cannot open destination file '%s': %s" \
+                        % (self._dest_path, err))
+
+        self._f_dest_needs_close = True
+
+    def _open_bmap_file(self):
+        """ Open the bmap file. """
+
+        try:
+            self._f_bmap = open(self._bmap_path, 'r')
+        except IOError as err:
+            raise Error("cannot open bmap file '%s': %s" \
+                        % (self._bmap_path, err.strerror))
+
+        self._f_bmap_needs_close = True
+
+    def __init__(self, image, dest, bmap = None):
+        """ The class constructor. The parameters are:
+            image - full path or file object of the image which should be copied
+            dest  - full path or file-like object of the destination file to
+                    copy the image to
+            bmap  - full path or file-like object of the bmap file to use for
+                    copying """
+
+        self._xml = None
+        self._image_is_compressed = True
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_dest_needs_close = False
+        self._f_image_needs_close = False
+        self._f_bmap_needs_close = False
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        if hasattr(dest, "write"):
+            self._f_dest = dest
+            self._dest_path = dest.name
+        else:
+            self._dest_path = dest
+            self._open_destination_file()
+
+        if hasattr(image, "read"):
+            self._f_image = image
+            self._image_path = image.name
+        else:
+            self._image_path = image
+            self._open_image_file()
+
+        st_mode = os.fstat(self._f_dest.fileno()).st_mode
+        self._dest_is_regfile = stat.S_ISREG(st_mode)
+
+        if bmap:
+            if hasattr(bmap, "read"):
+                self._f_bmap = bmap
+                self._bmap_path = bmap.name
+            else:
+                self._bmap_path = bmap
+                self._open_bmap_file()
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+            # We can initialize size-related attributes only if we the image is
+            # uncompressed.
+            if not self._image_is_compressed:
+                image_size = os.fstat(self._f_image.fileno()).st_size
+                self._initialize_sizes(image_size)
+
+        if not self._image_is_compressed:
+            self._validate_image_size()
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def __del__(self):
+        """ The class destructor which closes the opened files. """
+
+        if self._f_image_needs_close:
+            self._f_image.close()
+        if self._f_dest_needs_close:
+            self._f_dest.close()
+        if self._f_bmap_needs_close:
+            self._f_bmap.close()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown (the image is
+        compressed), the generator infinitely yields continuous ranges of
+        size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s" \
+                                % (first, last, hash_obj.hexdigest(), sha1))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The sync
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Save file positions in order to restore them at the end
+        image_pos = self._f_image.tell()
+        dest_pos = self._f_dest.tell()
+        if self._f_bmap:
+            bmap_pos = self._f_bmap.tell()
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+        if not self.image_size:
+            # The image size was unknown up until now, probably because this is
+            # a compressed image. Initialize the corresponding class attributes
+            # now, when we know the size.
+            self._initialize_sizes(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks, but should have %u - inconsistent " \
+                       "bmap file" % (blocks_written, self.mapped_cnt))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+        # Restore file positions
+        self._f_image.seek(image_pos)
+        self._f_dest.seek(dest_pos)
+        if self._f_bmap:
+            self._f_bmap.seek(bmap_pos)
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        try:
+            os.fsync(self._f_dest.fileno()),
+        except OSError as err:
+            raise Error("cannot synchronize '%s': %s " \
+                        % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _open_destination_file(self):
+        """ Open the block device in exclusive mode. """
+
+        try:
+            self._f_dest = os.open(self._dest_path, os.O_WRONLY | os.O_EXCL)
+        except OSError as err:
+            raise Error("cannot open block device '%s' in exclusive mode: %s" \
+                        % (self._dest_path, err.strerror))
+
+        try:
+            os.fstat(self._f_dest).st_mode
+        except OSError as err:
+            raise Error("cannot access block device '%s': %s" \
+                        % (self._dest_path, err.strerror))
+
+        # Turn the block device file descriptor into a file object
+        try:
+            self._f_dest = os.fdopen(self._f_dest, "wb")
+        except OSError as err:
+            os.close(self._f_dest)
+            raise Error("cannot open block device '%s': %s" \
+                        % (self._dest_path, err))
+
+        self._f_dest_needs_close = True
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError:
+            # No problem, this is just an optimization.
+            return
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError:
+            return
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError:
+                # No problem, this is just an optimization.
+                return
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError:
+                return
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        try:
+            self._tune_block_device()
+            BmapCopy.copy(self, sync, verify)
+        except:
+            self._restore_bdev_settings()
+            raise
+
+    def __init__(self, image, dest, bmap = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known (i.e., it is not compressed) - check that
+        # it fits the block device.
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        try:
+            if not os.path.exists(self._sysfs_base + "queue"):
+                self._sysfs_base = self._sysfs_base + "../"
+        except OSError:
+            # No problem, this is just an optimization.
+            pass
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_0.py b/tests/oldcodebase/BmapCopy2_0.py
new file mode 100644
index 0000000..7082287
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_0.py
@@ -0,0 +1,631 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file-like object of the destination file copy the image to
+    * full path or a file-like object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance. """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """ Setup the progress indicator which shows how much data has been
+        copied in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent. """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """ Set image size and initialize various other geometry-related
+        attributes. """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to " \
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize corresponding class instance
+        attributs. """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        major = int(self.bmap_version.split('.', 1)[0])
+        if major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file-like object of the destination file to copy the
+                         image to.
+            bmap       - file-like object of the bmap file to use for copying.
+            image_size - size of the image in bytes. """
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """ Print the progress indicator if the mapped area size is known and
+        if the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute. """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds = 250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A')
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s (image file %s)" \
+                                % (first, last, hash_obj.hexdigest(), \
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The 'sync'
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should " \
+                        "have %u - inconsistent bmap file '%s'" \
+                        % (blocks_written, self._image_path, self._dest_path, \
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            # No problem, this is just an optimization
+            raise Error("cannot enable the 'noop' I/O scheduler: %s" % err)
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            raise Error("cannot set max. I/O ratio to '1': %s" % err)
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s" \
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s" \
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_1.py b/tests/oldcodebase/BmapCopy2_1.py
new file mode 100644
index 0000000..7082287
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_1.py
@@ -0,0 +1,631 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file-like object of the destination file copy the image to
+    * full path or a file-like object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance. """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """ Setup the progress indicator which shows how much data has been
+        copied in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent. """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """ Set image size and initialize various other geometry-related
+        attributes. """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to " \
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize corresponding class instance
+        attributs. """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        major = int(self.bmap_version.split('.', 1)[0])
+        if major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file-like object of the destination file to copy the
+                         image to.
+            bmap       - file-like object of the bmap file to use for copying.
+            image_size - size of the image in bytes. """
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """ Print the progress indicator if the mapped area size is known and
+        if the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute. """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds = 250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A')
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s (image file %s)" \
+                                % (first, last, hash_obj.hexdigest(), \
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The 'sync'
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should " \
+                        "have %u - inconsistent bmap file '%s'" \
+                        % (blocks_written, self._image_path, self._dest_path, \
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            # No problem, this is just an optimization
+            raise Error("cannot enable the 'noop' I/O scheduler: %s" % err)
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            raise Error("cannot set max. I/O ratio to '1': %s" % err)
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s" \
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s" \
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_2.py b/tests/oldcodebase/BmapCopy2_2.py
new file mode 100644
index 0000000..76c28f7
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_2.py
@@ -0,0 +1,632 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file-like object of the destination file copy the image to
+    * full path or a file-like object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance. """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """ Setup the progress indicator which shows how much data has been
+        copied in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent. """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """ Set image size and initialize various other geometry-related
+        attributes. """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to " \
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize corresponding class instance
+        attributs. """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        major = int(self.bmap_version.split('.', 1)[0])
+        if major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file-like object of the destination file to copy the
+                         image to.
+            bmap       - file-like object of the bmap file to use for copying.
+            image_size - size of the image in bytes. """
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """ Print the progress indicator if the mapped area size is known and
+        if the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute. """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds = 250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s (image file %s)" \
+                                % (first, last, hash_obj.hexdigest(), \
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The 'sync'
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should " \
+                        "have %u - inconsistent bmap file '%s'" \
+                        % (blocks_written, self._image_path, self._dest_path, \
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            # No problem, this is just an optimization
+            raise Error("cannot enable the 'noop' I/O scheduler: %s" % err)
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        if match:
+            self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            raise Error("cannot set max. I/O ratio to '1': %s" % err)
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s" \
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s" \
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_3.py b/tests/oldcodebase/BmapCopy2_3.py
new file mode 100644
index 0000000..889f357
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_3.py
@@ -0,0 +1,667 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file object of the destination file copy the image to
+    * full path or a file object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance. """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """ Setup the progress indicator which shows how much data has been
+        copied in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent. """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """ Set image size and initialize various other geometry-related
+        attributes. """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to " \
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _verify_bmap_checksum(self):
+        """ This is a helper function which verifies SHA1 checksum of the bmap
+        file. """
+
+        import mmap
+
+        correct_sha1 = self._xml.find("BmapFileSHA1").text.strip()
+
+        # Before verifying the shecksum, we have to substitute the SHA1 value
+        # stored in the file with all zeroes. For these purposes we create
+        # private memory mapping of the bmap file.
+        mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
+                                access = mmap.ACCESS_COPY)
+
+        sha1_pos = mapped_bmap.find(correct_sha1)
+        assert sha1_pos != -1
+
+        mapped_bmap[sha1_pos:sha1_pos + 40] = '0' * 40
+        calculated_sha1 = hashlib.sha1(mapped_bmap).hexdigest()
+
+        mapped_bmap.close()
+
+        if calculated_sha1 != correct_sha1:
+            raise Error("checksum mismatch for bmap file '%s': calculated " \
+                        "'%s', should be '%s'" % \
+                        (self._bmap_path, calculated_sha1, correct_sha1))
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize corresponding class instance
+        attributs. """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        self.bmap_version_major = int(self.bmap_version.split('.', 1)[0])
+        self.bmap_version_minor = int(self.bmap_version.split('.', 1)[1])
+        if self.bmap_version_major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, self.bmap_version_major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
+            # Bmap file checksum appeard in format 1.3
+            self._verify_bmap_checksum()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes. """
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """ Print the progress indicator if the mapped area size is known and
+        if the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute. """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds = 250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s (image file %s)" \
+                                % (first, last, hash_obj.hexdigest(), \
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The 'sync'
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should " \
+                        "have %u - bmap file '%s' does not belong to this" \
+                        "image" \
+                        % (blocks_written, self._image_path, self._dest_path, \
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            # No problem, this is just an optimization
+            raise Error("cannot enable the 'noop' I/O scheduler: %s" % err)
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        if match:
+            self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            raise Error("cannot set max. I/O ratio to '1': %s" % err)
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s" \
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s" \
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_4.py b/tests/oldcodebase/BmapCopy2_4.py
new file mode 100644
index 0000000..889f357
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_4.py
@@ -0,0 +1,667 @@
+""" This module implements copying of images with bmap and provides the
+following API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied. """
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """ A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors. """
+    pass
+
+class BmapCopy:
+    """ This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file object of the destination file copy the image to
+    * full path or a file object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance. """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """ Setup the progress indicator which shows how much data has been
+        copied in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent. """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """ Set image size and initialize various other geometry-related
+        attributes. """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to " \
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _verify_bmap_checksum(self):
+        """ This is a helper function which verifies SHA1 checksum of the bmap
+        file. """
+
+        import mmap
+
+        correct_sha1 = self._xml.find("BmapFileSHA1").text.strip()
+
+        # Before verifying the shecksum, we have to substitute the SHA1 value
+        # stored in the file with all zeroes. For these purposes we create
+        # private memory mapping of the bmap file.
+        mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
+                                access = mmap.ACCESS_COPY)
+
+        sha1_pos = mapped_bmap.find(correct_sha1)
+        assert sha1_pos != -1
+
+        mapped_bmap[sha1_pos:sha1_pos + 40] = '0' * 40
+        calculated_sha1 = hashlib.sha1(mapped_bmap).hexdigest()
+
+        mapped_bmap.close()
+
+        if calculated_sha1 != correct_sha1:
+            raise Error("checksum mismatch for bmap file '%s': calculated " \
+                        "'%s', should be '%s'" % \
+                        (self._bmap_path, calculated_sha1, correct_sha1))
+
+    def _parse_bmap(self):
+        """ Parse the bmap file and initialize corresponding class instance
+        attributs. """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a " \
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        self.bmap_version_major = int(self.bmap_version.split('.', 1)[0])
+        self.bmap_version_minor = int(self.bmap_version.split('.', 1)[1])
+        if self.bmap_version_major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, " \
+                        "version %d is not supported" \
+                        % (SUPPORTED_BMAP_VERSION, self.bmap_version_major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match " \
+                        "blocks count (%d bytes != %d blocks * %d bytes)" \
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
+            # Bmap file checksum appeard in format 1.3
+            self._verify_bmap_checksum()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes. """
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """ Print the progress indicator if the mapped area size is known and
+        if the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute. """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds = 250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """ This is a helper generator that parses the bmap XML file and for
+        each block range in the XML file it yields ('first', 'last', 'sha1')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'. """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """ This is a helper generator which splits block ranges from the bmap
+        file to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1). """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """ This is generator  which reads the image file in '_batch_blocks'
+        chunks and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data. """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the " \
+                                    "image file '%s': %s" \
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: " \
+                                "calculated %s, should be %s (image file %s)" \
+                                % (first, last, hash_obj.hexdigest(), \
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync = True, verify = True):
+        """ Copy the image to the destination file using bmap. The 'sync'
+        argument defines whether the destination file has to be synchronized
+        upon return.  The 'verify' argument defines whether the SHA1 checksum
+        has to be verified while copying. """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s" \
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should " \
+                        "have %u - bmap file '%s' does not belong to this" \
+                        "image" \
+                        % (blocks_written, self._image_path, self._dest_path, \
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s" \
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """ Synchronize the destination file to make sure all the data are
+        actually written to the disk. """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """ This class is a specialized version of 'BmapCopy' which copies the
+    image to a block device. Unlike the base 'BmapCopy' class, this class does
+    various optimizations specific to block devices, e.g., switching to the
+    'noop' I/O scheduler. """
+
+    def _tune_block_device(self):
+        """" Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            # No problem, this is just an optimization
+            raise Error("cannot enable the 'noop' I/O scheduler: %s" % err)
+
+        # The file contains a list of scheduler with the current
+        # scheduler in square brackets, e.g., "noop deadline [cfq]".
+        # Fetch the current scheduler name
+        import re
+
+        match = re.match(r'.*\[(.+)\].*', contents)
+        if match:
+            self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            raise Error("cannot set max. I/O ratio to '1': %s" % err)
+
+    def _restore_bdev_settings(self):
+        """ Restore old block device settings which we changed in
+        '_tune_block_device()'. """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s" \
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s" \
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync = True, verify = True):
+        """ The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time. """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap = None, image_size = None):
+        """ The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices. """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s " \
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not " \
+                            "fit the block device '%s' which has %s capacity" \
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_5.py b/tests/oldcodebase/BmapCopy2_5.py
new file mode 100644
index 0000000..a025cb5
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_5.py
@@ -0,0 +1,724 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
+"""
+This module implements copying of images with bmap and provides the following
+API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied.
+"""
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import logging
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """
+    A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors.
+    """
+    pass
+
+class BmapCopy:
+    """
+    This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file object of the destination file copy the image to
+    * full path or a file object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance.
+    """
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """
+        Setup the progress indicator which shows how much data has been copied
+        in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent.
+        """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """
+        Set image size and initialize various other geometry-related attributes.
+        """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to "
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _verify_bmap_checksum(self):
+        """
+        This is a helper function which verifies SHA1 checksum of the bmap file.
+        """
+
+        import mmap
+
+        correct_sha1 = self._xml.find("BmapFileSHA1").text.strip()
+
+        # Before verifying the shecksum, we have to substitute the SHA1 value
+        # stored in the file with all zeroes. For these purposes we create
+        # private memory mapping of the bmap file.
+        mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
+                                access = mmap.ACCESS_COPY)
+
+        sha1_pos = mapped_bmap.find(correct_sha1)
+        assert sha1_pos != -1
+
+        mapped_bmap[sha1_pos:sha1_pos + 40] = '0' * 40
+        calculated_sha1 = hashlib.sha1(mapped_bmap).hexdigest()
+
+        mapped_bmap.close()
+
+        if calculated_sha1 != correct_sha1:
+            raise Error("checksum mismatch for bmap file '%s': calculated "
+                        "'%s', should be '%s'"
+                        % (self._bmap_path, calculated_sha1, correct_sha1))
+
+    def _parse_bmap(self):
+        """
+        Parse the bmap file and initialize corresponding class instance attributs.
+        """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a "
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        self.bmap_version_major = int(self.bmap_version.split('.', 1)[0])
+        self.bmap_version_minor = int(self.bmap_version.split('.', 1)[1])
+        if self.bmap_version_major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, "
+                        "version %d is not supported"
+                        % (SUPPORTED_BMAP_VERSION, self.bmap_version_major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match "
+                        "blocks count (%d bytes != %d blocks * %d bytes)"
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
+            # Bmap file checksum appeard in format 1.3
+            self._verify_bmap_checksum()
+
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes.
+            logger     - the logger object to use for printing messages.
+        """
+
+        self._logger = logger
+        if self._logger is None:
+            self._logger = logging.getLogger(__name__)
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def _update_progress(self, blocks_written):
+        """
+        Print the progress indicator if the mapped area size is known and if
+        the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute.
+        """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds=250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """
+        This is a helper generator that parses the bmap XML file and for each
+        block range in the XML file it yields ('first', 'last', 'sha1') tuples,
+        where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'.
+        """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """
+        This is a helper generator which splits block ranges from the bmap file
+        to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1).
+        """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """
+        This is generator  which reads the image file in '_batch_blocks' chunks
+        and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data.
+        """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the "
+                                    "image file '%s': %s"
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: "
+                                "calculated %s, should be %s (image file %s)"
+                                % (first, last, hash_obj.hexdigest(),
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync=True, verify=True):
+        """
+        Copy the image to the destination file using bmap. The 'sync' argument
+        defines whether the destination file has to be synchronized upon
+        return.  The 'verify' argument defines whether the SHA1 checksum has to
+        be verified while copying.
+        """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s"
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should "
+                        "have %u - bmap file '%s' does not belong to this"
+                        "image"
+                        % (blocks_written, self._image_path, self._dest_path,
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s"
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """
+        Synchronize the destination file to make sure all the data are actually
+        written to the disk.
+        """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """
+    This class is a specialized version of 'BmapCopy' which copies the image to
+    a block device. Unlike the base 'BmapCopy' class, this class does various
+    optimizations specific to block devices, e.g., switching to the 'noop' I/O
+    scheduler.
+    """
+
+    def _tune_block_device(self):
+        """
+        Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            self._logger.warning("failed to enable I/O optimization, expect "
+                                 "suboptimal speed (reason: cannot switch "
+                                 "to the 'noop' I/O scheduler: %s)" % err)
+        else:
+            # The file contains a list of schedulers with the current
+            # scheduler in square brackets, e.g., "noop deadline [cfq]".
+            # Fetch the name of the current scheduler.
+            import re
+
+            match = re.match(r'.*\[(.+)\].*', contents)
+            if match:
+                self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering, because we do not need too much of it when
+        # writing sequntially. Excessive buffering makes some systems not very
+        # responsive, e.g., this was observed in Fedora 17.
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            self._logger.warning("failed to disable excessive buffering, "
+                                 "expect worse system responsiveness "
+                                 "(reason: cannot set max. I/O ratio to "
+                                 "1: %s)" % err)
+
+    def _restore_bdev_settings(self):
+        """
+        Restore old block device settings which we changed in
+        '_tune_block_device()'.
+        """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s"
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s"
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync=True, verify=True):
+        """
+        The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time.
+        """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
+
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices.
+        """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size, logger=logger)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not "
+                            "fit the block device '%s' which has %s capacity"
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
diff --git a/tests/oldcodebase/BmapCopy2_6.py b/tests/oldcodebase/BmapCopy2_6.py
new file mode 100644
index 0000000..4a9c8cc
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy2_6.py
@@ -0,0 +1,724 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
+"""
+This module implements copying of images with bmap and provides the following
+API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied.
+"""
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import logging
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """
+    A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors.
+    """
+    pass
+
+class BmapCopy:
+    """
+    This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file object of the destination file copy the image to
+    * full path or a file object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the SHA1 checksum while copying or not. Note, this is done only in
+    case of bmap-based copying and only if bmap contains the SHA1 checksums
+    (e.g., bmap version 1.0 did not have SHA1 checksums).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance.
+    """
+
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes.
+            logger     - the logger object to use for printing messages.
+        """
+
+        self._logger = logger
+        if self._logger is None:
+            self._logger = logging.getLogger(__name__)
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 2
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """
+        Setup the progress indicator which shows how much data has been copied
+        in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent.
+        """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """
+        Set image size and initialize various other geometry-related attributes.
+        """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to "
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _verify_bmap_checksum(self):
+        """
+        This is a helper function which verifies SHA1 checksum of the bmap file.
+        """
+
+        import mmap
+
+        correct_sha1 = self._xml.find("BmapFileSHA1").text.strip()
+
+        # Before verifying the shecksum, we have to substitute the SHA1 value
+        # stored in the file with all zeroes. For these purposes we create
+        # private memory mapping of the bmap file.
+        mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
+                                access = mmap.ACCESS_COPY)
+
+        sha1_pos = mapped_bmap.find(correct_sha1)
+        assert sha1_pos != -1
+
+        mapped_bmap[sha1_pos:sha1_pos + 40] = '0' * 40
+        calculated_sha1 = hashlib.sha1(mapped_bmap).hexdigest()
+
+        mapped_bmap.close()
+
+        if calculated_sha1 != correct_sha1:
+            raise Error("checksum mismatch for bmap file '%s': calculated "
+                        "'%s', should be '%s'"
+                        % (self._bmap_path, calculated_sha1, correct_sha1))
+
+    def _parse_bmap(self):
+        """
+        Parse the bmap file and initialize corresponding class instance attributs.
+        """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            raise Error("cannot parse the bmap file '%s' which should be a "
+                        "proper XML file: %s" % (self._bmap_path, err))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        self.bmap_version_major = int(self.bmap_version.split('.', 1)[0])
+        self.bmap_version_minor = int(self.bmap_version.split('.', 1)[1])
+        if self.bmap_version_major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, "
+                        "version %d is not supported"
+                        % (SUPPORTED_BMAP_VERSION, self.bmap_version_major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match "
+                        "blocks count (%d bytes != %d blocks * %d bytes)"
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
+            # Bmap file checksum appeard in format 1.3
+            self._verify_bmap_checksum()
+
+    def _update_progress(self, blocks_written):
+        """
+        Print the progress indicator if the mapped area size is known and if
+        the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute.
+        """
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds=250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """
+        This is a helper generator that parses the bmap XML file and for each
+        block range in the XML file it yields ('first', 'last', 'sha1') tuples,
+        where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'sha1' is the SHA1 checksum of the range ('None' is used if it is
+            missing.
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'.
+        """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if 'sha1' in xml_element.attrib:
+                sha1 = xml_element.attrib['sha1']
+            else:
+                sha1 = None
+
+            yield (first, last, sha1)
+
+    def _get_batches(self, first, last):
+        """
+        This is a helper generator which splits block ranges from the bmap file
+        to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1).
+        """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """
+        This is generator  which reads the image file in '_batch_blocks' chunks
+        and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data.
+        """
+
+        try:
+            for (first, last, sha1) in self._get_block_ranges():
+                if verify and sha1:
+                    hash_obj = hashlib.new('sha1')
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the "
+                                    "image file '%s': %s"
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and sha1:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and sha1 and hash_obj.hexdigest() != sha1:
+                    raise Error("checksum mismatch for blocks range %d-%d: "
+                                "calculated %s, should be %s (image file %s)"
+                                % (first, last, hash_obj.hexdigest(),
+                                   sha1, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync=True, verify=True):
+        """
+        Copy the image to the destination file using bmap. The 'sync' argument
+        defines whether the destination file has to be synchronized upon
+        return.  The 'verify' argument defines whether the SHA1 checksum has to
+        be verified while copying.
+        """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s"
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should "
+                        "have %u - bmap file '%s' does not belong to this "
+                        "image"
+                        % (blocks_written, self._image_path, self._dest_path,
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s"
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """
+        Synchronize the destination file to make sure all the data are actually
+        written to the disk.
+        """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """
+    This class is a specialized version of 'BmapCopy' which copies the image to
+    a block device. Unlike the base 'BmapCopy' class, this class does various
+    optimizations specific to block devices, e.g., switching to the 'noop' I/O
+    scheduler.
+    """
+
+    def __init__(self, image, dest, bmap=None, image_size=None, logger=None):
+        """
+        The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices.
+        """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size, logger=logger)
+
+        self._batch_bytes = 1024 * 1024
+        self._batch_blocks = self._batch_bytes / self.block_size
+        self._batch_queue_len = 6
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not "
+                            "fit the block device '%s' which has %s capacity"
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" \
+                           % (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
+
+    def _tune_block_device(self):
+        """
+        Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            self._logger.warning("failed to enable I/O optimization, expect "
+                                 "suboptimal speed (reason: cannot switch "
+                                 "to the 'noop' I/O scheduler: %s)" % err)
+        else:
+            # The file contains a list of schedulers with the current
+            # scheduler in square brackets, e.g., "noop deadline [cfq]".
+            # Fetch the name of the current scheduler.
+            import re
+
+            match = re.match(r'.*\[(.+)\].*', contents)
+            if match:
+                self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering, because we do not need too much of it when
+        # writing sequntially. Excessive buffering makes some systems not very
+        # responsive, e.g., this was observed in Fedora 17.
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            self._logger.warning("failed to disable excessive buffering, "
+                                 "expect worse system responsiveness "
+                                 "(reason: cannot set max. I/O ratio to "
+                                 "1: %s)" % err)
+
+    def _restore_bdev_settings(self):
+        """
+        Restore old block device settings which we changed in
+        '_tune_block_device()'.
+        """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s"
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s"
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync=True, verify=True):
+        """
+        The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time.
+        """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
diff --git a/tests/oldcodebase/BmapCopy3_0.py b/tests/oldcodebase/BmapCopy3_0.py
new file mode 100644
index 0000000..fe8897c
--- /dev/null
+++ b/tests/oldcodebase/BmapCopy3_0.py
@@ -0,0 +1,764 @@
+# Copyright (c) 2012-2013 Intel, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License, version 2,
+# as published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+
+"""
+This module implements copying of images with bmap and provides the following
+API.
+  1. BmapCopy class - implements copying to any kind of file, be that a block
+     device or a regular file.
+  2. BmapBdevCopy class - based on BmapCopy and specializes on copying to block
+     devices. It does some more sanity checks and some block device performance
+     tuning.
+
+The bmap file is an XML file which contains a list of mapped blocks of the
+image. Mapped blocks are the blocks which have disk sectors associated with
+them, as opposed to holes, which are blocks with no associated disk sectors. In
+other words, the image is considered to be a sparse file, and bmap basically
+contains a list of mapped blocks of this sparse file. The bmap additionally
+contains some useful information like block size (usually 4KiB), image size,
+mapped blocks count, etc.
+
+The bmap is used for copying the image to a block device or to a regular file.
+The idea is that we copy quickly with bmap because we copy only mapped blocks
+and ignore the holes, because they are useless. And if the image is generated
+properly (starting with a huge hole and writing all the data), it usually
+contains only little mapped blocks, comparing to the overall image size. And
+such an image compresses very well (because holes are read as all zeroes), so
+it is beneficial to distributor them as compressed files along with the bmap.
+
+Here is an example. Suppose you have a 4GiB image which contains only 100MiB of
+user data and you need to flash it to a slow USB stick. With bmap you end up
+copying only a little bit more than 100MiB of data from the image to the USB
+stick (namely, you copy only mapped blocks). This is a lot faster than copying
+all 4GiB of data. We say that it is a bit more than 100MiB because things like
+file-system meta-data (inode tables, superblocks, etc), partition table, etc
+also contribute to the mapped blocks and are also copied.
+"""
+
+# Disable the following pylint recommendations:
+#   * Too many instance attributes (R0902)
+# pylint: disable=R0902
+
+import os
+import stat
+import sys
+import hashlib
+import logging
+import Queue
+import thread
+import datetime
+from xml.etree import ElementTree
+from bmaptools.BmapHelpers import human_size
+
+# The highest supported bmap format version
+SUPPORTED_BMAP_VERSION = 1
+
+class Error(Exception):
+    """
+    A class for exceptions generated by the 'BmapCopy' module. We currently
+    support only one type of exceptions, and we basically throw human-readable
+    problem description in case of errors.
+    """
+    pass
+
+class BmapCopy:
+    """
+    This class implements the bmap-based copying functionality. To copy an
+    image with bmap you should create an instance of this class, which requires
+    the following:
+
+    * full path or a file-like object of the image to copy
+    * full path or a file object of the destination file copy the image to
+    * full path or a file object of the bmap file (optional)
+    * image size in bytes (optional)
+
+    Although the main purpose of this class is to use bmap, the bmap is not
+    required, and if it was not provided then the entire image will be copied
+    to the destination file.
+
+    When the bmap is provided, it is not necessary to specify image size,
+    because the size is contained in the bmap. Otherwise, it is benefitial to
+    specify the size because it enables extra sanity checks and makes it
+    possible to provide the progress bar.
+
+    When the image size is known either from the bmap or the caller specified
+    it to the class constructor, all the image geometry description attributes
+    ('blocks_cnt', etc) are initialized by the class constructor and available
+    for the user.
+
+    However, when the size is not known, some of  the image geometry
+    description attributes are not initialized by the class constructor.
+    Instead, they are initialized only by the 'copy()' method.
+
+    The 'copy()' method implements image copying. You may choose whether to
+    verify the checksum while copying or not. Note, this is done only in case
+    of bmap-based copying and only if bmap contains checksums (e.g., bmap
+    version 1.0 did not have checksums support).
+
+    You may choose whether to synchronize the destination file after writing or
+    not. To explicitly synchronize it, use the 'sync()' method.
+
+    This class supports all the bmap format versions up version
+    'SUPPORTED_BMAP_VERSION'.
+
+    It is possible to have a simple progress indicator while copying the image.
+    Use the 'set_progress_indicator()' method.
+
+    You can copy only once with an instance of this class. This means that in
+    order to copy the image for the second time, you have to create a new class
+    instance.
+    """
+
+    def __init__(self, image, dest, bmap=None, image_size=None, log=None):
+        """
+        The class constructor. The parameters are:
+            image      - file-like object of the image which should be copied,
+                         should only support 'read()' and 'seek()' methods,
+                         and only seeking forward has to be supported.
+            dest       - file object of the destination file to copy the image
+                         to.
+            bmap       - file object of the bmap file to use for copying.
+            image_size - size of the image in bytes.
+            log     - the logger object to use for printing messages.
+        """
+
+        self._log = log
+        if self._log is None:
+            self._log = logging.getLogger(__name__)
+
+        self._xml = None
+
+        self._dest_fsync_watermark = None
+        self._batch_blocks = None
+        self._batch_queue = None
+        self._batch_bytes = 1024 * 1024
+        self._batch_queue_len = 6
+
+        self.bmap_version = None
+        self.bmap_version_major = None
+        self.bmap_version_minor = None
+        self.block_size = None
+        self.blocks_cnt = None
+        self.mapped_cnt = None
+        self.image_size = None
+        self.image_size_human = None
+        self.mapped_size = None
+        self.mapped_size_human = None
+        self.mapped_percent = None
+
+        self._f_bmap = None
+        self._f_bmap_path = None
+
+        self._progress_started = None
+        self._progress_index = None
+        self._progress_time = None
+        self._progress_file = None
+        self._progress_format = None
+        self.set_progress_indicator(None, None)
+
+        self._f_image = image
+        self._image_path = image.name
+
+        self._f_dest = dest
+        self._dest_path = dest.name
+        st_data = os.fstat(self._f_dest.fileno())
+        self._dest_is_regfile = stat.S_ISREG(st_data.st_mode)
+
+        # The bmap file checksum type and length
+        self._cs_type = None
+        self._cs_len = None
+        self._cs_attrib_name = None
+
+        # Special quirk for /dev/null which does not support fsync()
+        if stat.S_ISCHR(st_data.st_mode) and \
+           os.major(st_data.st_rdev) == 1 and \
+           os.minor(st_data.st_rdev) == 3:
+            self._dest_supports_fsync = False
+        else:
+            self._dest_supports_fsync = True
+
+        if bmap:
+            self._f_bmap = bmap
+            self._bmap_path = bmap.name
+            self._parse_bmap()
+        else:
+            # There is no bmap. Initialize user-visible attributes to something
+            # sensible with an assumption that we just have all blocks mapped.
+            self.bmap_version = 0
+            self.block_size = 4096
+            self.mapped_percent = 100
+
+        if image_size:
+            self._set_image_size(image_size)
+
+        self._batch_blocks = self._batch_bytes / self.block_size
+
+    def set_progress_indicator(self, file_obj, format_string):
+        """
+        Setup the progress indicator which shows how much data has been copied
+        in percent.
+
+        The 'file_obj' argument is the console file object where the progress
+        has to be printed to. Pass 'None' to disable the progress indicator.
+
+        The 'format_string' argument is the format string for the progress
+        indicator. It has to contain a single '%d' placeholder which will be
+        substitutes with copied data in percent.
+        """
+
+        self._progress_file = file_obj
+        if format_string:
+            self._progress_format = format_string
+        else:
+            self._progress_format = "Copied %d%%"
+
+    def _set_image_size(self, image_size):
+        """
+        Set image size and initialize various other geometry-related attributes.
+        """
+
+        if self.image_size is not None and self.image_size != image_size:
+            raise Error("cannot set image size to %d bytes, it is known to "
+                        "be %d bytes (%s)" % (image_size, self.image_size,
+                                              self.image_size_human))
+
+        self.image_size = image_size
+        self.image_size_human = human_size(image_size)
+        self.blocks_cnt = self.image_size + self.block_size - 1
+        self.blocks_cnt /= self.block_size
+
+        if self.mapped_cnt is None:
+            self.mapped_cnt = self.blocks_cnt
+            self.mapped_size = self.image_size
+            self.mapped_size_human = self.image_size_human
+
+    def _verify_bmap_checksum(self):
+        """
+        This is a helper function which verifies the bmap file checksum.
+        """
+
+        import mmap
+
+        if self.bmap_version_minor == 3:
+            correct_chksum = self._xml.find("BmapFileSHA1").text.strip()
+        else:
+            correct_chksum = self._xml.find("BmapFileChecksum").text.strip()
+
+        # Before verifying the shecksum, we have to substitute the checksum
+        # value stored in the file with all zeroes. For these purposes we
+        # create private memory mapping of the bmap file.
+        mapped_bmap = mmap.mmap(self._f_bmap.fileno(), 0,
+                                access = mmap.ACCESS_COPY)
+
+        chksum_pos = mapped_bmap.find(correct_chksum)
+        assert chksum_pos != -1
+
+        mapped_bmap[chksum_pos:chksum_pos + self._cs_len] = '0' * self._cs_len
+
+        hash_obj = hashlib.new(self._cs_type)
+        hash_obj.update(mapped_bmap)
+        calculated_chksum = hash_obj.hexdigest()
+
+        mapped_bmap.close()
+
+        if calculated_chksum != correct_chksum:
+            raise Error("checksum mismatch for bmap file '%s': calculated "
+                        "'%s', should be '%s'"
+                        % (self._bmap_path, calculated_chksum, correct_chksum))
+
+    def _parse_bmap(self):
+        """
+        Parse the bmap file and initialize corresponding class instance attributs.
+        """
+
+        try:
+            self._xml = ElementTree.parse(self._f_bmap)
+        except  ElementTree.ParseError as err:
+            # Extrace the erroneous line with some context
+            self._f_bmap.seek(0)
+            xml_extract = ""
+            for num, line in enumerate(self._f_bmap):
+                if num >= err.position[0] - 4 and num <= err.position[0] + 4:
+                    xml_extract += "Line %d: %s" % (num, line)
+
+            raise Error("cannot parse the bmap file '%s' which should be a "
+                        "proper XML file: %s, the XML extract:\n%s" %
+                        (self._bmap_path, err, xml_extract))
+
+        xml = self._xml
+        self.bmap_version = str(xml.getroot().attrib.get('version'))
+
+        # Make sure we support this version
+        self.bmap_version_major = int(self.bmap_version.split('.', 1)[0])
+        self.bmap_version_minor = int(self.bmap_version.split('.', 1)[1])
+        if self.bmap_version_major > SUPPORTED_BMAP_VERSION:
+            raise Error("only bmap format version up to %d is supported, "
+                        "version %d is not supported"
+                        % (SUPPORTED_BMAP_VERSION, self.bmap_version_major))
+
+        # Fetch interesting data from the bmap XML file
+        self.block_size = int(xml.find("BlockSize").text.strip())
+        self.blocks_cnt = int(xml.find("BlocksCount").text.strip())
+        self.mapped_cnt = int(xml.find("MappedBlocksCount").text.strip())
+        self.image_size = int(xml.find("ImageSize").text.strip())
+        self.image_size_human = human_size(self.image_size)
+        self.mapped_size = self.mapped_cnt * self.block_size
+        self.mapped_size_human = human_size(self.mapped_size)
+        self.mapped_percent = (self.mapped_cnt * 100.0) / self.blocks_cnt
+
+        blocks_cnt = (self.image_size + self.block_size - 1) / self.block_size
+        if self.blocks_cnt != blocks_cnt:
+            raise Error("Inconsistent bmap - image size does not match "
+                        "blocks count (%d bytes != %d blocks * %d bytes)"
+                        % (self.image_size, self.blocks_cnt, self.block_size))
+
+        if self.bmap_version_major >= 1 and self.bmap_version_minor >= 3:
+            # Bmap file checksum appeard in format 1.3 and the only supported
+            # checksum type was SHA1. Version 1.4 started supporting arbitrary
+            # checksum types. A new "ChecksumType" tag was introduce to specify
+            # the checksum function name. And all XML tags which contained
+            # "sha1" in their name were renamed to something more neutral.
+            if self.bmap_version_minor == 3:
+                self._cs_type = "sha1"
+                self._cs_attrib_name = "sha1"
+            else:
+                self._cs_type = xml.find("ChecksumType").text.strip()
+                self._cs_attrib_name = "chksum"
+
+            try:
+                self._cs_len = len(hashlib.new(self._cs_type).hexdigest())
+            except ValueError as err:
+                raise Error("cannot initialize hash function \"%s\": %s" %
+                            (self._cs_type, err))
+            self._verify_bmap_checksum()
+
+    def _update_progress(self, blocks_written):
+        """
+        Print the progress indicator if the mapped area size is known and if
+        the indicator has been enabled by assigning a console file object to
+        the 'progress_file' attribute.
+        """
+
+        if self.mapped_cnt:
+            assert blocks_written <= self.mapped_cnt
+            percent = int((float(blocks_written) / self.mapped_cnt) * 100)
+            self._log.debug("wrote %d blocks out of %d (%d%%)" %
+                            (blocks_written, self.mapped_cnt, percent))
+        else:
+            self._log.debug("wrote %d blocks" % blocks_written)
+
+        if not self._progress_file:
+            return
+
+        if self.mapped_cnt:
+            progress = '\r' + self._progress_format % percent + '\n'
+        else:
+            # Do not rotate the wheel too fast
+            now = datetime.datetime.now()
+            min_delta = datetime.timedelta(milliseconds=250)
+            if now - self._progress_time < min_delta:
+                return
+            self._progress_time = now
+
+            progress_wheel = ('-', '\\', '|', '/')
+            progress = '\r' + progress_wheel[self._progress_index % 4] + '\n'
+            self._progress_index += 1
+
+        # This is a little trick we do in order to make sure that the next
+        # message will always start from a new line - we switch to the new
+        # line after each progress update and move the cursor up. As an
+        # example, this is useful when the copying is interrupted by an
+        # exception - the error message will start form new line.
+        if self._progress_started:
+            # The "move cursor up" escape sequence
+            self._progress_file.write('\033[1A') # pylint: disable=W1401
+        else:
+            self._progress_started = True
+
+        self._progress_file.write(progress)
+        self._progress_file.flush()
+
+    def _get_block_ranges(self):
+        """
+        This is a helper generator that parses the bmap XML file and for each
+        block range in the XML file it yields ('first', 'last', 'chksum')
+        tuples, where:
+          * 'first' is the first block of the range;
+          * 'last' is the last block of the range;
+          * 'chksum' is the checksum of the range ('None' is used if it is
+            missing).
+
+        If there is no bmap file, the generator just yields a single range
+        for entire image file. If the image size is unknown, the generator
+        infinitely yields continuous ranges of size '_batch_blocks'.
+        """
+
+        if not self._f_bmap:
+            # We do not have the bmap, yield a tuple with all blocks
+            if self.blocks_cnt:
+                yield (0, self.blocks_cnt - 1, None)
+            else:
+                # We do not know image size, keep yielding tuples with many
+                # blocks infinitely.
+                first = 0
+                while True:
+                    yield (first, first + self._batch_blocks - 1, None)
+                    first += self._batch_blocks
+            return
+
+        # We have the bmap, just read it and yield block ranges
+        xml = self._xml
+        xml_bmap = xml.find("BlockMap")
+
+        for xml_element in xml_bmap.findall("Range"):
+            blocks_range = xml_element.text.strip()
+            # The range of blocks has the "X - Y" format, or it can be just "X"
+            # in old bmap format versions. First, split the blocks range string
+            # and strip white-spaces.
+            split = [x.strip() for x in blocks_range.split('-', 1)]
+
+            first = int(split[0])
+            if len(split) > 1:
+                last = int(split[1])
+                if first > last:
+                    raise Error("bad range (first > last): '%s'" % blocks_range)
+            else:
+                last = first
+
+            if self._cs_attrib_name in xml_element.attrib:
+                chksum = xml_element.attrib[self._cs_attrib_name]
+            else:
+                chksum = None
+
+            yield (first, last, chksum)
+
+    def _get_batches(self, first, last):
+        """
+        This is a helper generator which splits block ranges from the bmap file
+        to smaller batches. Indeed, we cannot read and write entire block
+        ranges from the image file, because a range can be very large. So we
+        perform the I/O in batches. Batch size is defined by the
+        '_batch_blocks' attribute. Thus, for each (first, last) block range,
+        the generator yields smaller (start, end, length) batch ranges, where:
+          * 'start' is the starting batch block number;
+          * 'last' is the ending batch block number;
+          * 'length' is the batch length in blocks (same as
+             'end' - 'start' + 1).
+        """
+
+        batch_blocks = self._batch_blocks
+
+        while first + batch_blocks - 1 <= last:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+            first += batch_blocks
+
+        batch_blocks = last - first + 1
+        if batch_blocks:
+            yield (first, first + batch_blocks - 1, batch_blocks)
+
+    def _get_data(self, verify):
+        """
+        This is generator  which reads the image file in '_batch_blocks' chunks
+        and yields ('type', 'start', 'end',  'buf) tuples, where:
+          * 'start' is the starting block number of the batch;
+          * 'end' is the last block of the batch;
+          * 'buf' a buffer containing the batch data.
+        """
+
+        try:
+            for (first, last, chksum) in self._get_block_ranges():
+                if verify and chksum:
+                    hash_obj = hashlib.new(self._cs_type)
+
+                self._f_image.seek(first * self.block_size)
+
+                iterator = self._get_batches(first, last)
+                for (start, end, length) in iterator:
+                    try:
+                        buf = self._f_image.read(length * self.block_size)
+                    except IOError as err:
+                        raise Error("error while reading blocks %d-%d of the "
+                                    "image file '%s': %s"
+                                    % (start, end, self._image_path, err))
+
+                    if not buf:
+                        self._batch_queue.put(None)
+                        return
+
+                    if verify and chksum:
+                        hash_obj.update(buf)
+
+                    blocks = (len(buf) + self.block_size - 1) / self.block_size
+                    self._log.debug("queueing %d blocks, queue length is %d" %
+                                    (blocks, self._batch_queue.qsize()))
+
+                    self._batch_queue.put(("range", start, start + blocks - 1,
+                                           buf))
+
+                if verify and chksum and hash_obj.hexdigest() != chksum:
+                    raise Error("checksum mismatch for blocks range %d-%d: "
+                                "calculated %s, should be %s (image file %s)"
+                                % (first, last, hash_obj.hexdigest(),
+                                   chksum, self._image_path))
+        # Silence pylint warning about catching too general exception
+        # pylint: disable=W0703
+        except Exception:
+            # pylint: enable=W0703
+            # In case of any exception - just pass it to the main thread
+            # through the queue.
+            self._batch_queue.put(("error", sys.exc_info()))
+
+        self._batch_queue.put(None)
+
+    def copy(self, sync=True, verify=True):
+        """
+        Copy the image to the destination file using bmap. The 'sync' argument
+        defines whether the destination file has to be synchronized upon
+        return.  The 'verify' argument defines whether the checksum has to be
+        verified while copying.
+        """
+
+        # Create the queue for block batches and start the reader thread, which
+        # will read the image in batches and put the results to '_batch_queue'.
+        self._batch_queue = Queue.Queue(self._batch_queue_len)
+        thread.start_new_thread(self._get_data, (verify, ))
+
+        blocks_written = 0
+        bytes_written = 0
+        fsync_last = 0
+
+        self._progress_started = False
+        self._progress_index = 0
+        self._progress_time = datetime.datetime.now()
+
+        # Read the image in '_batch_blocks' chunks and write them to the
+        # destination file
+        while True:
+            batch = self._batch_queue.get()
+            if batch is None:
+                # No more data, the image is written
+                break
+            elif batch[0] == "error":
+                # The reader thread encountered an error and passed us the
+                # exception.
+                exc_info = batch[1]
+                raise exc_info[0], exc_info[1], exc_info[2]
+
+            (start, end, buf) = batch[1:4]
+
+            assert len(buf) <= (end - start + 1) * self.block_size
+            assert len(buf) > (end - start) * self.block_size
+
+            self._f_dest.seek(start * self.block_size)
+
+            # Synchronize the destination file if we reached the watermark
+            if self._dest_fsync_watermark:
+                if blocks_written >= fsync_last + self._dest_fsync_watermark:
+                    fsync_last = blocks_written
+                    self.sync()
+
+            try:
+                self._f_dest.write(buf)
+            except IOError as err:
+                raise Error("error while writing blocks %d-%d of '%s': %s"
+                            % (start, end, self._dest_path, err))
+
+            self._batch_queue.task_done()
+            blocks_written += (end - start + 1)
+            bytes_written += len(buf)
+
+            self._update_progress(blocks_written)
+
+        if not self.image_size:
+            # The image size was unknown up until now, set it
+            self._set_image_size(bytes_written)
+
+        # This is just a sanity check - we should have written exactly
+        # 'mapped_cnt' blocks.
+        if blocks_written != self.mapped_cnt:
+            raise Error("wrote %u blocks from image '%s' to '%s', but should "
+                        "have %u - bmap file '%s' does not belong to this "
+                        "image"
+                        % (blocks_written, self._image_path, self._dest_path,
+                           self.mapped_cnt, self._bmap_path))
+
+        if self._dest_is_regfile:
+            # Make sure the destination file has the same size as the image
+            try:
+                os.ftruncate(self._f_dest.fileno(), self.image_size)
+            except OSError as err:
+                raise Error("cannot truncate file '%s': %s"
+                            % (self._dest_path, err))
+
+        try:
+            self._f_dest.flush()
+        except IOError as err:
+            raise Error("cannot flush '%s': %s" % (self._dest_path, err))
+
+        if sync:
+            self.sync()
+
+    def sync(self):
+        """
+        Synchronize the destination file to make sure all the data are actually
+        written to the disk.
+        """
+
+        if self._dest_supports_fsync:
+            try:
+                os.fsync(self._f_dest.fileno()),
+            except OSError as err:
+                raise Error("cannot synchronize '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+
+class BmapBdevCopy(BmapCopy):
+    """
+    This class is a specialized version of 'BmapCopy' which copies the image to
+    a block device. Unlike the base 'BmapCopy' class, this class does various
+    optimizations specific to block devices, e.g., switching to the 'noop' I/O
+    scheduler.
+    """
+
+    def __init__(self, image, dest, bmap=None, image_size=None, log=None):
+        """
+        The same as the constructor of the 'BmapCopy' base class, but adds
+        useful guard-checks specific to block devices.
+        """
+
+        # Call the base class constructor first
+        BmapCopy.__init__(self, image, dest, bmap, image_size, log=log)
+
+        self._dest_fsync_watermark = (6 * 1024 * 1024) / self.block_size
+
+        self._sysfs_base = None
+        self._sysfs_scheduler_path = None
+        self._sysfs_max_ratio_path = None
+        self._old_scheduler_value = None
+        self._old_max_ratio_value = None
+
+        # If the image size is known, check that it fits the block device
+        if self.image_size:
+            try:
+                bdev_size = os.lseek(self._f_dest.fileno(), 0, os.SEEK_END)
+                os.lseek(self._f_dest.fileno(), 0, os.SEEK_SET)
+            except OSError as err:
+                raise Error("cannot seed block device '%s': %s "
+                            % (self._dest_path, err.strerror))
+
+            if bdev_size < self.image_size:
+                raise Error("the image file '%s' has size %s and it will not "
+                            "fit the block device '%s' which has %s capacity"
+                            % (self._image_path, self.image_size_human,
+                               self._dest_path, human_size(bdev_size)))
+
+        # Construct the path to the sysfs directory of our block device
+        st_rdev = os.fstat(self._f_dest.fileno()).st_rdev
+        self._sysfs_base = "/sys/dev/block/%s:%s/" % \
+                           (os.major(st_rdev), os.minor(st_rdev))
+
+        # Check if the 'queue' sub-directory exists. If yes, then our block
+        # device is entire disk. Otherwise, it is a partition, in which case we
+        # need to go one level up in the sysfs hierarchy.
+        if not os.path.exists(self._sysfs_base + "queue"):
+            self._sysfs_base = self._sysfs_base + "../"
+
+        self._sysfs_scheduler_path = self._sysfs_base + "queue/scheduler"
+        self._sysfs_max_ratio_path = self._sysfs_base + "bdi/max_ratio"
+
+    def _tune_block_device(self):
+        """
+        Tune the block device for better performance:
+        1. Switch to the 'noop' I/O scheduler if it is available - sequential
+           write to the block device becomes a lot faster comparing to CFQ.
+        2. Limit the write buffering - we do not need the kernel to buffer a
+           lot of the data we send to the block device, because we write
+           sequentially. Limit the buffering.
+
+        The old settings are saved in order to be able to restore them later.
+        """
+        # Switch to the 'noop' I/O scheduler
+        try:
+            with open(self._sysfs_scheduler_path, "r+") as f_scheduler:
+                contents = f_scheduler.read()
+                f_scheduler.seek(0)
+                f_scheduler.write("noop")
+        except IOError as err:
+            self._log.warning("failed to enable I/O optimization, expect "
+                              "suboptimal speed (reason: cannot switch "
+                              "to the 'noop' I/O scheduler: %s)" % err)
+        else:
+            # The file contains a list of schedulers with the current
+            # scheduler in square brackets, e.g., "noop deadline [cfq]".
+            # Fetch the name of the current scheduler.
+            import re
+
+            match = re.match(r'.*\[(.+)\].*', contents)
+            if match:
+                self._old_scheduler_value = match.group(1)
+
+        # Limit the write buffering, because we do not need too much of it when
+        # writing sequntially. Excessive buffering makes some systems not very
+        # responsive, e.g., this was observed in Fedora 17.
+        try:
+            with open(self._sysfs_max_ratio_path, "r+") as f_ratio:
+                self._old_max_ratio_value = f_ratio.read()
+                f_ratio.seek(0)
+                f_ratio.write("1")
+        except IOError as err:
+            self._log.warning("failed to disable excessive buffering, expect "
+                              "worse system responsiveness (reason: cannot set "
+                              "max. I/O ratio to 1: %s)" % err)
+
+    def _restore_bdev_settings(self):
+        """
+        Restore old block device settings which we changed in
+        '_tune_block_device()'.
+        """
+
+        if self._old_scheduler_value is not None:
+            try:
+                with open(self._sysfs_scheduler_path, "w") as f_scheduler:
+                    f_scheduler.write(self._old_scheduler_value)
+            except IOError as err:
+                raise Error("cannot restore the '%s' I/O scheduler: %s"
+                            % (self._old_scheduler_value, err))
+
+        if self._old_max_ratio_value is not None:
+            try:
+                with open(self._sysfs_max_ratio_path, "w") as f_ratio:
+                    f_ratio.write(self._old_max_ratio_value)
+            except IOError as err:
+                raise Error("cannot set the max. I/O ratio back to '%s': %s"
+                            % (self._old_max_ratio_value, err))
+
+    def copy(self, sync=True, verify=True):
+        """
+        The same as in the base class but tunes the block device for better
+        performance before starting writing. Additionally, it forces block
+        device synchronization from time to time in order to make sure we do
+        not get stuck in 'fsync()' for too long time. The problem is that the
+        kernel synchronizes block devices when the file is closed. And the
+        result is that if the user interrupts us while we are copying the data,
+        the program will be blocked in 'close()' waiting for the block device
+        synchronization, which may last minutes for slow USB stick. This is
+        very bad user experience, and we work around this effect by
+        synchronizing from time to time.
+        """
+
+        self._tune_block_device()
+
+        try:
+            BmapCopy.copy(self, sync, verify)
+        except:
+            raise
+        finally:
+            self._restore_bdev_settings()
diff --git a/tests/oldcodebase/__init__.py b/tests/oldcodebase/__init__.py
new file mode 100644
index 0000000..e69de29
-- 
1.8.1.4



From dedekind1 at gmail.com  Thu Nov  7 10:28:40 2013
From: dedekind1 at gmail.com (Artem Bityutskiy)
Date: Thu,  7 Nov 2013 17:28:40 +0200
Subject: [bmap-tools] Announcement: bmap-tools-3.1 is out!
Message-ID: <1383838120-14765-1-git-send-email-dedekind1@gmail.com>

Bmap-tools version 3.1 is out!

Release notes: http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/release-3.0:/docs/RELEASE_NOTES
Tarball: ftp://ftp.infradead.org/pub/bmap-tools/

Packages for various distributions are available here:
http://download.tizen.org/tools/pre-release/

At some later point they will be propagated to here:
http://download.tizen.org/tools/latest-release/

--
Artem Bityutskiy


