[aiaiai PATCH v3 11/12] queue: add scripts to enable creation of combined queue branch

Artem Bityutskiy artem.bityutskiy at linux.intel.com
Fri Feb 7 02:36:16 PST 2014


From: Jacob Keller <jacob.e.keller at intel.com>

Our development works based on queue patches, where each driver has its
own queue for development. There is no easy way to automatically
determine which queue a patch should apply to, and developers are not
strict enough in their patch prefixes, nor can I enforce them to do the
correct thing. Instead, I provide a script which will generate a
specialized combined queue which merges all of the development queues
into a single (linear) history. This linearity is important for the
specalized version detection, so I used a cherry-pick to perform the
creation.

This setup is intended for use with a separate public tree that aiaiai will
use. The script will push the change of the aiaiai-queue branch directly to
the public (bare) repository. The advantage to this, is that the update
appears "atomic", so that before the push, it is a correct (out-dated) queue,
and after the push it is a correct (up-to-date) queue. This should enable
prevention of a clone in between "correct" queue points, as we will already be
resetting to the correct thing, rather than trying to reset somewhere
inbetween the master and tip-of-queue.

Signed-off-by: Jacob Keller <jacob.e.keller at intel.com>
---
 0 files changed

diff --git a/queue/aiaiai-rebase-queues b/queue/aiaiai-rebase-queues
new file mode 100755
index 0000000..d00b3f7
--- /dev/null
+++ b/queue/aiaiai-rebase-queues
@@ -0,0 +1,170 @@
+#!/bin/bash -efu
+#
+# Create "merged" ND queue branch by rebasing each of the queue branches onto
+# aiaiai-queue in sequence. This creates a single (non-merged) branch history
+# of each of the current queues combined.
+#
+
+# Name of base branch
+: ${AIAIAI_MASTER:="master"}
+export AIAIAI_MASTER
+
+# The name of the remote to fetch
+: ${AIAIAI_REMOTE:="origin"}
+export AIAIAI_REMOTE
+
+# The name of the public repo
+: ${AIAIAI_PUBLIC:=""}
+export AIAIAI_PUBLIC
+
+# The queues to rebase into the new combined branch
+: ${AIAIAI_QUEUES:="ixgbe-queue:ixgb-queue:igb-queue:i40e-queue:e1000e-queue:e1000-queue:e100-queue:core-queue"}
+export AIAIAI_QUEUES
+
+# How long to wait between each check
+: ${AIAIAI_WAIT_TIME:="1m"}
+export AIAIAI_WAIT_TIME
+
+# Whether to force an update
+: ${AIAIAI_FORCE_UPDATE:=""}
+export AIAIAI_FORCE_UPDATE
+
+PROG="${0##*/}"
+
+show_usage() {
+	cat <<-EOF
+Usage: $PROG <repository> <branch>
+
+This script uses a process of sequential rebase,fastforward calls in order to
+create a branch which combines the queues into a single branch with a
+sequentiel linear history for use by AiAiAi.
+
+Note that it will destroy the local branches, so should not be used on a tree
+you actually work on. Instead, clone the upstream tree (Jeff Kirsher's
+net-next) and perform the creation on a seperate tree.
+
+The process used relies on the assumption that rebase will not fail. This is a
+safe assumption because each queue represents a seperate driver, and therefor
+will not have any conflicts with other driver changes. This assumption
+shouldn't change in the future.
+
+The script does perform a check to see if it needs an update, but this does
+rely on an already existing nd-queue and uses the assumption that nothing else
+is performing a git-fetch on this repository.
+
+Note that when the script updates it does so destructively, and will destroy
+the current queue setup. However, since it is automatically generated anyways
+this should not cause issues.
+EOF
+}
+
+# Check if the current aiaiai-queue commit exists, and is up to date. This
+# method ensures no break even if something else performs a git-fetch
+queue_has_updated() {
+	local current new queue commit
+	queue=$1
+	current=$(git log --reverse --oneline "$AIAIAI_REMOTE/$AIAIAI_MASTER..$AIAIAI_REMOTE/$queue")
+	new=$(git log -1 $branch --pretty="format:%B" | awk '/-- '"$queue"' --/ {flag=1;next} /^$/ {flag=0} flag {print}')
+
+	if diff -u <(echo "$new" | sort) <(echo "$current" | sort) >&2; then
+		return 1
+	else
+		return 0
+	fi
+}
+
+# Make sure we got a branch and a repository
+if [ $# -lt 2 ]; then
+	printf "Missing required arguments\n"
+	show_usage
+	exit 1
+fi
+
+repository=$1; shift
+branch=$1; shift
+
+if [ ! -d $repository ]; then
+	printf "directory does not exist\n"
+	show_usage
+	exit 1
+fi
+
+cd $repository
+
+# Fetch any new changes
+git fetch --quiet $AIAIAI_REMOTE
+
+updated=
+# Check to see if branch already exists
+if ! ( git show-ref --verify --quiet refs/heads/$branch ); then
+	updated=1
+	printf "$branch does not exist yet\n"
+else
+	# Check to see if anything has changed
+	IFS=":"
+	for queue in ${AIAIAI_QUEUES}; do
+		if queue_has_updated $queue; then
+			updated=1
+			printf "The $queue branch has changed, update of $branch is required\n"
+		fi
+	done
+	unset IFS
+fi
+
+# We require a reset/update or we are forced
+if [ -n "$updated" ] || [ -n "$AIAIAI_FORCE_UPDATE" ]; then
+	# Perform the actual update
+	printf "Performing update of $branch...\n"
+	# update the remotes
+	git fetch --quiet $AIAIAI_REMOTE
+	# Create clone of master
+	git checkout -B $branch $AIAIAI_REMOTE/$AIAIAI_MASTER
+	# quit the last cherry-pick incase we failed
+	git cherry-pick --quit
+	# loop through each queue and perform rebase
+	( IFS=":"
+	for queue in $AIAIAI_QUEUES; do
+		printf -- "%s\n" "--->Cherry-picking $queue<---"
+		# Cherry pick every commit on the queue branch
+		if [ -n "$(git rev-list $AIAIAI_REMOTE/$AIAIAI_MASTER..$AIAIAI_REMOTE/$queue)" ]; then
+			git cherry-pick -s -- $AIAIAI_REMOTE/$AIAIAI_MASTER..$AIAIAI_REMOTE/$queue
+		fi
+	done
+	)
+	# Checkout final combined queue
+	git checkout $branch
+
+	date=$(date -I'ns')
+
+	# Create an explanatory (empty) commit for the tip of the queue
+	git commit --allow-empty -s -F - <<EOF
+aiaiai-queue: $(date --date=$date)
+
+This is a combined queue containing all the current patches from the ND
+upstream queues. This branch was created via git-cherry-pick, in order to
+facilitate finding and replacing a new version of a commit in-place.
+
+It combines the following patches (showing their commit id):
+
+$(IFS=":"; for queue in $AIAIAI_QUEUES; do echo "-- $queue --"; echo "$(git log --reverse --oneline "$AIAIAI_REMOTE/$AIAIAI_MASTER..$AIAIAI_REMOTE/$queue")"; echo ""; done )
+
+The state of the master upstream branch:
+$(git describe $AIAIAI_REMOTE/$AIAIAI_MASTER)
+$(git log -1 --oneline $AIAIAI_REMOTE/$AIAIAI_MASTER)
+
+This branch has been tagged "aiaiai-$(date --date=$date +%FT%H-%M-%S,%N%z)
+
+EOF
+
+	# Tag the commit so that we can reference it later.
+	git tag "aiaiai-$(date --date=$date +%FT%H-%M-%S,%N%z)"
+
+	# Force update public if it has been specified
+	if [ -n $AIAIAI_PUBLIC ]; then
+		echo "Updating $AIAIAI_PUBLIC..."
+		git push --force -- $AIAIAI_PUBLIC "aiaiai-$(date --date=$date +%FT%H-%M-%S,%N%z)" $branch
+	fi
+
+else
+	printf "No update needed...\n"
+fi
diff --git a/queue/aiaiai-watch-queue b/queue/aiaiai-watch-queue
new file mode 100755
index 0000000..87bf29f
--- /dev/null
+++ b/queue/aiaiai-watch-queue
@@ -0,0 +1,56 @@
+#!/bin/bash -efu
+
+srcdir="$(readlink -ev -- ${0%/*}/..)"
+PATH="$srcdir:$srcdir/queue:$srcdir/external/libshell:$PATH${AI_PATH:+:$AI_PATH}"
+
+# The master branch to start the rebase
+: ${AIAIAI_MASTER:="master"}
+export AIAIAI_MASTER
+
+# The remote name
+: ${AIAIAI_REMOTE:="origin"}
+export AIAIAI_REMOTE
+
+# The name of the public repo
+: ${AIAIAI_PUBLIC:=""}
+export AIAIAI_PUBLIC
+
+# The queues to rebase into the new combined queue
+: ${AIAIAI_QUEUES:="ixgbe-queue:ixgb-queue:igb-queue:i40e-queue:e1000e-queue:e1000-queue:e100-queue:core-queue"}
+export AIAIAI_QUEUES
+
+# How long to wait between each check
+: ${AIAIAI_WAIT_TIME:="1m"}
+export AIAIAI_WAIT_TIME
+
+# Whether to force an update
+: ${AIAIAI_FORCE_UPDATE:=""}
+export AIAIAI_FORCE_UPDATE
+
+# The location of the repository
+: ${AIAIAI_REPO:="/home/aiaiai/git/net-next"}
+export AIAIAI_REPO
+
+# The name of the branch to create
+: ${AIAIAI_BRANCH:="aiaiai-queue"}
+export AIAIAI_BRANCH
+
+failed=""
+
+# Loop forever and run the rebase-queues script
+while :; do
+	# this script will perform the update (if required)
+	if aiaiai-rebase-queues $AIAIAI_REPO $AIAIAI_BRANCH; then
+		failed=""
+	else
+		if [ -n "$failed" ]; then
+			echo "aiaiai-rebase-queues failed twice in a row. Aborting."
+			exit 1
+		else
+			failed="1"
+		fi
+	fi
+	# Sleep for 1minute between checks
+	sleep 1m
+done
+


-- 
Best Regards,
Artem Bityutskiy




More information about the aiaiai mailing list