[buildbot] phase1,phase2: improve round robin builds

[LEDE Commits]

ynezz pushed a commit to buildbot.git, branch master:
https://git.openwrt.org/87ed6cf749bee5a22ffed22fad248d78a1144352

commit 87ed6cf749bee5a22ffed22fad248d78a1144352
Author: Petr Štetiar <ynezz at true.cz>
AuthorDate: Wed Jul 21 21:31:28 2021 +0200

    phase1,phase2: improve round robin builds
    
    There seems to be some issue with database updates, where database
    updates are asynchronous and thus the status update of the previous
    build (complete_at DB column) might take little bit longer, then
    preparation of the next/current build.
    
    This is issue during job prioritization as this might result in the
    build of the same builder twice in a row, for example:
    
     2021-06-22 02:42:54+0000 [-]  <Build mipsel_24kc number:95 results:success>: build finished
     2021-06-22 02:42:55+0000 [-] prioritizeBuilders:          mipsel_24kc complete_at: 2021-06-21 20:17:14+00:00
     2021-06-22 03:14:30+0000 [-] prioritizeBuilders:          mipsel_24kc complete_at: 2021-06-22 02:42:54+00:00
    
    Build finishes at 02:42:54, scheduler then asks for next build at
    02:42:55, but the build still has the old complete_at timestamp
    2021-06-21 20:17:14 instead of the correct one 2021-06-22 02:42:54, thus
    scheduling the build of oldest mipsel_24kc builder one more time.
    
    This is so far very promising workaround attempt which checks latest
    builder complete_at in builds table which seems to be updated faster,
    thus using greater complete_at value seems to work for now.
    
    References: https://github.com/buildbot/buildbot/issues/4592#issuecomment-801163587
    Signed-off-by: Petr Štetiar <ynezz at true.cz>
---
 phase1/master.cfg | 16 +++++++++++++++-
 phase2/master.cfg | 16 +++++++++++++++-
 2 files changed, 30 insertions(+), 2 deletions(-)

diff --git a/phase1/master.cfg b/phase1/master.cfg
index a85382a..be3a894 100644
--- a/phase1/master.cfg
+++ b/phase1/master.cfg
@@ -154,7 +154,21 @@ def getNewestCompleteTime(bldr):
 	if not completed:
 		return
 
-	return completed[0]['complete_at']
+	complete_at = completed[0]['complete_at']
+
+	last_build = yield bldr.master.data.get(
+			('builds', ),
+			[
+				resultspec.Filter('builderid', 'eq', [bldrid]),
+			],
+			order=['-started_at'], limit=1)
+
+	if last_build and last_build[0]:
+		last_complete_at = last_build[0]['complete_at']
+		if last_complete_at and (last_complete_at > complete_at):
+                   return last_complete_at
+
+	return complete_at
 
 @defer.inlineCallbacks
 def prioritizeBuilders(master, builders):
diff --git a/phase2/master.cfg b/phase2/master.cfg
index b9342cf..7742ad6 100644
--- a/phase2/master.cfg
+++ b/phase2/master.cfg
@@ -351,7 +351,21 @@ def getNewestCompleteTime(bldr):
 	if not completed:
 		return
 
-	return completed[0]['complete_at']
+	complete_at = completed[0]['complete_at']
+
+	last_build = yield bldr.master.data.get(
+			('builds', ),
+			[
+				resultspec.Filter('builderid', 'eq', [bldrid]),
+			],
+			order=['-started_at'], limit=1)
+
+	if last_build and last_build[0]:
+		last_complete_at = last_build[0]['complete_at']
+		if last_complete_at and (last_complete_at > complete_at):
+                   return last_complete_at
+
+	return complete_at
 
 @defer.inlineCallbacks
 def prioritizeBuilders(master, builders):