db schema differs between master and new installed member

problem:

The problem described below occurred on XenServer 8.2.1 but has also been observed on XenServer 8.2 before.

The problem description is about XenServer 8.2.1 and therefore contains XenServer 8.2.1 specific updates/update numbers.

XenServer 8.2.1 update XS82ECU1017 contains an update for openvswitch. This openvswitch update comes amongst others with a new database schema version. The new database schema version is 5.603, the version used before was 5.602.

Error message in log or on screen which will be thrown by script sub-38_pool_join-default.sh during the join of a new XenServer to a pool:

ERROR : cannot join pool [The host joining the pool must have the same database schema as the pool master. host_db_schema: 5.602 master_db_schema: 5.603] - abort

investigation:

The following commands can be used to get relevant entries from XenServers main logfile xensource.log:

The grep with the version numbers of course is specific to the currently used db_schema versions at the time of this writing (09.02.2024) and the used XenServer version which is 8.2.1

The following grep commands all provide log entries which help to better understand what is happening during openvswitch db_schema upgrade.

egrep -w '5, 602|5, 603|5.602|5.603' /var/log/xensource.log*
 
grep hi-level /var/log/xensource.log*
 
grep db_hiupgrade /var/log/xensource.log
 
grep "database engine" /var/log/xensource.log*
 
grep bumping /var/log/xensource.log*

The following is the line from the xensource.log which finally shows the version upgrade:

v2xen-4 xapi: [debug||0 |hi-level database upgrade D:bad5ea3985c4|db_hiupgrade] Upgrade rules applied, bumping schema version to 5.603

The db_schema version can be queried from XAPI with the following commands:

xe host-param-get uuid=<host-uuid> param-name=software-version
xe host-param-get uuid=<host-uuid> param-name=software-version param-key=db_schema

Example:

# xe host-param-get uuid=a7906aab-d44b-48a5-95dd-605d4e45d011 param-name=software-version param-key=db_schema
5.603

solution:

The only way to solve the problem is to reboot the XenServers after installing all XenServer updates. This can be done by simply writing the name of the last update WITHOUT a prefix (“!” or “_”) into the update.lst file.