Lessons learned upgrading PHP and MariaDB for Nextcloud

Introduction

A few years ago, I decided to try hosting my own Nextcloud instance. The project was a very good introduction to self-hosting, and I would definitely recommend it to anyone wanting to get into the topic. The instance in itself ended up being very useful to me, especially at a time when I was running a completely “de-googled” phone, and needed to routinely sync-up a few files between my phone, and desktop.

Fast-forward a couple years, and I have not been using my Nextcloud instance as much as I initially thought I would. My server had been collecting dust, or more appropriately collecting vulnerabilities, as newer and newer releases of Nextcloud were published. My instance was still running the (by now out-of-support) version I had initially installed, and running the Security Scan offered by Nextcloud gave my server a failing grade. As such, it was time to take back responsibility for my server and upgrade it to a point where it was up to the security requirements expected of an Internet-facing machine.

In this article, I would like to summarize the main difficulties and issues I faced during the update process.

Updating Nextcloud

As mentioned earlier, my Nextcloud instance was very out of date. Specifically, my instance was still running on version 20.0.1 while the latest released version was 26.0.2. As such, I had to walk my way through six major releases. The update process can be conducted fully using the web-interface available to the instance’s administrator under the menu Administration settings > Overview¹. Successively updating from one release to the next boiled down to the same three following steps:

• Step 1: Resolution of any outstanding errors and warnings in the Nextcloud Overview menu. These warnings include missing indices in the used database, required PHP settings, etc. The warning messages also usually mention the steps needed to solve the corresponding issues.
• Step 2: Upgrading PHP and/or MariaDB if required. This is because newer releases of Nextcloud require newer versions of both. Upgrading to a newer Nextcloud release is blocked in the administration settings if the installed PHP and MariaDB versions do not match the release’s requirements.
• Step 3: When the PHP and MariaDB requirements are fulfilled, the upgrade to the next release is made available in the administration settings. The update can then be performed using the web-based updater, available under the same menu. The upgrade process in itself is completely automated, and there usually is no need for manual intervention.

The steps above, worked for me in all but one of the consecutive updates I had to conduct on my instance. The issue I faced in that one problematic update is covered in the next section.

Error: Web-updater not responding, and showing “White Screen Of Death”

When upgrading to release 25 of Nextcloud, after triggering the upgrade through the web-updater menu (Step 3 from the previous section), the upgrade seemed to stop unexpectedly. This was followed by the instance’s web interface now only showing a white blank screen, not allowing for any user interaction. The solution was to log in to the server hosting my instance using SSH, and force the upgrade manually using the following commands:

1
2
3
4

root@localhost:/# cd /path/to/nextcloud/directory
root@localhost:/# sudo -u www-data /usr/bin/phpX.Y ./occ maintenance:mode --on
root@localhost:/# sudo -u www-data /usr/bin/phpX.Y ./occ upgrade
root@localhost:/# sudo -u www-data /usr/bin/phpX.Y ./occ maintenance:mode --off

Here are a few notes about the commands above:

• The commands make use of occ, the OwnCloud Console tool², which is a PHP script available in the directory Nextcloud is installed in, hence the first cd command.
• The script needs to be executed using the same version of PHP used by Nextcloud. This means that in the case multiple PHP versions are installed on the server (as was the case for me), the full path to the version to be used needs to be used when calling the occ script.
• Additionally, because the occ script will make modifications to the Nextcloud web-server configuration files, it needs to be executed as the HTTP user. Running it as the root user might cause the modified files to be set-up with the wrong access rights and ownership, causing later issues on the web-server². In the case of the Debian/Ubuntu distributions, the HTTP user is www-data, hence the sudo -u www-data prefixed to the commands above.
• Finally, in order to prevent users from logging in to the instance while we are upgrading it, we enable maintenance mode before we initiate the upgrade, and disable it once we are done.

After performing the upgrade manually using the commands above, my Nextcloud web-interface was back to functioning correctly.

Updating MariaDB

At the start of this whole process, I was running Nextcloud version 20, and the initial upgrade to version 21 required me to first update MariaDB on my server to version 10.2 or higher³. Upgrading from the version of MariaDB my server was running (10.1) to version 10.2 is documented in a specific article on the MariaDB Knowledge Base⁴. One issue I did face after following the installation steps mentioned in that article is detailed in the following section.

Error: mysql_upgrade fails to connect to database

After installing the new version of MariaDB and starting the mariadb service, I wanted to make sure it was functioning correctly. To do this I looked into the service’s status using the command service mariadb status (executed as root). The command’s output also includes the last few lines from the service’s log. These are shown below:

1
2
3
4
5
6
7
8
9

root@localhost:/# service mariadb status
<...service status output...>
May 29 14:53:55 localhost systemd[1]: Started MariaDB 10.2.44 database server.
May 29 14:53:55 localhost /etc/mysql/debian-start[24569]: Upgrading MySQL tables if necessary.
May 29 14:53:55 localhost /etc/mysql/debian-start[24572]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
<...more log output...>
May 29 14:53:55 localhost /etc/mysql/debian-start[24572]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
May 29 14:53:55 localhost /etc/mysql/debian-start[24572]: FATAL ERROR: Upgrade failed
May 29 14:53:55 localhost /etc/mysql/debian-start[24578]: Checking for insecure root accounts.

What these lines from the log show, is that after MariaDB starts-up, it attempts to run mysql_upgrade. This is meant to perform checks on the tables in the mysql database following the MariaDB update, and to ensure that they are compatible with the new version. However, the call to mysql_upgrade fails, as the process attempts but fails to log in to the database as the root user.

I still am unsure about the exact reason why this error was now showing in the mariadb logs. The most plausible theory I came up with is that, I believe when I initially set up my mariadb database years ago, I did so with a password-less root user. As such, previously, the mysql_upgrade could log in to the mysql database without providing a password. During the MariaDB update, I was prompted for a root password, and did provide one. This meant that the root account on the mysql database was now password-protected, and mysql_upgrade had no access to the used password.

I figured out two possible solutions to this issue:

• Solution 1: Providing mysql_upgrade with the new root password. This could be done by adding the following section to one of MariaDB’s option files:

  1 2 3

  [mysql_upgrade] user=root password=<root_password_here>

  In my case I initially added this section to the /etc/mysql/debian.cnf file. However, comments in the file explicitly advise against editing it, as it is automatically generated. A better place to store the password would probably be /etc/mysql/mariadb.conf.d/50-mysql-clients.cnf as it already contains a [mysql_upgrade] section. However, access rights to the file will probably need to be changed to prohibit read access from other users, so that the root password would not be exposed that way. I have not tried this, as I was not sure such rights are not needed by any other mysql client program.

• Solution 2: Switching from password to Unix socket authentication. This is probably the more practical, more secure, and all around better solution. Instead of requiring users to provide a password to authenticate themselves, MariaDB can be configured with a socket file to use for each user. When a user is attempting to log in to the server, MariaDB can check their uid and verify that it matches that of the process connected to the socket configured for that user⁵.
  Switching to Unix socket authentication requires reconfiguration in two places:
  • The socket authentication module needs to be activated on the server. This can be achieved by adding the following section to one of MariaDB’s option files.

    1 2	[mysqld] plugin_load_add = auth_socket.so

    In my case I added this section to /etc/mysql/my.cnf. Additionally, it is also possible to add, the plugin_load_add parameter to the [mariadb] section instead of the [mysqld] section⁵.

  • The user table needs to be updated to use socket- instead of password-based authentication for the root user. This is achieved by logging in to the mysql database as root, and issuing the first two MySQL commands below.

    1 2 3 4 5 6 7 8 9

    MariaDB [mysql]> UPDATE user SET plugin="unix_socket" WHERE User='root'; MariaDB [mysql]> FLUSH PRIVILEGES; MariaDB [mysql]> SELECT host, user, password, plugin FROM mysql.user LIMIT 0,1; +-----------+------+-------------------------------------------+-------------+ | host | user | password | plugin | +-----------+------+-------------------------------------------+-------------+ | localhost | root | *<password_hash_goes_here> | unix_socket | +-----------+------+-------------------------------------------+-------------+ 1 row in set (0.00 sec)

    The output of the last MySQL command above shows how the entry for the root user in the user table should look like once these modifications are performed.

  • Finally, the mariadb service needs to be restarted for the changes to take effect:

    1	root@localhost:/# service mariadb restart

After using either of these fixes, examining the status of the mariadb service shows that the issue has been fixed, and that mysql_upgrade is able to log in as root into the mysql database:

1
2
3
4
5
6
7
8
9

root@localhost:/# service mariadb status
<...service status output...>
May 29 19:43:16 localhost systemd[1]: Started MariaDB 10.2.44 database server.
May 29 19:43:16 localhost /etc/mysql/debian-start[29415]: Upgrading MySQL tables if necessary.
May 29 19:43:16 localhost /etc/mysql/debian-start[29418]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
<...more log output...>
May 29 19:43:16 localhost /etc/mysql/debian-start[29418]: This installation of MariaDB is already upgraded to 10.2.44-MariaDB.
May 29 19:43:16 localhost /etc/mysql/debian-start[29418]: There is no need to run mysql_upgrade again for 10.2.44-MariaDB.
May 29 19:43:16 localhost /etc/mysql/debian-start[29418]: You can use --force if you still want to run mysql_upgrade

A final and very important note here concerns a mistake which caused me a lot of headache: The plugin which needs to be used in the user table is called unix_socket, while the module that we instructed the MariaDB server to load in the option file is called auth_socket. The difference between the names of the plugin and the module is even more confusing because MySQL uses a plugin named auth_socket to implement socket authentication. It is very easy to mix-up the two name, if like me you were frantically looking through dozens of forum posts for a solution to your problem. This mix-up on my part lead me to use auth_socket instead of unix_socket in my user table. This caused mariadb to fail on startup with the error:

1

ERROR 1524 (HY000): Plugin 'auth_socket' is not loaded

In fact, if like me you made the mistake of initially using auth_socket in your user table, you will not even be able to manually log in to the mysql database as root, and will be denied access with the same error. In this case, the only recourse is to stop mysql and start it in safe mode (using mysql_safe) while completely disabling authentication checks (using option --skip-grant-tables). You can then log in to the mysql database as root and modify the user table correctly. Once the table is updated, you can restart mysql. If you do end-up resorting to this, it is a good idea to also disable remote access to the database during this manipulation (using option --skip-networking), since authentication checks would be disabled. The commands below show these steps:

1
2
3
4
5
6
7
8
9

root@localhost:/# service mysql stop
root@localhost:/# mysqld_safe --skip-grant-tables --skip-networking &
root@localhost:/# mysql mysql

MariaDB [mysql]> UPDATE user SET plugin="unix_socket" WHERE User='root';
MariaDB [mysql]> FLUSH PRIVILEGES;
MariaDB [mysql]> EXIT;

root@localhost:/# service mysql restart

Updating PHP

The Nextcloud backend is written in PHP. As such, running a Nextcloud instance requires the use of a PHP interpreter as well as a multitude of PHP modules for the web-server which also need to be kept up-to-date. More recent releases of Nextcloud require more recent versions of the PHP interpreter, as such, it is necessary to upgrade PHP before the option to update Nextcloud is made available in its administration settings.

Upgrading PHP is usually pretty straight-forward, and consists of the following steps:

• Step 1: Installing the newer phpX.Y-fpm module and any required PHP-modules.
• Step 2: Disabling the web-server module and configuration for the older PHP version and enabling those for the newly installed version. If like me you are using Apache as a web-server, this is achieved using the a2enmod and a2dismod commands. As an example, disabling php7.3 and enabling php7.4 would look as follows:

  1 2	root@localhost:/# a2dismod php7.3 root@localhost:/# a2enmod php7.4

  The command below can be used to check which version of PHP the Apache server is using. Following the example above, the output is showing Apache to be using php7.4.

  1 2	root@localhost:/# ls /etc/apache2/mods-enabled/php* /etc/apache2/mods-enabled/php7.4.conf /etc/apache2/mods-enabled/php7.4.load

• Step 3: Making any necessary modifications to the configuration file of the newly installed PHP version. For PHP version X.Y this file can be found under /etc/php/X.Y/fpm/php.ini.

As I was upgrading through different versions of PHP, the three previous steps usually worked flawlessly. However, during a couple upgrades, I did encounter some errors which I am listing in the following sections, along with the corresponding solutions.

Error: Module phpX.Y does not exist!

I faced this error when attempting to enable the phpX.Y module on Apache (using a2enmod phpX.Y), after having only installed the corresponding phpX.Y-fpm package. As it turns out, the phpX.Y-fpm package, provides only the binary PHP interpreter. In order for Apache to be able to use the interpreter, the corresponding Apache module needs to be installed. This can be achieved by installing the corresponding libapache2-mod-phpX.Y package.

Error: PHP memory_limit setting not taking effect

Nextcloud recommends to use a memory_limit for the PHP interpreter of 512MB. This parameter can be configured in the PHP interpreter’s configuration file /etc/php/X.Y/fpm/php.ini (where X.Y is the PHP version used by Nextcloud). However, in order for any changes in this file to take effect, it is required to restart the PHP service:

1

root@localhost:/# service phpX.Y-fpm restart

Error: Web-server not responding, showing vague error message

In my case, this occurred after upgrading to php7.4. Following the upgrade, my instance’s web-interface would not load anymore, showing only a blank page, with a vague error message recommending to contact the page’s admin (I did not think of taking a screenshot at the time). I solved this issue by running the following commands:

1
2
3

root@localhost:/# a2enmod proxy_fcgi setenvif
root@localhost:/# service php7.4-fpm restart
root@localhost:/# systemctl reload apache2

• The proxy_fcgi module is, as I understand it, a new PHP module which replaces older modules used to forward PHP script to the php-fpm process manager. I believe that, in my case it was not enabled by default after the upgrade, which required me to enable it manually.
• The setenvif sets environment variables for Apache based on those defined by proxy_fcgi.
• Finally we restart the php7.4-fpm service and reload the apache2 service configuration.

Conclusions

This whole ordeal has been a great learning experience for me. I managed to get a good look under the hood of my Nextcloud instance, and to understand how the different software components of my server operate and interact with each other. Luckily for me, I had stopped storing any private information or critical data on my instance for some time now, so hopefully the risk I was taking by leaving it fall behind on updates is limited. However, I now know better than to repeat this mistake, and will make sure to keep up with newer releases of Nextcloud, especially if I ever decide to start using this instance more seriously again.

References