Detecting container initialization on DATADIR is fragile

[alexjh]

I'm using mysql 5.7.22 from the Kubernetes helm chart, but I think the issue is in the entrypoint script of the underlying container.

When installing the mysql helm chart on a busy system I've seen kube restart the mysql container when it's past the point where it's created $DATADIR/mysql, but before the rest of the script
completed.

mysql/5.7/docker-entrypoint.sh

Line 98 in fc3e856

if [ ! -d "$DATADIR/mysql" ]; then

In one example it looks like it happened after the time zone information was loaded.

$ kubectl -n mysql logs mysql-c6f58677-64frn --previous
...
2018-06-06T18:43:57.236675Z 0 [Warning] CA certificate ca.pem is self signed.
2018-06-06T18:43:57.309617Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2018-06-06T18:43:57.321969Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.322025Z 0 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.322041Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.322078Z 0 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.322087Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.322102Z 0 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.348619Z 0 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.348674Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-06-06T18:43:57.404561Z 0 [Note] Event Scheduler: Loaded 0 events
2018-06-06T18:43:57.404806Z 0 [Note] mysqld: ready for connections.
Version: '5.7.22'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server (GPL)
2018-06-06T18:44:01.967527Z 4 [Note] Access denied for user 'root'@'localhost' (using password: YES)
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.

It never got to the point where the permissions for remote root access was granted, resulting in a running container that could only be connected to from localhost.

I think this could be improved by putting a marker file down at the end of the init section, and checking for that instead of $DATADIR/mysql.

[yosifkit]

Sorry for the long tail.

While that could be a partial solution if the data directory is only created by the entrypoint but would then prevent a user from providing their own directory from a host installed MySQL (or an old version of the container). If we did only check for a custom marker file, then mysqld --initialize-insecure would most likely fail to run since it requires an empty directory.

I think the real solution is to increase your health timeout on first start so that Kubernetes doesn't just kill your container before it has finished starting.