Skip to content

feat: add option to wait on apt locks #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
L0RD-ZER0 wants to merge 2 commits into
base: master
Choose a base branch
from
Open

feat: add option to wait on apt locks #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f267288
feat: add option to wait on apt locks
L0RD-ZER0 Jun 24, 2026
8baec11
fix: address review changes
L0RD-ZER0 Jun 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 41 additions & 14 deletions functions
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,31 @@
# These functions have been adapted from
# https://github.com/dokku/dokku/blob/master/plugins/common/functions

# DPkg lock-wait tiers, activated by EE_APT_LOCK_WAIT. Duplicated from
# setup.sh so the tier vars stay defined (no-op) when this file is sourced
# standalone by migrate.sh / remote-migrate — note those scripts' own
# apt-get calls and remote (ssh) installs do not inherit the wait.
# Any non-empty EE_APT_LOCK_WAIT (e.g. =1) enables the dpkg lock-wait; it is an
# on/off flag, not a duration (the timeouts below are fixed). Tier variables
# are left empty (no-op) when unset.
if [ -n "${EE_APT_LOCK_WAIT:-}" ]; then
_EE_APT_LOCK_LIGHT_INSTALL="-o DPkg::Lock::Timeout=900"
_EE_APT_LOCK_INSTALL="-o DPkg::Lock::Timeout=1800"
else
_EE_APT_LOCK_LIGHT_INSTALL="${_EE_APT_LOCK_LIGHT_INSTALL:-}"
_EE_APT_LOCK_INSTALL="${_EE_APT_LOCK_INSTALL:-}"
fi

ee_apt_update() {
local i
for i in $(seq 1 30); do
apt-get update && return 0
ee_log_info2 "apt lists lock busy; retrying in 10s"
sleep 10
done
apt-get update
}
Comment on lines +21 to +29

@mrrobot47 mrrobot47 Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ee_apt_update retries on any non-zero apt-get update, not just a busy lock, and always prints "apt lists lock busy". Two consequences:

  • A genuinely broken repo (404 / nonexistent suite / GPG error — I confirmed these exit 100 on apt 2.8.3) gets retried 30×10s ≈ 5 min before the real error surfaces, with a misleading "lock busy" line hiding the actual cause. (A transient/unreachable mirror is not affected — apt-get update returns 0 there, so no retry.)
  • It's unconditional, so it runs even when EE_APT_LOCK_WAIT is unset — whereas the install tiers stay flag-gated. The two halves of the feature now disagree on whether lock-waiting is opt-in.

Consider retrying only when the failure is actually a lock, and surfacing other errors immediately. e.g. (the if-condition keeps it set -e-safe; mirror the same change in setup.sh's copy):

ee_apt_update() {
  local i err
  for i in $(seq 1 30); do
    if err="$(apt-get update 2>&1)"; then printf '%s\n' "$err"; return 0; fi
    printf '%s\n' "$err" >&2
    printf '%s' "$err" | grep -qiE "Could not get lock|Unable to lock" || return 1
    ee_log_info2 "apt lists lock busy; retrying in 10s"
    sleep 10
  done
  apt-get update
}


has_tty() {
declare desc="return 0 if we have a tty"
if [[ "$(/usr/bin/tty || true)" == "not a tty" ]]; then
Expand Down Expand Up @@ -220,6 +245,8 @@ get_ondrej_php_ppa_release_status() {
# unsatisfiable deps (e.g. a PPA build needing a libxml2 the OS lacks) are caught
# before touching the system.
ee_php_cli_installable() {
# apt-get install -s uses Debug::NoLocking
#
Comment on lines +248 to +249

@mrrobot47 mrrobot47 Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tighten this — line 249 is a whitespace-only # (no information), and the first line states a fact without the point. Per the concise-comments convention, say the why in one line:

Suggested change
# apt-get install -s uses Debug::NoLocking
#
# -s (simulate) takes no dpkg lock, so a lock-wait option here would be a no-op

apt-get install -s "php${1}-cli" >/dev/null 2>&1
}

Expand All @@ -228,8 +255,8 @@ ee_php_cli_installable() {
ee_php_add_thirdparty_repo() {
[ "$EE_PHP_THIRDPARTY_ADDED" == "1" ] && return 0
if [ "$EE_LINUX_DISTRO" == "Ubuntu" ]; then
apt-get install -y software-properties-common curl ca-certificates
add-apt-repository -y ppa:ondrej/php || true
apt-get $_EE_APT_LOCK_INSTALL install -y software-properties-common curl ca-certificates
(add-apt-repository -ny ppa:ondrej/php && ee_apt_update) || true

@mrrobot47 mrrobot47 Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The && ee_apt_update here is redundant: nothing between this line and the unconditional ee_apt_update at line 287 reads the apt lists (the codename probe via get_ondrej_php_ppa_release_status is an HTTP check), so line 287 already performs the one refresh needed before ee_php_cli_installable. On a persistent failure this also doubles the retry wait (~10 min) for a single repo-add. Drop it and let line 287 update once:

Suggested change
(add-apt-repository -ny ppa:ondrej/php && ee_apt_update) || true
add-apt-repository -ny ppa:ondrej/php || true

local want use="" c f
want="$(lsb_release -sc)"
if [ "$(get_ondrej_php_ppa_release_status "$want")" == "200" ]; then
Expand All @@ -251,13 +278,13 @@ ee_php_add_thirdparty_repo() {
return 1
fi
elif [ "$EE_LINUX_DISTRO" == "Debian" ]; then
apt-get install -y apt-transport-https lsb-release ca-certificates curl locales locales-all
apt-get $_EE_APT_LOCK_INSTALL install -y apt-transport-https lsb-release ca-certificates curl locales locales-all
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
curl -fsSL https://packages.sury.org/php/apt.gpg -o /etc/apt/trusted.gpg.d/php.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list
fi
apt-get update
ee_apt_update
EE_PHP_THIRDPARTY_ADDED=1
return 0
}
Expand All @@ -267,7 +294,7 @@ ee_php_drop_thirdparty_repo() {
[ "$EE_PHP_THIRDPARTY_ADDED" == "1" ] || return 0
if [ "$EE_LINUX_DISTRO" == "Ubuntu" ]; then
rm -f /etc/apt/sources.list.d/ondrej-ubuntu-php* /etc/apt/preferences.d/ee-ondrej-php
apt-get update
ee_apt_update
EE_PHP_THIRDPARTY_ADDED=""
fi
}
Expand All @@ -290,13 +317,13 @@ ee_select_and_install_php() {
# Native: drop any probe PPA so it isn't shadowed.
ee_php_drop_thirdparty_repo
ee_log_info2 "Installing PHP ${v} (native)"
apt-get install -y "php${v}-cli"
apt-get $_EE_APT_LOCK_INSTALL install -y "php${v}-cli"
EE_INSTALLED_PHP_VERSION="$v"
return 0
fi
if ee_php_add_thirdparty_repo && ee_php_cli_installable "$v"; then
ee_log_info2 "Installing PHP ${v} (third-party repo)"
apt-get install -y "php${v}-cli"
apt-get $_EE_APT_LOCK_INSTALL install -y "php${v}-cli"
EE_INSTALLED_PHP_VERSION="$v"
return 0
fi
Expand Down Expand Up @@ -327,12 +354,12 @@ function setup_php() {
ee_log_info2 "No PHP pinned; trying in order (version-first): ${candidates}"
fi

apt-get update
ee_apt_update
ee_select_and_install_php $candidates || true

# Unpinned fallback: distro default php-cli, so an unknown OS still gets a php.
if [ -z "$EE_INSTALLED_PHP_VERSION" ] && [ -z "$EE_PHP_VERSION" ]; then
if apt-get install -y php-cli && command -v php >/dev/null 2>&1; then
if apt-get $_EE_APT_LOCK_INSTALL install -y php-cli && command -v php >/dev/null 2>&1; then
EE_INSTALLED_PHP_VERSION="$(php -r 'echo PHP_MAJOR_VERSION.".".PHP_MINOR_VERSION;' 2>/dev/null)"
[ -n "$EE_INSTALLED_PHP_VERSION" ] && ee_log_info2 "Fell back to distro default php-cli (${EE_INSTALLED_PHP_VERSION})"
fi
Expand All @@ -359,7 +386,7 @@ function setup_php_extensions() {
if command -v php >/dev/null 2>&1; then
php_extensions=(curl sqlite3 zip)
if ! command -v gawk >/dev/null 2>&1; then
apt-get install gawk -y
apt-get $_EE_APT_LOCK_LIGHT_INSTALL install gawk -y
fi
# Use the version setup_php installed; else detect from the `php` symlink.
default_php_version="${EE_INSTALLED_PHP_VERSION:-}"
Expand All @@ -378,7 +405,7 @@ function setup_php_extensions() {
fi
done
if [ -n "$packages" ]; then
apt-get install -y $packages
apt-get $_EE_APT_LOCK_INSTALL install -y $packages
fi
fi
}
Expand Down Expand Up @@ -407,7 +434,7 @@ function setup_host_dependencies() {
if [ "$EE_LINUX_DISTRO" == "Ubuntu" ] || [ "$EE_LINUX_DISTRO" == "Debian" ]; then
if ! command -v ip >> $LOG_FILE 2>&1; then
ee_log_info2 "Installing iproute2"
apt-get install iproute2 -y
apt-get $_EE_APT_LOCK_LIGHT_INSTALL install iproute2 -y
fi
fi
}
Expand All @@ -431,11 +458,11 @@ function check_dependencies() {
# can find their packages. On a freshly booted machine the apt cache
# is typically empty and installs would fail without this.
ee_log_info2 "Updating apt cache"
apt-get update
ee_apt_update

if ! command -v sqlite3 >/dev/null 2>&1; then
ee_log_info2 "Installing sqlite3"
apt-get install sqlite3 -y
apt-get $_EE_APT_LOCK_LIGHT_INSTALL install sqlite3 -y
fi

setup_host_dependencies
Expand Down
26 changes: 25 additions & 1 deletion setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,30 @@ export LOG_FILE="$EE_ROOT_DIR/logs/install.log"
# Ensure EE_QUIET_OUTPUT is always defined so that set -u does not cause
# "unbound variable" errors when the sourced functions file checks it.
export EE_QUIET_OUTPUT="${EE_QUIET_OUTPUT:-}"
export EE_APT_LOCK_WAIT="${EE_APT_LOCK_WAIT:-}"

# When EE_APT_LOCK_WAIT is set, wait for the dpkg/apt lock instead of
# failing immediately. Tier variables are left empty (no-op) when unset.
Comment on lines +12 to +13

@mrrobot47 mrrobot47 Jun 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EE_APT_LOCK_WAIT is tested with [ -n … ], so it's presence-only: EE_APT_LOCK_WAIT=0 / false / no all enable waiting, and a numeric value like EE_APT_LOCK_WAIT=300 is silently ignored (the hardcoded 900/1800 win). The name suggests a tunable duration, which invites both mistakes. Either make the value the timeout, or document it as an on/off flag:

Suggested change
# When EE_APT_LOCK_WAIT is set, wait for the dpkg/apt lock instead of
# failing immediately. Tier variables are left empty (no-op) when unset.
# Any non-empty EE_APT_LOCK_WAIT (e.g. =1) enables the dpkg lock-wait; it is an
# on/off flag, not a duration (the timeouts below are fixed). Tier variables
# are left empty (no-op) when unset.

if [ -n "$EE_APT_LOCK_WAIT" ]; then
_EE_APT_LOCK_LIGHT_INSTALL="-o DPkg::Lock::Timeout=900"
_EE_APT_LOCK_INSTALL="-o DPkg::Lock::Timeout=1800"
else
_EE_APT_LOCK_LIGHT_INSTALL=""
_EE_APT_LOCK_INSTALL=""
fi

# Retry-based apt-get update that tolerates a busy apt lists lock.
# Defined here for bootstrap(); the functions file overrides with its own
# copy (identical, but uses ee_log_info2 for logging).
ee_apt_update() {
local i
for i in $(seq 1 30); do
apt-get update && return 0
echo "=====> apt lists lock busy; retrying in 10s"
sleep 10
done
apt-get update
}

# Run apt/dpkg non-interactively so package installation never blocks on a
# debconf or needrestart prompt during an unattended setup.
Expand Down Expand Up @@ -34,7 +58,7 @@ function bootstrap() {
if ! command -v wget > /dev/null 2>&1; then
packages="${packages} wget"
fi
apt-get update && apt-get install $packages -y
ee_apt_update && apt-get $_EE_APT_LOCK_LIGHT_INSTALL install $packages -y
fi

# Use the locally patched functions file when present (set via EE_LOCAL_FUNCTIONS),
Expand Down