Skip to main content

Arch Linux Rescue: Interrupted Upgrade Repair

Recovery guide for Arch Linux systems broken by an interrupted upgrade. Covers Btrfs subvolume mounting, pacman lock resolution, DNS configuration inside chroot, and fixing broken DBus/PAM/systemd library mismatches.

Symptoms​

  • Boot Failure: 
    error: fs/fshelp.c:find_file:260:file '/vmlinuz-linux' not found.
    error: loader/i386/linux.c:grub_cmd_initrd:1082:you need to load the kernel first.
  • Emergency Mode / Mount Failure:
    • Boot halts at emergency shell with message: Failed to mount /boot.
  • Systemd Service Cascade Failures:
    • Boot hangs or loops on: 
      [FAILED] Failed to start D-Bus System Message Bus.
      [DEPEND] Dependency failed for Network Manager.
      [FAILED] Failed to start User Login Management.
    • TTY and display managers (SDDM, GDM, etc.) reject login credentials due to Pluggable Authentication Modules (PAM) library mismatches.

Root Causes​

  1. Interrupted Pacman Transaction: System left with missing kernel files (vmlinuz-linux), missing initramfs, and stale database locks (db.lck).
  2. Partial Upgrade State: Mismatched versions between core libraries (glibc, pam, systemd, dbus-broker), causing systemd services and authentication modules to crash on startup.

Step-by-Step Resolution​

Phase 1: Boot Live Environment & Establish Network​

  1. Boot into the official Arch Linux live ISO.
  2. Connect to a wireless network (if using Wi-Fi): 
    iwctl
    # Inside iwctl shell:
    station wlan0 connect "SSID"
    exit
  3. Verify internet connectivity: 
    ping -c 3 1.1.1.1

Phase 2: Mount Btrfs Subvolumes & EFI Partition​

Standard mounts fail if the root filesystem resides inside a Btrfs subvolume. Locate and mount the correct subvolume layout.

  1. List block devices to identify partitions:

    lsblk

    Example layout: /dev/nvme0n1p1 (EFI, 1G), /dev/nvme0n1p2 (Btrfs root, 930G).

  2. Query the Btrfs partition to identify subvolumes:

    btrfs subvolume list /dev/nvme0n1p2
    # Look for subvolume directories starting with '@' (e.g., '@', '@home', '@pkg')

  3. Mount the active root subvolume (@) to /mnt:

    mount -o subvol=@ /dev/nvme0n1p2 /mnt

  4. Mount the EFI system partition:

    mount /dev/nvme0n1p1 /mnt/boot

    (Use /mnt/boot/efi if that is where the EFI partition is mapped in your fstab.)

Phase 3: Enter Chroot & Fix Networking​

  1. Chroot into the mounted installation:

    arch-chroot /mnt

  2. Configure DNS inside chroot to resolve mirrors:

    echo "nameserver 1.1.1.1" > /etc/resolv.conf

  3. Test name resolution:

    ping -c 3 archlinux.org

Phase 4: Repair Pacman Databases & Mirrors​

Interrupted transactions often lock the database and corrupt locally cached repository sync states.

  1. Remove the stale pacman lock file:

    rm -f /var/lib/pacman/db.lck

  2. Purge potentially corrupt database sync files:

    rm -rf /var/lib/pacman/sync/*

  3. Generate a fresh, localized mirrorlist via curl if existing mirror configurations fail to sync. Adjust country= parameters to match your region:

    curl -o /etc/pacman.d/mirrorlist "https://archlinux.org/mirrorlist/?country=ID&country=SG&protocol=https&use_mirror_status=on"

  4. Uncomment Server entries in the generated mirrorlist:

    sed -i 's/^#Server/Server/' /etc/pacman.d/mirrorlist

Phase 5: Reconstruct Core Packages & Resolve Library Mismatch​

Reinstalling critical packages overwrites truncated binaries and aligns mismatched libraries causing DBus/PAM to crash.

  1. Force database synchronization and install core libraries:

    pacman -Syyu dbus dbus-broker systemd systemd-libs glibc pam --overwrite "*"

  2. Run a system-wide upgrade to ensure all packages are aligned:

    pacman -Syu --overwrite "*"

  3. (Optional) If massive file corruption is suspected, force reinstall all explicitly installed native packages:

    pacman -S $(pacman -Qqen) --overwrite "*"

Phase 6: Rebuild Ramdisks & GRUB Config​

Ensure new kernel modules are packed and the bootloader configuration matches the updated system state.

  1. Regenerate initramfs for all installed kernels:

    mkinitcpio -P

  2. Regenerate the GRUB configuration file:

    grub-mkconfig -o /boot/grub/grub.cfg

    (If using systemd-boot instead, run bootctl install && bootctl update.)

Phase 7: Clean Exit & Reboot​

  1. Exit chroot:

    exit

  2. Recursively unmount all partitions:

    umount -R /mnt

  3. Reboot the system:

    reboot