curtin

zfs + encryption fails with plucky iso dated 20250415 - Requested offset is beyond real size of device /dev/zvol/rpool/keystore

Bug #2107381 reported by Tim Andersson on 2025-04-15

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
curtin	Fix Committed	Undecided	Olivier Gayot
subiquity	Fix Released	Undecided	Unassigned
Ubuntu	Fix Released	Critical	Olivier Gayot	Ubuntu ubuntu-25.04
Noble	Fix Released	Undecided	Unassigned
subiquity (Ubuntu)	Fix Released	Undecided	Unassigned
Noble	Fix Released	Undecided	Dan Bungert	Ubuntu ubuntu-24.04.3

Bug Description

Title, log attached, installer crashes simply with system problem detected etc

```
An error occured handling 'zpool-1': ProcessExecutionError - Unexpected error while running command.
        Command: ['cryptsetup', 'open', '--type', 'luks', '/dev/zvol/rpool/keystore', 'keystore-rpool', '--key-file', '/tmp/zpool-key-p4shdf2v']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: Requested offset is beyond real size of device /dev/zvol/rpool/keystore.

        finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: configuring zpool: zpool-1
        TIMED BLOCK_META: 22.242
        finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta
        Traceback (most recent call last):
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/commands/main.py", line 202, in main
            ret = args.func(args)
                  ^^^^^^^^^^^^^^^
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/log.py", line 97, in wrapper
            return log_time("TIMED %s: " % msg, func, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/log.py", line 79, in log_time
            return func(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/commands/block_meta.py", line 128, in block_meta
            return meta_custom(args)
                   ^^^^^^^^^^^^^^^^^
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/commands/block_meta.py", line 2284, in meta_custom
            handler(command, storage_config_dict, context)
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/commands/block_meta.py", line 2041, in zpool_handler
            zfs.zpool_create(poolname, vdevs_byid,
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/block/zfs.py", line 303, in zpool_create
            encryption.setup(storage_config, context)
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/block/zfs.py", line 114, in setup
            util.subp(cmd, capture=True)
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/util.py", line 323, in subp
            return _subp(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^
          File "/snap/ubuntu-desktop-bootstrap/361/lib/python3.12/site-packages/curtin/util.py", line 172, in _subp
            raise ProcessExecutionError(stdout=out, stderr=err,
        curtin.util.ProcessExecutionError: Unexpected error while running command.
        Command: ['cryptsetup', 'open', '--type', 'luks', '/dev/zvol/rpool/keystore', 'keystore-rpool', '--key-file', '/tmp/zpool-key-p4shdf2v']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: Requested offset is beyond real size of device /dev/zvol/rpool/keystore.

        Unexpected error while running command.
        Command: ['cryptsetup', 'open', '--type', 'luks', '/dev/zvol/rpool/keystore', 'keystore-rpool', '--key-file', '/tmp/zpool-key-p4shdf2v']
        Exit code: 1
        Reason: -
        Stdout: ''
        Stderr: Requested offset is beyond real size of device /dev/zvol/rpool/keystore.

Stderr: ''

```

See original description

Related branches

~dbungert/curtin:lp-2118578-noble-like-plucky

Merged into curtin:ubuntu/noble at revision 0ab394917d66d2b6c51e297bf9bf5d26ef36c360

Olivier Gayot: Approve on 2025-07-29

Server Team CI bot: Needs Fixing (continuous-integration) on 2025-07-29

Chris Peterson: Approve on 2025-07-28

curtin:lp-2118578-noble-cryptsetup-header-size

Merged into curtin:ubuntu/noble at revision e9127fb40fe8f7d36771ac7814188af2da42209c

Chris Peterson: Approve on 2025-07-25

Olivier Gayot: Approve on 2025-07-25

Server Team CI bot: Approve (continuous-integration) on 2025-07-24

~dbungert/curtin:lp-2107381-ctd

Merged into curtin:master at revision 894d53b639bdb917e0614c0a3727fe3a9c8a71b6

Chris Peterson: Approve on 2025-05-29

Server Team CI bot: Needs Fixing (continuous-integration) on 2025-05-27

~ogayot/curtin:larger-keystore

Merged into curtin:master at revision 148b02f5f4027ca508727110604e376f325a1078

Server Team CI bot: Needs Fixing (continuous-integration) on 2025-04-15

Dan Bungert: Approve on 2025-04-15

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

installer-logs.tar.gz Edit (84.7 KiB, application/x-tar)

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

(ignore the duplicated files in the tar, that was just a whoopsie, any duplicate should be identical)

description:	updated
summary:	- zfs + encryption fails with plucky iso dated 20250415 + zfs + encryption fails with plucky iso dated 20250415 - Requested + offset is beyond real size of device /dev/zvol/rpool/keystore

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

This failed for me twice on 2 separate VMs, and failed also on:
HP pro sff 400 g9
hp elite sff 805 g9

I can also confirm this passed (at least on a VM) as recently as 2025-04-03

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

ogayot has confirmed this has been failing as early as 04/01/2025

Tim Andersson (andersson123) on 2025-04-15

no longer affects:

ubuntu-desktop-provision

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

From cursory git archaeology it seems it may be as a result of this commit:
https://github.com/canonical/curtin/commit/204562c277b096229ca20ba546149deef1d85e38

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

plucky-beta-20250415-manifest.diff Edit (30.2 KiB, text/plain)

diff between the manifest from the beta release and today's ISO (61d088d5b4f9a361ee95dc5bae65d9245c64cfe2f455f50caea046066b949a58)

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

https://iso.qa.ubuntu.com/qatracker/milestones/466/builds/327173/testcases/1790/results

it was reported as functional in the beta.

Revision history for this message

Olivier Gayot (ogayot) wrote on 2025-04-15 (last edit on 2025-04-15):

$ sudo dd if=/dev/zero of=test.img count=1000 bs=1M
$ sudo zpool create testrpool "$PWD"/test.img
$ sudo zfs create -o encryption=off -V 20971520 testrpool/keystore
$ sudo cryptsetup luksFormat /dev/zvol/testrpool/keystore /tmp/key

On oracular, this produces no warnings. On plucky this leads to:

Device /dev/zvol/testrpool/keystore is too small for activation, there is no remaining space for data.

And then, when running cryptsetup open, it fails on plucky with:

Requested offset is beyond real size of device /dev/zvol/testrpool/keystore

Revision history for this message

Olivier Gayot (ogayot) wrote on 2025-04-15:

$ sudo cryptsetup luksDump /dev/zvol/testrpool/keystore

On oracular
-----------

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

On plucky
---------

Data segments:
0: crypt
offset: 33554432 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

This would explain why 20M (i.e., 20971520) is too low on plucky.

Olivier Gayot (ogayot) on 2025-04-15

Changed in curtin:
status:	New → In Progress
assignee:	nobody → Olivier Gayot (ogayot)

Tim Andersson (andersson123) on 2025-04-15

Changed in ubuntu:
milestone:	none → ubuntu-25.04
status:	New → In Progress
importance:	Undecided → Critical
assignee:	nobody → Olivier Gayot (ogayot)

Revision history for this message

Tim Andersson (andersson123) wrote on 2025-04-15:

#10

added ubuntu distribution and subscribed ubuntu release, targeted to 25.04 milestone.

Olivier Gayot (ogayot) on 2025-04-15

Changed in curtin:
status:	In Progress → Fix Committed
Changed in subiquity:
status:	New → In Progress

Revision history for this message

Olivier Gayot (ogayot) wrote on 2025-04-15:

#11

Pull request for main (includes other changes):

https://github.com/canonical/subiquity/pull/2195

Pull request for plucky (single commit):

https://github.com/canonical/subiquity/pull/2196

Changed in subiquity (Ubuntu):
status:	New → In Progress

Olivier Gayot (ogayot) on 2025-04-15

Changed in subiquity (Ubuntu):
status:	In Progress → Fix Committed

Olivier Gayot (ogayot) on 2025-04-16

Changed in subiquity:
status:	In Progress → Fix Committed
Changed in ubuntu:
status:	In Progress → Fix Committed

Revision history for this message

Utkarsh Gupta (utkarsh) wrote on 2025-04-16:

#12

This is done already - unsubscribing the release team.

Olivier Gayot (ogayot) on 2025-04-17

Changed in ubuntu:
status:	Fix Committed → Fix Released
Changed in subiquity (Ubuntu):
status:	Fix Committed → Fix Released

Revision history for this message

Olivier Gayot (ogayot) wrote on 2025-04-17:

#13

The fix is included in Subiquity 25.04 https://github.com/canonical/subiquity/releases/tag/25.04

Changed in subiquity:
status:	Fix Committed → Fix Released

Dan Bungert (dbungert) on 2025-07-24

Changed in subiquity (Ubuntu Noble):
milestone:	none → ubuntu-24.04.3
assignee:	nobody → Dan Bungert (dbungert)
status:	New → In Progress

Revision history for this message

Paride Legovini (paride) wrote on 2025-07-30:

#14

Hey, what is the status of this one in Noble?

Dan Bungert (dbungert) on 2025-07-30

Changed in subiquity (Ubuntu Noble):
status:	In Progress → Fix Committed

Revision history for this message

Dan Bungert (dbungert) wrote on 2025-07-30:

#15

> Hey, what is the status of this one in Noble?

Server is unaffected because the running kernel doesn't trigger this bug. Server still has the bugfix.
The fix was merged to noble for subiquity in https://github.com/canonical/subiquity/pull/2241 and further merged to noble for bootstrap in https://github.com/canonical/ubuntu-desktop-provision/pull/1136

Just now I confirmed that the version of ./curtin/block/zfs.py in bootstrap build 413, as found on 24.04/stable, indeed matches the expected copy of that file.

Revision history for this message

Olivier Gayot (ogayot) wrote on 2025-08-01:

#16

I tested on noble using the desktop installer with the latest ubuntu-desktop-bootstrap snap (rev 413) and latest daily ISO (20250801). The installation went successfully.

When installing on a VM, I did run into an issue on first boot and filed bug 2119293 (which I don't consider critical for .3 since it is not a regression).

Marking fix released since the regression with the zfs luks header size is addressed.