You sure want to die on this hill.

I think you underestimate the cost of shooting a real guy sitting at a table. Casting, lighting, filming, makeup, props, location, color correction all cost money. And I probably forgot about a few things. Film is ridiculously complex.

It’s good you found some pathological examples, but I’m at the end of my rope here.

You can use these examples and the other information you gathered so far and ask specifically how these size discrepancies can be explained and maybe mitigated. I suggest more specialized communities for this such as !linux@lemmy.ml, !linux@programming.dev, !linux@lemmy.world, !linux4noobs@programming.dev, !linux4noobs@lemmy.world, !linuxquestions@lemmy.zip.

These differences really are insane. Maybe someone more knowledgeable can comment on why different tools differ so wildly in the total size they report.

I have never used BTRFS, so I must resort to forwarding googled results like this one.

Could you try compsize ~? If the Perc column is much lower than 100% or the Disk Usage column is much lower than the Uncompressed column, then you have some BTRFS-specific file-size reduction on your hands, which your external exFAT naturally can’t replicate.

du --count-links only counts hard-linked files multiple types. I assumed you had a symlink loop that rsync would have tried to unwrap.

For instance:

$ ls -l
foo -> ./bar
bar -> ./foo

If you tried to rsync that, you’d end up with the directories foo, bar, foo/bar, bar/foo, foo/bar/foo, bar/foo/bar, foo/bar/foo/bar, ad infinitum, in the target directory.

Personally, I have no more tips that those that have already been presented in this comment section. What I would do now to find out what’s going on is the age-old divide-and-conquer debugging technique:

Using rsync or a file manager (yours is Dolphin), only copy a few top-level directories at a time to your external drive. Note the directories you are about to move before each transfer. After each transfer check if the sizes of the directories on your internal drive (roughly) match those on your external drive (They will probably differ a little bit). You can also use your file manager for that.

If all went fine for the first batch, proceed to the next until you find one where the sizes differ significantly. Then delete that offending batch from the external drive. Divide the offending batch into smaller batches (select fewer directories if you tried transferring multiple; or descend into a single directory and copy its subdirectories piecewise like you did before).

In the end you should have a single directory or file which you have identified as problematic. That can then be investigated further.

I’d say you can trust that.

I’m sorry. I was stupid. If you had duplicates due to a file system loop or symlinks, they would all be under different names. So you wouldn’t be able to find them with this method.

You checked 385GiB of files by hand? Is that size made up by a few humongously large files?

I suggest using uniq to check if you have duplicate files in there. (uniq’s input must be sorted first). If you still have the output file from the previous step, and it’s called rsync-output.txt, do sort rsync-output.txt | uniq -dc. This will print the duplicates and the number of their occurrences.

Idk if rsync traverses symlinks and filesystems by default,

From the man page:

Beginning with rsync 3.0.0, rsync always sends these implied directories as real directories in the file list, even if a path element is really a symlink on the sending side. This prevents some really unexpected behaviors when copying the full path of a file that you didn’t realize had a symlink in its path.

That means, if you’re transferring the file ~/foo/bar/file.txt, where ~/foo/bar/ is a symlink to ~/foo/baz, the baz directory will essentially be duplicated and end up as the real directory /SSD/foo/bar and /SSD/foo/baz.

Let’s back up and check your assumptions: How did you check that the disk usage of your home folder is 385GiB and that there are 780GiB of free disk space on your external drive?

You can store the output of rsync in a file by using rsync ALL_THE_OPTIONS_YOU_USED > rsync-output.txt. This creates a file called rsync-output.txt in your current directory which you can inspect later.

This, however means that you won’t see the output right away. You can also use rsync ALL_THE_OPTIONS_YOU_USED | tee rsync-output.txt, which will both create the file and display the output on your terminal while it is being produced.

What does that mean?

Imagine your hard drive like a giant cupboard of drawers. Each drawer can only have one label, so you must only ever store one “thing” in one drawer, otherwise you wouldn’t be able to label the thing accurately and end up not knowing what went where.

If you have giant drawers (a large block size), but only tiny things (small files) to store, you end up wasting a lot of space in the drawer. It could fit a desktop computer, but you’re only putting in a phone. This problem is called “internal fragmentation” and causes files to take up way more space than it would seem they need.

–––––

However, in your case, the target block size is actually smaller, so this is not the issue you’re facing.

Spaces are enough to handle spaces. That file path is valid.

Could it be you have lots of tiny files and/or a rather large-ish block size on your SSD?

You can check the block size with sudo blockdev --getbsz /dev/$THE_DEVICE.

For anyone else having trouble parsing the lower part: “Seeing a shooting star [is] itself a wish for many.”

If you want to see shooting stars, the next good opportunity is around April 22, where the Lyrids produce about one shooting star every three minutes (in good conditions).

If you’re patient but very easily distracted, wait for the Geminids around December 14 producing about 2 shooting stars per minute; or the Perseids around August 12 with slightly fewer.

All of the above assumes you’re in the Northern hemisphere.

https://en.wikipedia.org/wiki/List_of_meteor_showers

Personally, I would be fine if GIMP had just said, “too many operations, shrinking Undo history” or something like that. No amount of optimization gets rid of this issue. At some point, the machine’s resources are just exhausted.

I was waiting for this comment. No, I won’t troubleshoot magical hardware that only fails in GIMP.

Hmm, but that shouldn’t end up taking that much space either, no?

I can confirm that it indeed uses little RAM.

I was wrangling lots of big images and noticed that RAM usage was increasing steadily. Though that only seems to happen when generating new layers (e.g. by combinding them). Moving, for instance, does nothing. However I just noticed that doing things like adjusting the contrast (using the values histogram tool, sorry don’t know its English name) or adjusting the color temperature adds 300–400MBs. Do each operation three times on 9–12 images and you have a deadly amount of RAM usage.

Then look around in the comments to this post.