500 GB disk full welll before 500GB used

Help Desk
1
  • Distribution & Version: Ubuntu 20.4
  • Kernel information: 5.4.0-72

I just reinstalled my home server. I went from a 500GB disk to 500GB disk. I have almost nothing installed except docker with Jellyfin, and have copied all my movie files over. It then stopped and said disk full when doing my music.

df -h shows the below.

image
image1244×291 14.3 KB

\ is almost almost totally full. But this is a really confusing output. I don’t know why it split my 500GB between /dev/mapper/ubuntu--vg-ubuntu--lv which I take to be my ‘main’ storage and overlay that’s mapped to /var/lib/docker/overlay2/dbf94c0a9217d4b2b528bec5a14ee1cdf60e2340527f0eb4bbe66f3738717876/merged.

When installing, I told it to install the whole disk, and I didn’t install anything from the setup screen except ssh; I did the docker install myself after the server setup was finished, and I’m not sure what overlay is or how docker is using so much of it, and WHAT is being used in there.

1 Like
2

Is there a yaml file that directs your jellyfin configuration? If you don’t mind, may we see it? While I’ve never seen this before, I would start the diagnostic process in that configuration.

1 Like
3

No, I followed their documentation for install

  1. sudo apt install docker-ce
  2. sudo docker pull jellyfin/jellyfin:latest
  3. sudo docker run -d -v /srv/jellyfin/config:/config -v /srv/jellyfin/cache:/cache -v /media:/media --net=host jellyfin/jellyfin:latest

Later I made this always come back up with sudo docker update --restart=always b97a99c521ea9d91edab25ed59df430b7a4381ecc24d8a6b7ad51d00c3bf7365 that way it just runs if the machine ever goes off and back on. But that docker container there and the one from the screenshot aren’t the same characters.

1 Like
4

there is a file or folder in the Jellyfin container that is not being passed to your host as a volume, check if there are transcodes in another folder inside the container using docker exec -it jellyfincontainer /bin/bash to make a shell

5

hmmm. Have you made any progress on this? I’m sorry that I can’t tell you exactly how to fix this – but I suspect that there is log generation happening inside the container that is filling overlay2. Here is what I would try next:

  • stop the container
  • delete the container
  • delete the image
  • run docker prune
  • reinstall jellyfin/jellyfin without the lastest tag.
6

I’m not sure I understand what this is supposed to do. I’ve run this, but it drops me into a rot shell in / and that’s it. I can ls the jellyfin folder but it’s xml, dll, and json files is all. I don’t know what I’m looking for in there.

I know for a fact the docker image looks in /media for the media for that, and that’s where I have everything. I don’t even know what the overlay piece is though.

7

I will try this once I finish running through what rwaltr is suggesting. Then I can try yours or just blow the whole thing away, I haven’t done a ton on there, was just trying to save myself a few hours of redoing work!

8

just run du on everything, its bound to be there somewhere

9

I’m still not understanding this. It’s like overlay2 duplicated whatever was in /. The space it takes up is identical. I don’t know what would cause that. What generates overlay2? Did my docker command cause that to happen somehow?

10

overlay2 is the storagee driver docker uses by default.

Do you have a bunch of spare images or containerse docker images -a docker ps -a

What TZ are you in, If you happen to see that I am online in Discord you can give me a ping and we can do a screen share to take a look.

11

It won’t let me run your command, but docker images -a shows only `jellyfin/jellyfin latest 256eac9aa7e3 ’ with a storage size of 717MB.

But docker ps -a shows 7 dockers, all exited except just the one listed above (the other were before I set the same one to come back up). I don’t have Discord, unfortunately.

12

How big and how many log files will this command show?

sudo find /var/lib/docker/containers/ -type f -iname "*.log" | xargs ls -l

If they appear to be too large and you suspect this is the problem, you could purge those log files with:

sudo find /var/lib/docker/containers/ -type f -iname "*.log" -delete

This should probably be run after stopping docker and then restarting after the pruge.

13

delete those unused images, they are just taking space.

1 Like
14

with it running, it’s only a few hundred kb

15

docker image prune -a removed nothing.

16

hmmm. A few hundred k is nothing. FYI this command will tally all of the files for you:

sudo find /var/lib/docker/containers/ -type f -iname "*.log" | xargs sudo wc -c

But, seeing that that doesn’t seem to be the problem, may I ask to see a couple of docker reports from your machine? Specifically, please let me see:

docker info

docker ps -a

docker images

17

docker info
docker ps -a
docker images

18

Thanks for the info. Looks like you have multiple stopped containers from the same image. You should delete the unused jellyfin containers.

  • first list the containers and look for the container ID that have been “exited” with:

    • docker container ls -a

  • Then remove those exited containers by their IDs with:

    • docker container remove 866f17300389 ca5cf2f29310 e9f91ca103ab eb431f5b304b 703c2119ac3e 8a57a719fd95

  • You might be able to remove them in one fell swoop with:

    • docker container prune

You should then look to see if you have dangling volumes that are hanging around with:

  • docker volume ls

Let me know if this make any headway for you.

19

Thanks. I removed the others, but still shows 97% full in overlay2

20

Let’s run this command and see what files are taking up space. Perhaps that might give us a clue to what is going on.

sudo find / -size +5G 2>/dev/null | xargs -d '\n' ls -lh

What this does: (this is not necessarily for you but for others who might read this post)

  • sudo many of the files are not owned by the local user so we need root privileges to see them
  • find standard linux cli search command
  • / starts at the top of the filesystem
  • -size +5G looks for files that are greater than 5 Gigs in size
  • 2>/dev/null tosses the errors and keeps them off the display
  • | xargs passes the results into the subsequent command
  • -d '-n' sets the delimiter for xargs to new lines only otherwise file names with spaces create errors
  • ls -lh lists all of the files in a long format with easy human readable sizes
1 Like