| 00:07.36 | *** join/#uphpu bigdog_ut (n=bigdog_u@shylonhunter.dsl.xmission.com) |
| 00:07.36 | *** mode/#uphpu [+o bigdog_ut] by ChanServ |
| 00:32.11 | wps | we have a dilemma, the server's hard drive is full |
| 00:32.24 | wps | 90% of it is the list archives |
| 00:32.42 | wps | anyone have an opinion/suggestion of what to do? |
| 00:33.16 | wps | I have to do something quick because it's messing the server up (doesn't have enough free disk space to perform basic functions such as run MySQL and mail… not scratch space) |
| 00:33.27 | wps | should we just keep one year's worth of archives? |
| 00:35.12 | tierra | well, at the least, backup previous years, and we can put them up somewhere else |
| 00:35.35 | tierra | for now |
| 00:35.54 | wps | i never imagined that three years worth of messages would fill up 100 GBs |
| 00:36.23 | tierra | is it not gzipped? (not that I know if the archive scripts support it) |
| 00:36.47 | tierra | I can't imagine it taking up anywhere near that much... even uncompressed |
| 00:37.14 | wps | me either |
| 00:37.17 | tierra | unless there's all sorts of spam it archived and people's massive attachments (which I haven't seen come across the list |
| 00:37.50 | wps | but the 133 GB had drives (RAID'd) are 95% full and I have narrowed it down to mailman's archives |
| 00:38.27 | tierra | I've got the last year's worth of archived mail from 10 lists with more massively more traffic than UPHPU on my gmail account taking up less than 300MB (and gmail doesn't report compressed size I doubt) |
| 00:40.55 | tierra | -more |
| 00:40.58 | tierra | heh |
| 00:42.10 | tierra | something definately weird/wrong with UPHPU archives taking up 100GB |
| 00:43.04 | wps | when I do a "du" on the dir, it don't get anything back because it takes so long for it to calculate |
| 00:43.46 | wps | if I add up the contents of the dir manually, I get 21000 MB |
| 00:43.57 | wps | hmm… that's only 21 GB |
| 00:44.02 | wps | sheesh… I don't understand this |
| 00:44.05 | tierra | odd |
| 00:44.35 | wps | I don't understand what is taking up all the space |
| 00:45.39 | tierra | what package is in charge of the archives? is it some hack of an online script filling up archives? |
| 00:46.41 | wps | well, Mailman comes with the stuff to write out the HTML and all that and there is a cron that updates them nightly |
| 00:48.36 | wps | each month has three things: |
| 00:48.54 | tierra | hmm, I've had a mailman setup before, but it's been a while |
| 00:49.53 | wps | 1. a text file containing all of the emails from that month (some as large as 2 GB) |
| 00:51.34 | wps | 2. a gzipped version of the text file (approx 50K each) |
| 00:51.42 | tierra | the mailman mailing list is very valuable btw... clean off a few to keep it running for now, and investigate |
| 00:52.07 | wps | 3. a directory containing an HTML file for each message (several thousand files, some as large as 30 M each) |
| 00:52.11 | tierra | I've got to go actually, sorry I can't help much more |
| 00:52.23 | wps | no problem |
| 00:52.24 | wps | thanks |
| 03:21.16 | *** join/#uphpu jakea (n=jakea@128.187.221.84) |
| 03:51.10 | *** join/#uphpu synic (n=synic@c-67-177-16-249.hsd1.ut.comcast.net) |
| 04:34.02 | RyanE | @wps: you still around? |
| 04:34.08 | wps | yeah |
| 04:34.19 | RyanE | do you run X on the server that the disk is nearly full on? |
| 04:34.27 | wps | no |
| 04:34.29 | wps | it's headless |
| 04:35.11 | RyanE | you have an X server available on another machine though, right? i.e. you can run X apps on your mac, etc. |
| 04:35.20 | RyanE | so you can ssh -X into the server and run an X app? |
| 04:35.52 | wps | I don't currently have X11 installed on my Mac, but I could |
| 04:36.48 | RyanE | there's an app, called 'dirstat' which graphically shows each file and groups of files and is *very* helpful figuring out that sort of problem. |
| 04:37.00 | RyanE | you could use it to see if you can figure out why the archives are so large |
| 04:37.29 | RyanE | just thought I'd suggest it as I'm now reading IRC backlog a bit. |
| 04:37.46 | wps | thanks, I might have to try that |
| 04:37.53 | wps | well, now I am not sure if it is the logs |
| 04:37.57 | wps | I am positive it's in /var though |
| 04:38.29 | wps | I added up the size of all of the contents of the archives and it only came out to 22 GB |
| 04:38.38 | RyanE | if you run dirstat, you WILL know. |
| 04:38.54 | RyanE | it shows it graphically, and you can zoom down into directories and files. |
| 04:38.56 | RyanE | it rocks. |
| 04:39.04 | wps | interesting |
| 04:39.15 | RyanE | it takes a while to crunch through and get all the sizes, but when it finishes, it's clear what takes up the space. |
| 04:39.22 | wps | and you can run a GUI app locally, pulling the data from a remote machine? |
| 04:39.40 | wps | or do I have to be physically at the server? |
| 04:39.41 | RyanE | no, you run the gui app on the server. |
| 04:39.46 | RyanE | but the display can go to a remote machine |
| 04:39.51 | RyanE | X is a network protocol. |
| 04:40.05 | wps | oh, I thought you were talking about X windows |
| 04:40.12 | RyanE | yes, I am |
| 04:40.19 | RyanE | X Windows is a network protocol |
| 04:40.23 | RyanE | network-based |
| 04:40.28 | wps | guess I'm confused then |
| 04:40.40 | wps | I thought x-windows was the backbone of OS GUI's |
| 04:40.54 | RyanE | it is, for Linux / BSD |
| 04:41.02 | RyanE | but it can also be run across a network connection. |
| 04:41.06 | wps | the skeletong that things like GNOME were built on top of |
| 04:41.08 | RyanE | originally, that's how it would always work. |
| 04:41.11 | RyanE | yes |
| 04:41.15 | wps | hmm… interesting |
| 04:41.21 | RyanE | originally, you had graphical terminals, called X servers |
| 04:41.37 | RyanE | and the machine that actually ran the app, which was an X client, was the main servers. |
| 04:41.54 | RyanE | so the Server-Client terminology is a bit confusing |
| 04:42.05 | RyanE | but you can still remote them pretty easily over SSH. |
| 04:42.21 | *** topic/#uphpu by wps -> yes, the site and lists are down |
| 04:42.28 | RyanE | If you SSH into your headless box with SSH -X hostname, it sends the X display info to the machine you're SSHing from |
| 04:43.38 | RyanE | I just realized, it's actually 'kdirstat'. |
| 04:43.51 | RyanE | so it may not work for you, if QT/KDE has to be installed first. |
| 04:43.59 | RyanE | because you might not have the space to get int all installed. |
| 04:44.06 | RyanE | s/int/it/ |
| 04:44.40 | wps | I would have to install it. I do not have any GUIs installed |
| 04:44.56 | wps | man, my server is taking a beating |
| 04:44.57 | RyanE | well, we could do that if you want. |
| 04:45.02 | wps | MySQL can't run |
| 04:45.05 | RyanE | just from the list? |
| 04:45.08 | wps | IMAP can't run |
| 04:45.14 | wps | SMTP is stopped |
| 04:45.16 | RyanE | sure you weren't hacked? |
| 04:45.27 | wps | 'bout all it's doing is serving static pages |
| 04:45.29 | wps | no, not sure |
| 04:46.27 | wps | everything seems to be in order though… just doesn't have enough disk space to run |
| 04:48.00 | RyanE | well, if I can do anything to help, let me know. |
| 04:51.02 | wps | thanks, I really appreciate the offer |
| 04:51.13 | wps | I ran du and found out that it was /var |
| 04:51.24 | wps | so no I am running du on var to check it's contents' |
| 04:51.35 | wps | it is taking FOREVER for the results to come back thoughh |
| 04:53.03 | wps | would it be of any benefit to stop apache, sendmail, mysql, and mailman until I figure it out? |
| 04:53.51 | RyanE | does top show anything taking up a lot of cpu? |
| 04:54.01 | RyanE | if not, I don't think there'd be much advantage to stop them. |
| 04:54.15 | wps | k |
| 05:58.26 | wps | whoa, it doesn't have anything to do with the mail logs |
| 05:59.03 | RyanE | what distro is it running? |
| 05:59.13 | wps | FC3 |
| 05:59.20 | wps | 110GB in /var/total |
| 05:59.21 | RyanE | so what's weighing in? |
| 05:59.35 | RyanE | /var/total? whassat? |
| 05:59.39 | wps | took an hour and a half to return that |
| 05:59.42 | wps | ha ha ha |
| 05:59.45 | wps | was hoping you'd know |
| 05:59.48 | wps | I don't have a clue |
| 05:59.52 | wps | oh… wait |
| 05:59.54 | wps | I am stupid |
| 06:00.00 | wps | that's the total for the dir |
| 06:00.05 | RyanE | HAHAHAHA |
| 06:00.22 | RyanE | dang that /var/total. :) |
| 06:01.30 | RyanE | so what's big in /var, other than the total? |
| 06:01.40 | wps | 12K account |
| 06:01.42 | wps | 402M cache |
| 06:01.42 | wps | 16K crash |
| 06:01.42 | wps | 28K db |
| 06:01.42 | wps | 16K empty |
| 06:01.43 | wps | 109G lib |
| 06:01.45 | wps | 8.0K local |
| 06:01.47 | wps | 48K lock |
| 06:01.50 | wps | 637M log |
| 06:01.53 | wps | 4.0K mail |
| 06:01.55 | wps | 232K named |
| 06:01.57 | wps | 8.0K nis |
| 06:02.00 | wps | 8.0K opt |
| 06:02.02 | wps | 8.0K preserve |
| 06:02.06 | wps | 4.0K racoon |
| 06:02.07 | wps | 208K run |
| 06:02.10 | wps | 38M spool |
| 06:02.13 | wps | 25M state |
| 06:02.15 | wps | 8.0K tmp |
| 06:02.17 | wps | 8.0K tux |
| 06:02.20 | wps | 11M www |
| 06:02.22 | wps | 24K yp |
| 06:02.26 | wps | 110G total |
| 06:02.27 | wps | so, it probably is the mailman archives |
| 06:02.39 | RyanE | /var/lib |
| 06:02.43 | RyanE | is that where the mailman archives are? |
| 06:02.51 | RyanE | have you done du in /var/lib? |
| 06:03.36 | wps | yes, the archives are in /var/lib/mailman/archives/private/uphpu |
| 06:03.49 | wps | doing du in /var/lib now |
| 11:22.10 | *** join/#uphpu synic (n=synic@pdpc/supporter/student/synic) |
| 14:21.59 | *** join/#uphpu RyanE_ (n=ryan@rberick.dsl.xmission.com) |
| 18:22.02 | *** join/#uphpu osynic (n=synic@pdpc/supporter/student/synic) |