| 01:00.44 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |
| 01:50.40 | *** join/#openmoko-cdevel pabs3 (~pabs3@unaffiliated/pabs3) |
| 02:19.59 | pabs3 | DocScrutinizer05: looks like massive load from a lot of web processes |
| 02:20.22 | pabs3 | load average: 112.52, 109.11, 111.62 |
| 02:21.04 | DocScrutinizer05 | eeek |
| 02:21.11 | DocScrutinizer05 | io? |
| 02:21.21 | DocScrutinizer05 | or real cPU |
| 02:21.27 | DocScrutinizer05 | CPU* |
| 02:21.42 | DocScrutinizer05 | maybe check SMART |
| 02:22.18 | pabs3 | aha, also there is a RAID recheck going on |
| 02:22.21 | DocScrutinizer05 | or stop apache temporarily? |
| 02:22.37 | DocScrutinizer05 | aaah that explains it, almost suspected that |
| 02:22.44 | pabs3 | apache stopped already |
| 02:23.36 | DocScrutinizer05 | I would also suspect one of the RAID drives has a nasty SMART report |
| 02:25.15 | DocScrutinizer05 | though last (and only) time I had trouble with my RAID1, in the end a 50ct worth power Y-cable was the culprit |
| 02:26.01 | pabs3 | SMART seems fine |
| 02:26.03 | DocScrutinizer05 | it shown as sdb having more power cycles in SAMRT than sda |
| 02:26.32 | DocScrutinizer05 | nothing in syslog during last 3 days? |
| 02:27.14 | DocScrutinizer05 | maybe the controller |
| 02:28.04 | DocScrutinizer05 | actually, what's a recheck? |
| 02:29.23 | pabs3 | scans the raid1 to make sure both disks agree |
| 02:29.46 | pabs3 | <PROTECTED> |
| 02:29.52 | pabs3 | <PROTECTED> |
| 02:33.43 | DocScrutinizer05 | so a frankenstein fsck |
| 02:34.28 | DocScrutinizer05 | is that stuff scheduled, or maybe after N boots like fsck in fstab? |
| 02:34.42 | DocScrutinizer05 | I mean, what triggered it? |
| 02:34.52 | DocScrutinizer05 | the reboot? |
| 02:36.19 | DocScrutinizer05 | that was like >49h ago already, and the recheck is at 6.7% |
| 02:36.27 | DocScrutinizer05 | 40* |
| 02:39.37 | pabs3 | there is a cron job in mdadm and also a bunch of RAID1 devices |
| 02:40.05 | pabs3 | one more RAID1 to check after the current one too |
| 02:40.29 | pabs3 | it is possible the reboot triggered a check somehow too |
| 02:41.23 | DocScrutinizer05 | aiui those checks should run in background and not slow down normal functional IO, no? |
| 02:42.11 | DocScrutinizer05 | IOW the job runs only while system-IO idle |
| 02:42.31 | DocScrutinizer05 | and gets suspended for all normal IO activity |
| 02:43.27 | DocScrutinizer05 | which would explain why it's at 7% after 40h, *if* there was lots of normal system load like actually somebody hogging HTTP |
| 02:43.27 | pabs3 | yeah that should be how it works, not sure tho |
| 02:44.03 | pabs3 | starting apache again, lets see what happens |
| 02:44.09 | DocScrutinizer05 | :-) |
| 02:44.12 | pabs3 | <PROTECTED> |
| 02:44.25 | DocScrutinizer05 | maybe check access logs, if there are any |
| 02:47.16 | pabs3 | bing is walking the svn web interface... |
| 02:47.57 | pabs3 | load is still okish |
| 02:48.05 | pabs3 | <PROTECTED> |
| 02:48.18 | pabs3 | I'll keep an eye on it |
| 02:59.51 | DocScrutinizer05 | let me check mail RTT |
| 03:01.22 | DocScrutinizer05 | wow, speed of light |
| 03:02.04 | DocScrutinizer05 | swap hell? |
| 03:02.18 | DocScrutinizer05 | not now, back when it was slooow |
| 03:02.38 | pabs3 | probably |
| 03:03.00 | DocScrutinizer05 | swap on webservers almost never makes sense in my experience |
| 03:04.09 | DocScrutinizer05 | most of the webserver programs have some method of limiting the max memory they allocate |
| 03:04.34 | DocScrutinizer05 | should be real RAM - N |
| 03:16.55 | DocScrutinizer05 | on maemo the real RAM hogs were the scripting plugins, particularly php. They exploded when somebody mirrored the whole dynamically generated content |
| 03:17.59 | DocScrutinizer05 | a 500MB per process were not unusual, and there were too many parallel processes for that php stuff |
| 03:37.55 | DocScrutinizer05 | hmmm, my ssh login fails |
| 03:38.33 | DocScrutinizer05 | thought that worked once |
| 03:47.13 | DocScrutinizer05 | aah never mind, wrong user |
| 04:04.03 | *** join/#openmoko-cdevel dos1 (~dos1@neo900/coreteam/dos) |
| 04:41.38 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |
| 05:25.58 | *** join/#openmoko-cdevel mickeyl (~mickeyl@b2b-130-180-126-235.unitymedia.biz) |
| 05:26.02 | *** join/#openmoko-cdevel mickeyl (~mickeyl@openmoko/coreteam/mickey) |
| 06:51.40 | *** join/#openmoko-cdevel ao2 (~ao2@host58-137-dynamic.56-79-r.retail.telecomitalia.it) |
| 07:16.00 | *** join/#openmoko-cdevel pabs3 (~pabs3@unaffiliated/pabs3) |
| 07:48.49 | pabs3 | https://vizzzion.org/blog/2017/09/the-evolution-of-plasma-mobile/ |
| 09:40.57 | *** join/#openmoko-cdevel patinux (~patinux@unaffiliated/patinux) |
| 10:00.28 | *** join/#openmoko-cdevel patinux (~patinux@unaffiliated/patinux) |
| 11:57.51 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |
| 12:04.34 | *** join/#openmoko-cdevel jonwil (~jonwil@27-33-80-219.tpgi.com.au) |
| 13:09.07 | *** join/#openmoko-cdevel patinux (~patinux@unaffiliated/patinux) |
| 14:01.16 | *** join/#openmoko-cdevel patinux (~patinux@unaffiliated/patinux) |
| 14:08.34 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |
| 15:18.54 | *** join/#openmoko-cdevel pabs3 (~pabs3@unaffiliated/pabs3) |
| 16:04.16 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |
| 18:23.47 | *** join/#openmoko-cdevel sicelo (~sicelo@Maemo/community/council/sicelo) |
| 21:07.07 | *** join/#openmoko-cdevel patinux (~patinux@unaffiliated/patinux) |
| 21:15.09 | *** join/#openmoko-cdevel __Chris (~Chris@p4FCA12AF.dip0.t-ipconnect.de) |
| 21:22.15 | *** join/#openmoko-cdevel daniele_athome (~daniele_a@93-40-14-81.ip36.fastwebnet.it) |