| 00:00.24 | *** join/#uphpu macnewbold (~Adium@c-67-166-115-242.hsd1.ut.comcast.net) |
| 00:00.25 | *** mode/#UPHPU [+o macnewbold] by ChanServ |
| 00:02.37 | SunSparc | laterz |
| 00:02.43 | mindjuju | cya |
| 00:14.24 | *** part/#uphpu mgeary (~mgeary@166.70.44.49) |
| 01:21.17 | *** join/#uphpu ninnypants (~ninnypant@69.169.155.194.provo.static.broadweavenetworks.net) |
| 01:38.25 | *** join/#uphpu redroy (~Adium@c-174-52-210-81.hsd1.ut.comcast.net) |
| 01:39.16 | *** join/#uphpu wps (~wps@97-126-239-240.slkc.qwest.net) |
| 01:39.17 | *** mode/#UPHPU [+o wps] by ChanServ |
| 02:03.14 | *** join/#uphpu stderr- (~stderr@truedat.org) |
| 02:03.56 | *** join/#uphpu unum (~unum@li32-23.members.linode.com) |
| 02:05.43 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-008.slc1.dsl.speakeasy.net) |
| 03:06.15 | *** join/#uphpu wps (~wps@97-126-239-240.slkc.qwest.net) |
| 03:06.15 | *** mode/#UPHPU [+o wps] by ChanServ |
| 04:02.54 | carmony | hrm |
| 04:03.03 | carmony | I'm thinking about re-doing my WordPress blog theme. |
| 04:03.31 | wps | may the schwartz be with you |
| 04:03.39 | carmony | lol |
| 04:03.41 | carmony | :) |
| 04:14.10 | herlo | carmony: nice blog post |
| 04:15.07 | carmony | herlo: heh, I'm surprised I could remember about split at 3 AM |
| 04:31.05 | synic | it's surprising what the motivation for "get the crap out of here" can accomplish |
| 04:40.00 | *** join/#uphpu jtsnow (~jtsnow@VDSL-151-118-130-166.DNVR.QWEST.NET) |
| 04:40.03 | herlo | carmony: honestly have never heard of the split command before |
| 04:40.16 | carmony | oh really? |
| 04:40.21 | carmony | it was kinda convinient |
| 04:40.22 | carmony | lol |
| 04:40.42 | herlo | I would have just done a dd of thewhole drive and been done with it, though it may not have worked |
| 04:41.21 | herlo | not sure I would have had the patience at 3am to figure out split |
| 04:51.39 | *** join/#uphpu mgeary (~mgeary@69.169.142.45.provo.static.broadweavenetworks.net) |
| 04:51.39 | *** mode/#UPHPU [+v mgeary] by ChanServ |
| 05:38.00 | *** part/#uphpu mgeary (~mgeary@69.169.142.45.provo.static.broadweavenetworks.net) |
| 05:41.30 | *** join/#uphpu jkr801 (~jkr801@c-67-186-232-131.hsd1.ut.comcast.net) |
| 13:07.16 | *** join/#uphpu jnbek (~jnbek@pdpc/supporter/active/jnbek) |
| 13:49.19 | *** join/#uphpu jtsnow (~jtsnow@VDSL-151-118-130-166.DNVR.QWEST.NET) |
| 13:52.57 | *** join/#uphpu itimber (~Adium@74.51.216.34) |
| 13:52.57 | *** mode/#UPHPU [+v itimber] by ChanServ |
| 13:54.51 | *** join/#uphpu mastergunns (~mastergun@216.49.181.128) |
| 14:27.09 | *** join/#uphpu bhosie (~bhosie@c-98-202-142-106.hsd1.ut.comcast.net) |
| 14:32.01 | *** join/#uphpu ninnypants (~ninnypant@69.169.155.194.provo.static.broadweavenetworks.net) |
| 14:45.14 | *** join/#uphpu redroy (~Adium@c-174-52-210-81.hsd1.ut.comcast.net) |
| 14:52.12 | *** join/#uphpu mgeary (~mgeary@166.70.44.49) |
| 14:52.12 | *** mode/#UPHPU [+v mgeary] by ChanServ |
| 15:04.37 | *** join/#uphpu jtsnow (~jtsnow@VDSL-151-118-130-166.DNVR.QWEST.NET) |
| 15:07.43 | *** join/#uphpu Utah_Dave (~boucha@72-166-51-162.dia.static.qwest.net) |
| 15:07.43 | *** mode/#UPHPU [+o Utah_Dave] by ChanServ |
| 15:10.59 | *** join/#uphpu fungus (~olsonl@bromine.sosstaffing.com) |
| 15:10.59 | *** mode/#UPHPU [+o fungus] by ChanServ |
| 15:30.38 | mindjuju | hey fungus |
| 15:30.54 | fungus | sugnuf yeh |
| 15:31.55 | *** join/#uphpu wps (~wps@208.53.57.58) |
| 15:31.55 | *** mode/#UPHPU [+o wps] by ChanServ |
| 15:38.07 | *** join/#uphpu ninnypants (~ninnypant@64.72.210.74) |
| 15:38.14 | herlo | fungus: sugnuf reminds me of snuggleupagus |
| 15:38.30 | herlo | even though I know it's your name backward |
| 15:38.57 | fungus | snuffleupagus |
| 15:39.20 | *** join/#uphpu josephscott (~josephsco@74.200.246.169) |
| 15:39.20 | *** mode/#UPHPU [+v josephscott] by ChanServ |
| 15:39.22 | herlo | or that |
| 15:39.23 | herlo | :) |
| 15:39.27 | herlo | that's what I meant |
| 15:39.54 | herlo | wow, can't believe I said that, probably the snug part threw me in your backward name |
| 15:40.01 | *** join/#uphpu jtsnow (~jtsnow@VDSL-151-118-130-166.DNVR.QWEST.NET) |
| 15:41.12 | *** join/#uphpu cro (~cro@204-228-149-217.ip.xmission.com) |
| 15:42.24 | influx | Mornin'. |
| 15:43.20 | mindjuju | sup influx |
| 15:47.16 | *** join/#uphpu cro (~cro@204-228-149-217.ip.xmission.com) |
| 15:48.23 | *** join/#uphpu cro (~cro@204-228-149-217.ip.xmission.com) |
| 15:49.30 | josephscott | $lart days that end in y |
| 15:49.30 | raifbot | farts in days that end in y's general direction. |
| 15:49.57 | mindjuju | josephscott, you're really struggling over there aren't you? |
| 15:50.24 | josephscott | painful week with plenty more pain coming later |
| 15:50.36 | influx | mindjuju: Nm. Et tu? |
| 15:52.17 | mindjuju | ~give josephscott a team of engys to help him |
| 15:52.17 | ibot | ACTION gives a team of engys to help him to josephscott |
| 15:54.11 | ninnypants | mornin |
| 15:55.57 | influx | mornin ninnypants |
| 15:56.18 | josephscott | howdy |
| 15:57.09 | ninnypants | influx: how's your orders dash board coming? |
| 16:02.45 | *** join/#uphpu SunSparc (~jonathan@24-104-129.146.hfc.mediarain.com) |
| 16:02.46 | *** mode/#UPHPU [+v SunSparc] by ChanServ |
| 16:03.39 | *** join/#uphpu carmony (~justin@c-24-10-194-172.hsd1.ut.comcast.net) |
| 16:03.39 | *** mode/#UPHPU [+v carmony] by ChanServ |
| 16:06.00 | *** join/#uphpu TuxToaster (~TuxToaste@173-10-25-150-BusName-utah.ut.hfc.comcastbusiness.net) |
| 16:20.08 | influx | ninnypants: The project I've been working on? |
| 16:20.18 | influx | I mean, the one I've showed you. |
| 16:22.06 | influx | Right now I'm revising the MySQL query (after some help from DtD yesterday) to include some more information. |
| 16:22.27 | influx | Going to have to create another table for some data that gets tied into the current one. |
| 16:36.54 | ninnypants | influx: nice |
| 16:37.07 | ninnypants | I'm sure it's more fun than what I'm doing right now |
| 16:38.36 | influx | What are you working on? |
| 16:40.56 | herlo | Garbage collector? |
| 16:41.16 | herlo | like for a programming language? |
| 16:41.20 | herlo | :P |
| 16:42.54 | ninnypants | nope |
| 16:43.09 | ninnypants | have to work in an actual office that isn't my basement |
| 16:43.17 | ninnypants | just moving some servers |
| 16:43.30 | ninnypants | the not my basement part is the bad part |
| 16:43.58 | josephscott | $chuck-fact |
| 16:43.59 | raifbot | Chuck Norris can fit a square peg in a round hole |
| 16:45.41 | ninnypants | $chuck-fact |
| 16:45.41 | raifbot | Someone once tried to tell Chuck Norris that roundhouse kicks aren't the best way to kick someone. This has been recorded by historians as the worst mistake anyone has ever made. |
| 16:52.19 | josephscott | $weather 84092 |
| 16:52.19 | raifbot | josephscott - current weather for 84092 - Cloudy 53f / 12c Humidity: 45% Wind: S at 15 mph |
| 16:54.00 | Utah_Dave | $weather 84043 |
| 16:54.00 | raifbot | Utah_Dave - current weather for 84043 - Cloudy 53f / 12c Humidity: 48% Wind: S at 19 mph |
| 16:55.02 | ninnypants | there should be an app that auto blocks all recruiters numbers |
| 17:00.14 | mgeary | i just tell them what i'd have to make to be interested in a different job |
| 17:15.40 | influx | So, a domain that I had wanted in 2008 or so recently expired. |
| 17:16.05 | influx | I didn't know of the registrar would follow through on registering it for me automatically. |
| 17:16.18 | influx | But it did! ^_^ |
| 17:20.28 | *** join/#uphpu jtsnow (~jtsnow@74-95-52-198-Utah.hfc.comcastbusiness.net) |
| 17:39.10 | mindjuju | some days, i'm frustrated at the police for doing dumb things; other days, i get frustrated with the parents - http://news.yahoo.com/s/ap/us_child_pepper_sprayed |
| 17:39.30 | mindjuju | the last place i'd take this child is to be on TV, parents + child need a headshrinker |
| 17:43.00 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-008.slc1.dsl.speakeasy.net) |
| 17:43.59 | *** join/#uphpu redroy (~Adium@67.137.120.26) |
| 18:23.01 | mindjuju | herlo, you around? |
| 18:27.59 | mindjuju | fungus, you around? |
| 18:29.03 | herlo | mindjuju: yes |
| 18:29.30 | mindjuju | hi, i've been googling to find out similar situation, but seems like there are so many sources... |
| 18:29.59 | mindjuju | i have a problem where a server is operating just fine, and then it just crashes, the last thing we've added to the server was mod_security |
| 18:30.23 | mindjuju | the symptom is that everything is going boringly perfect, then sudden high swap usage, and then dead |
| 18:30.40 | mindjuju | now, i've got these servers at liquidweb, and they had a monitoring script running |
| 18:30.54 | mindjuju | and the server crashed so fast, their script didn't catch it |
| 18:31.49 | mindjuju | wouldn't that cause a huge spike in swap ? |
| 18:36.41 | herlo | hmm |
| 18:36.50 | herlo | mindjuju: do you have munin monitoring this box? |
| 18:37.00 | herlo | well, I wonder if that will help |
| 18:37.01 | mindjuju | not on this one |
| 18:37.11 | mindjuju | the script liquidweb has running is simlar though |
| 18:37.14 | herlo | you could try vmstat and iostat |
| 18:37.49 | herlo | mindjuju: can you reliably crash it again? |
| 18:38.04 | mindjuju | nope, we have no idea what the catalyst is |
| 18:38.23 | mindjuju | it seems to be on a 2 - 3 cycle though |
| 18:38.38 | mindjuju | crash <-- 2 - 3 days --> crash |
| 18:38.44 | herlo | so, liquidweb is running on another right? |
| 18:39.00 | herlo | another server |
| 18:39.02 | mindjuju | not sure i follow, but we've got all our assets there |
| 18:39.22 | herlo | oh, liquidweb isn't a monitoring service, sorry |
| 18:39.27 | herlo | I meant their monitoring script |
| 18:39.27 | mindjuju | oh, liquidweb is the host provider for the dedicated servers, they are running a monitors script on that server |
| 18:39.35 | mindjuju | yup, monitoring script is on that server |
| 18:39.41 | herlo | does it log? |
| 18:39.50 | mindjuju | it does, but it didn't catch the event |
| 18:40.00 | herlo | so, it logs locally? |
| 18:40.14 | mindjuju | i'll have to find that out, i'm not sure |
| 18:40.25 | herlo | if it doesn't log locally, it will probably be hard to catch the event |
| 18:40.35 | herlo | I suspect they run their monitor script every few minutes |
| 18:40.38 | mindjuju | k, i'll mention that |
| 18:40.47 | mindjuju | i believe so |
| 18:41.02 | herlo | you need something to monitor it a lot more often than that |
| 18:41.17 | herlo | I bet though, that there's some sort of indicator in a graph tha twould help |
| 18:41.30 | herlo | if something like munin were installed, you could look at the graphs and see the build up |
| 18:42.03 | herlo | it won't indicate *what* is causing the problem, but might give you an indication as to the subsystem which the issues is occuring |
| 18:42.18 | mindjuju | so quick question then, if munin showed a huge amount of swapping, clear out of the blue, where would you start poking around first? |
| 18:42.26 | mindjuju | swap partition/? |
| 18:42.27 | herlo | is that what happened? |
| 18:42.42 | mindjuju | yup, huge swap spike is what they see, then crash |
| 18:43.07 | herlo | well, I would say it's probably a memory leak |
| 18:43.22 | herlo | if you swap on a web or db server, you are basically dead in the water |
| 18:43.37 | mindjuju | that's what i was thinking, something causing it to fill up so fast, it started swapping |
| 18:43.48 | herlo | munin will tell you when this happens |
| 18:44.01 | herlo | my guess is though, you already have graphs that show this |
| 18:44.13 | herlo | other graphs might indicate if cpu load is related |
| 18:44.33 | herlo | odds are, it's not. |
| 18:44.41 | herlo | mindjuju: what is the main purpose of this system? |
| 18:45.04 | Utah_Dave | herlo: it's a random techno generator |
| 18:45.22 | mindjuju | so i think we need to do 2 things, get munin on this server, and put apache on crash prevention, with like 2000 maxrequestsperchild and reduce serverlimit/maxclients to 175 |
| 18:45.26 | mindjuju | see if that fixes |
| 18:45.26 | herlo | Utah_Dave: nice! thump, thump, thump, thump.... |
| 18:45.42 | herlo | mindjuju: how much ram is on that box? |
| 18:46.02 | mindjuju | the server does multiinstall wordpress, and a couple other functions |
| 18:46.05 | herlo | if it's less than 8G, I'd reduce it down to 125 maxclients/serverlimit |
| 18:46.14 | herlo | mindjuju: are you using apc or eaccelerator? |
| 18:47.04 | mindjuju | no apc, wp supercache |
| 18:47.07 | herlo | if not, I would suggest looking into that, plus memcached (or wp supercache) per josephscott's recommendations there |
| 18:48.02 | herlo | apc/eaccelerator are opcode optimizers and give benefit above and beyond memcached/wp supercache so it might help extend your load some. But at some point, you are running out of ram |
| 18:48.06 | herlo | at least that's what it looks like |
| 18:48.25 | herlo | mindjuju: can you get to the physical terminals on the box? |
| 18:49.08 | josephscott | php in production should always be using APC or one of the other opcode caches |
| 18:50.16 | herlo | josephscott: do you prefer APC over eaccelerator? If so, why? |
| 18:50.30 | josephscott | I generally use APC |
| 18:50.50 | mindjuju | nope, no physical access |
| 18:50.51 | josephscott | it is written/maintained by core PHP devs |
| 18:50.56 | herlo | mindjuju: not even an ILO? |
| 18:51.11 | josephscott | and is possible candidate for inclusion in PHP6 (someday) |
| 18:51.20 | mindjuju | nope, just ssh |
| 18:52.04 | mindjuju | what does ILO offer us that ssh doesn't? are are you wanting to see what happens when computer is starting up? |
| 18:52.05 | herlo | mindjuju: so your hosting provider, liquidweb. Ask them if you can have access to the terminals via ILO or something. You are going to need it at some point if not now |
| 18:52.30 | herlo | mindjuju: nope, I want ot see what happens on the ttys (physical terminals) |
| 18:52.51 | herlo | it's different from ssh terminals pty (called pseduoterminals/pseduo ttys) |
| 18:53.16 | herlo | and when a core kernel component dumps, it will dump it there, rather than to your ssh terminal |
| 18:53.28 | mindjuju | oh |
| 18:53.41 | herlo | you might have access via some sort of serial emulator, since that's how ILO/ILOm/DRAC do it |
| 18:55.54 | mindjuju | i'll find out about ilo |
| 19:16.50 | cycokilr | if you have root access just edit your kernel boot options to redirect messages to the active console. seems like a shot in the dark to get remote access through a hosting provider. |
| 19:20.37 | *** join/#uphpu beandog (~sdibb@gentoo/developer/beandog) |
| 19:20.37 | *** mode/#UPHPU [+v beandog] by ChanServ |
| 19:21.39 | herlo | cycokilr: the only problem with that is knowing in advance what the active console's tty will be |
| 19:22.32 | cycokilr | redirect it to all of them.. or turn off all but one in your inittab.. you don't really need more than one if you only have ssh access to the box anyway. |
| 19:22.45 | cycokilr | i don't see how this is a problem |
| 19:22.55 | cycokilr | unless you don't have root access |
| 19:25.09 | *** join/#uphpu itimber1 (~Adium@74.51.216.34) |
| 19:25.45 | cycokilr | also, there's a pretty good chance that if you're the first one to ssh into the system, you'll end up with pts/1 so sending it there is a pretty safe bet if you don't want it going to all consoles.. |
| 19:25.57 | herlo | cycokilr: but he doesn't even have access to one of the physical consolse |
| 19:26.13 | herlo | but pts/1 might work |
| 19:26.49 | herlo | physical consoles are what is controlled in inittab, though I wonder if you could use a pty. |
| 19:29.03 | cycokilr | inittab most certainly does not only handle physical consoles, but this is beyond the scope of this conversation.. |
| 19:30.05 | herlo | I don't think it is |
| 19:30.37 | herlo | why not just turn on a couple ptys instead of ttys? |
| 19:30.52 | herlo | seems like it would make things possibly easier |
| 19:31.09 | cycokilr | turn on? |
| 19:31.14 | cycokilr | they're automatic |
| 19:31.31 | cycokilr | ssh into a server a few times and type 'who'.. you'll see what i mean. |
| 19:32.21 | cycokilr | so add 'console=/dev/pts/1' to your kernel boot options and reboot the server. then ssh back in and hope you land on pts/1 (which you should if you're first) |
| 19:32.39 | herlo | cycokilr: I get that |
| 19:32.45 | herlo | it's not what I was implying |
| 19:32.55 | cycokilr | it will be super annoying to try to actually work this way because of all the messages.. but it will work. |
| 19:33.12 | herlo | well, you could set it to something else and use minicom maybe |
| 19:33.32 | cycokilr | as an alternative, ssh in, run 'screen' then 'tail -f /var/log/messages' in one of the screen windows.. |
| 19:34.01 | cycokilr | messages should contain enough detail for you. |
| 19:34.22 | cycokilr | no reason to make things more complicated just because it's possible.. keep it simple. |
| 19:38.25 | herlo | cycokilr: oomkiller messages do *not* get dumped to /var/log/message |
| 19:38.48 | herlo | usually it's because the system is already running out of memory and doesn't have any other place to write but the console |
| 19:40.26 | cycokilr | that makes no sense.. |
| 19:40.55 | herlo | how does it not make sense? |
| 19:41.00 | herlo | that *is* what happens |
| 19:41.05 | cycokilr | it's job is to keep the system from running out of memory by killing off stuff. it couldn't run at all if the system didn't have enough memory to write to syslog. |
| 19:41.12 | cycokilr | its* |
| 19:41.25 | herlo | except when it's overloaded with more and more things to kill |
| 19:41.36 | cycokilr | then it wouldn't be able to do its job |
| 19:41.38 | herlo | and memory stays at 100% |
| 19:41.47 | herlo | or near |
| 19:41.57 | cycokilr | i don't believe it. |
| 19:41.58 | herlo | well, it does exactly that. I've experienced it first hand many times |
| 19:42.05 | cycokilr | check the syslog.conf.. it's probably just not in there. |
| 19:42.15 | herlo | dude, it's not there |
| 19:42.21 | cycokilr | there are tons of posts from people with oomkiller messages in your logs |
| 19:42.22 | herlo | syslog can't log it, I promise. |
| 19:42.36 | cycokilr | they must be lying |
| 19:43.00 | cycokilr | http://www.howtoforge.com/forums/showthread.php?t=5806 <-- what's that? |
| 19:43.05 | cycokilr | am i reading it wrong? |
| 19:43.23 | cycokilr | don't give up man.. you can do it. |
| 19:43.47 | herlo | cycokilr: I see nothing in there that reads 'OOMKiller' |
| 19:43.53 | herlo | cycokilr: don't mock me. |
| 19:44.00 | herlo | it's rude and unfriendly |
| 19:44.03 | cycokilr | :51:50 localhost kernel: oom-killer: gfp_mask=0x1d2 <-- that? |
| 19:44.30 | cycokilr | seriously man.. not being a noob doesn't automatically mean you're perfect |
| 19:44.40 | cycokilr | be open.. it's fun. |
| 19:44.41 | herlo | I didn't say it did |
| 19:44.45 | herlo | and I am open |
| 19:44.53 | herlo | but that message tells you absolutely nothing |
| 19:44.54 | cycokilr | it's cool man.. i'll leave you alone. |
| 19:45.03 | cycokilr | just trying to help |
| 19:45.09 | cycokilr | but i can see i'm not needed. |
| 19:45.15 | herlo | cycokilr: heh |
| 19:48.44 | herlo | cycokilr: it's probably a particular instance then, but the kernel logs are usually in /var/log/messages. I get that, but the message I see don't look like anything you showed in that log. I'm not saying they don't show up in logs, I guess. Instead, what I am saying is that the information gets dumped to the console and it has useful information about the process that is being killed when you *cannot* get to the logs |
| 19:49.43 | herlo | usually because the system is so out of memory it's impossible for ssh to even run, let alone start an interactive session, or that it would take extremely too long to get any useful information |
| 19:50.14 | beandog | herlo: kernel panic? |
| 19:50.38 | herlo | beandog: nope |
| 19:52.12 | mindjuju | @herlo & @cycokilr -- i've been reading what you titans were talking about, didn't catch it all, but like half of it. i've got var/log/messages running in a screen (forgot how to toggle in between screens, but i'll figure that out shortly), but i'm curious about this OOM Killer |
| 19:52.18 | cycokilr | herlo: you don't need to explain how linux works to me.. i've used linux exclusively and based my career on it for the past 15 years since i got hired at Caldera. i don't profess to know everything, but i'm not so arrogant that i'm not open to listening to others when they offer help. i'm done talking about this because it's a waste of time and it's not helping anyone. and i have work to get done. |
| 19:52.37 | cycokilr | s/but/and/ |
| 19:52.54 | cycokilr | wait, but worked there. |
| 19:52.55 | cycokilr | sorry |
| 19:53.10 | mindjuju | is OOM Killer something safe to run? |
| 19:53.13 | cycokilr | anyway, back to work. |
| 19:53.16 | mindjuju | it looks pretty cool |
| 19:53.24 | beandog | cycokilr: see theres your problem .. youre not using Gentoo! |
| 19:53.26 | beandog | :D |
| 19:53.48 | cycokilr | sheesh.. does caldera even have linux still? |
| 19:53.56 | beandog | I dont think so |
| 19:53.58 | cycokilr | i generally run suse stuff.. |
| 19:54.04 | beandog | Suse is good. |
| 19:54.14 | cycokilr | but we have some rhel & centos servers around too |
| 19:54.16 | beandog | I have one of their stuffed lizards at home |
| 19:54.42 | cycokilr | cute |
| 19:54.52 | beandog | My loyalty is easily won by schwag |
| 19:55.09 | mindjuju | so oom killer is a script that runs in cron? |
| 19:55.39 | herlo | mindjuju: essentially, it's job is to watch for out of memory processes and kill them |
| 19:55.47 | herlo | oom = out of memory |
| 19:56.27 | mgeary | OOM = Octopi on Muffins, mmmmmm |
| 19:56.38 | beandog | gros. |
| 19:56.39 | beandog | gross |
| 19:57.57 | fungus | mindjuju, if oom killer has to ever kill any apps, you have a problem. |
| 19:58.18 | beandog | what if oomkiller is the first one killed by oom? |
| 19:58.26 | beandog | or cron for that matter. |
| 19:58.27 | fungus | it's purpose is only to keep your system alive long enough for you to look at it. |
| 19:58.56 | fungus | though in my experience it never actually helps. But it doesn't hurt to leave active. |
| 19:59.02 | herlo | beandog: haha, funny |
| 19:59.13 | herlo | fungus: it's helped me a few times |
| 19:59.16 | mindjuju | @fungus, getting desperate |
| 19:59.42 | herlo | but I needed console access to see it |
| 19:59.48 | cycokilr | "it's purpose is only to keep your system alive long enough for you to look at it" <-- what's your source on that statement |
| 19:59.50 | cycokilr | ? |
| 20:01.03 | cycokilr | oom killer frees up memory.. it has nothing to do with keeping a kernel from panicking or an appliction from segfaulting. it frees memory based on thresholds and once the thresholds are met it goes back to sleep.. regardless of system state. |
| 20:01.21 | mindjuju | so it isn't a cron? |
| 20:01.38 | cycokilr | the kernel manages it. |
| 20:01.50 | beandog | cycokilr: o yar? now that is cool. |
| 20:02.04 | beandog | I was just gonna say, I dont put much faith in magical fairy dust linux apps |
| 20:02.22 | cycokilr | you can turn it on or off in /proc/sys/vm/ if you want.. and there are some other 'files' in there that may contain info, but probably not.. |
| 20:02.27 | beandog | mindjuju: you can also use monit to have it check memory usage and kill it if its gettingo ut of control |
| 20:02.39 | beandog | ~monit++ |
| 20:02.39 | mindjuju | i was thinking about that beandog |
| 20:02.42 | mindjuju | the problem i have though |
| 20:03.04 | mindjuju | is that the server crashed so fast, the "watcher" script liquidweb had on the box didn't catch much of the event |
| 20:03.12 | mindjuju | all they saw was high swap, then down |
| 20:03.19 | mindjuju | down as in , server crash |
| 20:03.46 | beandog | what made it spike? |
| 20:03.51 | cycokilr | http://linux-mm.org/OOM <-- some good reading on the oom stuff if anyone is interested. |
| 20:03.51 | mindjuju | who knows |
| 20:04.22 | mindjuju | servers run swimingly then pow, down |
| 20:04.27 | mindjuju | cause unknown |
| 20:04.35 | cycokilr | bad application.. |
| 20:04.38 | cycokilr | that's my guess |
| 20:04.44 | mindjuju | i wondered about that |
| 20:04.52 | mindjuju | some kind of killer mem leak in php or something |
| 20:05.09 | mindjuju | so i was going to reduce the maxrequestperchild and drop the maxservers |
| 20:05.14 | mindjuju | and clients |
| 20:05.27 | beandog | mindjuju: did you check the logs? |
| 20:05.37 | beandog | should be something to give a hint |
| 20:05.53 | mindjuju | still looking, but i've got multiple sites, so lots of logs ...sigh... |
| 20:06.05 | mindjuju | was trying to bandage, then look at logs |
| 20:07.49 | cycokilr | or a mysql query running away with all the server resources.. |
| 20:08.08 | beandog | mindjuju: maybe you got ddos'd |
| 20:08.14 | cycokilr | so wait, it's crashing so fast that you can't even look at the messages log? |
| 20:08.20 | mindjuju | no bandwidh spike prior |
| 20:08.39 | mindjuju | yeah, i looked in /var/log/messages and there is nothing |
| 20:09.01 | mindjuju | i thought it might be bad ram? |
| 20:10.40 | beandog | mindjuju: better upgrade to a six-core with 64GB of RAM to be sure. |
| 20:10.45 | beandog | You can't be too careful. |
| 20:10.55 | cycokilr | you try running vmstat and outputting to a file so you can grab it later after the server crashes? |
| 20:11.16 | mindjuju | haven't tried that |
| 20:11.27 | mindjuju | so that would be run vmstat as a background process and have pipe to a file ? |
| 20:11.36 | cycokilr | might think about it.. it's easier than all the console redirection garbage and might help some.. |
| 20:11.40 | cycokilr | yeah |
| 20:11.53 | mindjuju | k |
| 20:12.11 | cycokilr | there are a lot of good vmstat example & explanation pages online if you look around.. |
| 20:14.16 | herlo | mindjuju: something like vmstat 2 > /dev/pts/2 might be a good way to look at it iirc |
| 20:14.33 | herlo | and do it inside screen |
| 20:15.13 | *** join/#uphpu josephscott (~josephsco@74.200.246.169) |
| 20:15.13 | *** mode/#UPHPU [+v josephscott] by ChanServ |
| 20:15.27 | herlo | ooh, wait, do a true file. then when things go south, you can reset the machine and then tail the file |
| 20:15.42 | mindjuju | that would be cool |
| 20:15.45 | herlo | or maybe a tee could work well |
| 20:16.01 | mindjuju | anything is better than driving in the blind, and that's where i feel i am |
| 20:16.52 | herlo | mindjuju: also for logs, try grep -v hostname /path/to/logfile |
| 20:17.04 | herlo | er, haha |
| 20:17.05 | herlo | no -v |
| 20:21.59 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-008.slc1.dsl.speakeasy.net) |
| 20:33.05 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-015.slc1.dsl.speakeasy.net) |
| 20:41.11 | *** join/#uphpu jkr801_ (~jkr801@dsl093-228-015.slc1.dsl.speakeasy.net) |
| 20:41.48 | *** join/#uphpu cro (~cro@204-228-149-217.ip.xmission.com) |
| 20:42.21 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-015.slc1.dsl.speakeasy.net) |
| 20:45.10 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-015.slc1.dsl.speakeasy.net) |
| 20:48.04 | DexterTheDragon | ~rlart |
| 20:48.04 | ibot | brandishes Excalibur! "With this sword, I vanquish thee, stderr-!" and lops off stderr-'s head |
| 20:48.38 | mindjuju | anybody here use splunk |
| 20:50.08 | fungus | I used it for a while, it's pretty neat. |
| 20:50.32 | fungus | But the free version is too limited for my needs, and i couldn't get it in the budget. |
| 20:50.45 | mindjuju | was trying to find out how much it costs |
| 20:51.06 | herlo | $$$$$$$$$$$$$$$$$$$$$$4 |
| 20:51.13 | herlo | s/4// |
| 20:51.22 | mindjuju | just my luck |
| 20:51.59 | herlo | well, it might only be $$$$$$$$$$$$$$$$ |
| 20:57.04 | fungus | Splunk Enterprise pricing starts at U.S. $6,000 for a 500 megabyte-per-day perpetual license, including first year support, or U.S. $2,000 per year for a term license including support. To speak to a Splunk representative about licensing, click here or call +1.866.GET.SPLUNK (866.438.7758). |
| 20:57.43 | fungus | yup |
| 20:58.17 | fungus | FYI, the free version can do up to 500MB |
| 21:00.45 | fungus | So far, a central syslog server w/ plenty of space, and GNU tools is plenty sufficient. |
| 21:01.02 | fungus | perhaps logwatch might be nice too |
| 21:03.43 | beandog | ~rage |
| 21:03.43 | ibot | extra, extra, read all about it, rage is a warrior's infinite mana bar. |
| 21:03.52 | beandog | Why dont people use SSL |
| 21:04.10 | beandog | oh well |
| 21:04.20 | fungus | because they are cheap bastards? |
| 21:05.45 | fungus | or too lazy to get a good PKI setup |
| 21:06.13 | beandog | whats pki |
| 21:06.23 | fungus | public key infrastructure |
| 21:06.43 | fungus | eg. deploying CA's everywhere, managing certs, etc |
| 21:07.13 | beandog | Hmm |
| 21:07.18 | beandog | I wish I was as smart as fungus. |
| 21:07.34 | fungus | lol |
| 21:07.49 | fungus | not really, cause when it doesn't come to these types of topics, i'm a real idiot. |
| 21:08.01 | influx | ~cookie fungus |
| 21:08.06 | beandog | Hmm |
| 21:08.09 | influx | ... okay |
| 21:08.19 | beandog | fungus: I got TRON on bluray today :D |
| 21:08.24 | influx | I forget that cookies are not doled out through that order. |
| 21:08.39 | beandog | and Peter Pan |
| 21:08.40 | beandog | and Robots |
| 21:08.41 | beandog | woo |
| 21:08.42 | fungus | i still haven't seen it. |
| 21:08.46 | fungus | ~shame fungus |
| 21:08.46 | ibot | Shame on you fungus! |
| 21:08.55 | beandog | It was pretty awesome in 3D |
| 21:19.42 | herlo | oh, wow, hail! |
| 21:27.21 | *** join/#uphpu macnewbold (~Adium@c-67-166-115-242.hsd1.ut.comcast.net) |
| 21:27.21 | *** mode/#UPHPU [+o macnewbold] by ChanServ |
| 21:27.38 | *** join/#uphpu jkr801 (~jkr801@dsl093-228-008.slc1.dsl.speakeasy.net) |
| 21:28.39 | *** join/#uphpu mindjuju (~mindjuju@173-10-26-193-BusName-utah.ut.hfc.comcastbusiness.net) |
| 21:28.39 | *** mode/#UPHPU [+o mindjuju] by ChanServ |
| 21:31.30 | beandog | poor japan |
| 21:31.48 | mindjuju | yeah, heard they had another equake |
| 21:48.55 | mgeary | serve's 'em right. Remember Pearl Harbor! |
| 21:49.23 | mgeary | oh, come on guys |
| 21:49.28 | mgeary | no trollbait? |
| 21:53.17 | influx | Your face. |
| 21:53.22 | influx | I tried. |
| 21:53.49 | jsmith | mgeary: In my line of work, you learn quickly not to feed the trolls :-p |
| 21:54.18 | mgeary | well, i'm also not a very experienced troll. I need to study mindjuju more closely |
| 21:54.36 | beandog | influx: lol |
| 21:54.39 | jsmith-away | mgeary: Be careful... he's sneakier than you'd think :-) |
| 21:54.50 | mgeary | that's my point |
| 21:55.03 | mgeary | i'm all clumsy and obtuse |
| 21:55.16 | mgeary | he's a master of stealth and rickrolling |
| 21:55.17 | beandog | thats why we dont troll you |
| 21:55.39 | beandog | its like stealing lunch money from the kid whose just gonna buy fruit anyway. |
| 21:55.43 | mgeary | hahahaha |
| 22:02.00 | mindjuju | hey influx, would you like some very tasty Salt and Vinegar chips |
| 22:02.11 | mindjuju | i LOVE salt and VINEGAR chips |
| 22:02.22 | mindjuju | where would the world be without VINEGAR as a condiment? |
| 22:03.23 | cycokilr | they are good.. and on the splunk/syslog thing earlier (sorry i was doing a shell programming training), we use sawmill. it's a lot cheaper than splunk and does quite nicely if you do need a gui.. |
| 22:03.40 | cycokilr | we also use the syslog and gnu text processing utils a lot :) |
| 22:04.00 | mindjuju | i'll take a look at sawmill and syslog |
| 22:04.14 | mindjuju | not afraid to use the console, still learning ... |
| 22:04.19 | cycokilr | i hear ya |
| 22:04.53 | cycokilr | sawmill afaik isn't based on amount of data either.. just users. |
| 22:05.31 | cycokilr | the licensing costs that is.. |
| 22:06.53 | mindjuju | oh there it is, price isn't so bad on sawmill |
| 22:07.03 | mindjuju | pro is $199 |
| 22:08.06 | mindjuju | oh wait |
| 22:08.09 | mindjuju | :( |
| 22:08.31 | beandog | I really need a break |
| 22:08.39 | mindjuju | ~give beandog a break |
| 22:08.39 | ibot | ACTION gives a break to beandog |
| 22:10.39 | josephscott | $weather 84092 |
| 22:10.40 | raifbot | josephscott - current weather for 84092 - Cloudy 45f / 7c Humidity: 69% Wind: E at 19 mph |
| 22:10.50 | josephscott | cloudy, try crazy hail |
| 22:15.51 | herlo | yeah, that's what we just had too |
| 22:16.12 | herlo | josephscott: about 5 minutes of it, then a bit of rain. Now it's drizzling |
| 22:16.31 | josephscott | so much hail on the ground it looks like snow |
| 22:17.25 | influx | mindjuju: Nice try. |
| 22:17.40 | herlo | almost did here too |
| 22:18.03 | influx | Kettle Cooked Salt & Vinegar ftw |
| 22:18.33 | beandog | $weather 84103 |
| 22:18.33 | raifbot | beandog - current weather for 84103 - Snow 36f / 2c Humidity: 95% Wind: W at 15 mph |
| 22:19.12 | carmony | Hrm, this is kind of a cool WordPress Theme |
| 22:20.47 | carmony | I really should stop tinkering |
| 22:23.03 | josephscott | carmony: ? |
| 22:23.12 | herlo | carmony: which? |
| 22:23.26 | carmony | http://www.justincarmony.com/blog/ |
| 22:28.22 | herlo | carmony: nice |
| 22:31.30 | herlo | I like the modern font and the slideshow design |
| 22:32.14 | carmony | yeah, it uses something called cufon |
| 22:32.16 | carmony | to do the font |
| 22:32.55 | herlo | cool |
| 22:45.18 | *** join/#uphpu jkr801 (~jkr801@c-67-186-232-131.hsd1.ut.comcast.net) |
| 22:46.58 | *** join/#uphpu jkr801 (~jkr801@c-67-186-232-131.hsd1.ut.comcast.net) |
| 23:06.48 | mgeary | anyone here do any Facebook development/apps? I'm trying to figure out if there's any way for User A to use an app to post something onto User B's wall |
| 23:09.08 | mindjuju | hmm |
| 23:09.23 | mindjuju | i seem to recall _pychic_ has, or maybe it was sunsparc |
| 23:11.54 | mgeary | yeah, i think _psychic_, but he's apparently left us for greener pastures |
| 23:12.01 | mindjuju | oh |
| 23:12.12 | mgeary | aha, but he's on IM... |
| 23:12.29 | mgeary | ~seen _psychic_ |
| 23:12.41 | ibot | _psychic_ <~johnander@64.0.193.15> was last seen on IRC in channel #uphpu, 22d 2h 7m 13s ago, saying: 'yeah. Old php at that.'. |
| 23:12.41 | mindjuju | no doubt, he was tired of your trollish ways |
| 23:12.41 | mindjuju | ;) |
| 23:12.41 | mgeary | psh. impossible |
| 23:24.26 | TuxToaster | $rlart |
| 23:24.26 | raifbot | takes out a seltzer bottle and sprays eggyknap in the face. You know, one of those old-school seltzer bottles clowns have? Yeah those. Anyway, consider yourself spritzed. |
| 23:32.27 | mgeary | niiiice |
| 23:32.48 | mgeary | i see you Telegraph Road and raise you... |
| 23:33.45 | mgeary | i would counter, but i want to finish this song |
| 23:55.58 | redroy | The Bravery - An Honest Mistake |
| 23:56.12 | redroy | booya |
| 23:56.45 | redroy | I'd do the cool itunes thing but I'm on pandora :( |
| 23:56.54 | mindjuju | in this case redroy, the trick is, to make the song name patch the previous song in some way |
| 23:57.02 | mindjuju | errr, not patch, match i mean |
| 23:57.09 | mindjuju | i'm doing some patching at the moment |
| 23:57.22 | redroy | son of ⦠|
| 23:57.38 | redroy | I thought we were just doing good music :) |
| 23:57.43 | mindjuju | so where i played telegraph road, he counters with On Every Street, i counter with streets with no name |
| 23:57.53 | redroy | I see... |
| 23:57.58 | mindjuju | it's both, has to match and has to be a killer song |
| 23:58.37 | mindjuju | and, you really should be able to play the song, as in, you really do have it in your library |
| 23:58.48 | mindjuju | so right now, i'm remoted home and am listening to my music library over vpn |
| 23:58.51 | mindjuju | :) |
| 23:59.04 | redroy | I need to set that up |
| 23:59.10 | redroy | that'd rock, literally |
| 23:59.28 | mindjuju | redroy, here it is in a heart beat, cheap computer box + pfsense |
| 23:59.35 | mindjuju | box doesn't even have to be new |