| 01:20.11 | *** join/#uphpu _carbohydrate (~textual@192.150.9.201) |
| 01:29.44 | *** join/#uphpu clintberry (~clintberr@38.73.150.199) |
| 01:29.44 | *** mode/#UPHPU [+v clintberry] by ChanServ |
| 02:06.18 | *** join/#uphpu _carbohydrate (~textual@192.150.9.201) |
| 02:57.43 | *** join/#uphpu jtsnow (~jsnow@71-219-31-74.slkc.qwest.net) |
| 02:57.43 | *** mode/#UPHPU [+v jtsnow] by ChanServ |
| 03:03.24 | *** join/#uphpu _carbohydrate (~textual@192.150.9.201) |
| 03:17.05 | *** join/#uphpu kendsnyder (~Adium@67-2-115-168.slkc.qwest.net) |
| 03:17.05 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 05:19.25 | *** join/#uphpu _carbohy_ (~textual@192.150.9.201) |
| 06:31.42 | *** join/#uphpu jsmith (~jsmith@207.198.119.103) |
| 06:31.50 | *** join/#uphpu jsmith (~jsmith@fedora/jsmith) |
| 06:31.50 | *** mode/#UPHPU [+v jsmith] by ChanServ |
| 06:48.39 | *** join/#uphpu tierra (tierra@2600:3c00::f03c:91ff:fe50:1850) |
| 06:56.12 | *** join/#uphpu tierra (tierra@2600:3c00::f03c:91ff:fe50:1850) |
| 08:07.45 | *** join/#uphpu OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 09:45.06 | *** join/#uphpu infobot (ibot@rikers.org) |
| 09:45.06 | *** topic/#UPHPU is September Meeting: PHP 5.6: New and Deprecated Features by Mark Niebergall | Utah PHP Users Group | uphpu.org |
| 09:45.06 | *** mode/#UPHPU [+v infobot] by ChanServ |
| 10:22.34 | *** join/#uphpu xuu (~xuu@c-24-10-241-126.hsd1.ut.comcast.net) |
| 11:35.48 | *** join/#uphpu OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 11:37.36 | *** join/#uphpu bbankes (~bbankes@67-2-219-238.slkc.qwest.net) |
| 13:53.55 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 14:04.37 | *** join/#uphpu eggyknap (~jtolley@184-11-47-170.dr01.dlta.ut.frontiernet.net) |
| 14:04.37 | *** join/#uphpu eggyknap (~jtolley@unaffiliated/eggyknap) |
| 14:04.37 | *** mode/#UPHPU [+v eggyknap] by ChanServ |
| 14:38.21 | *** join/#uphpu macnewbold (~macnewbol@inferno.apxalarm.com) |
| 14:38.21 | *** mode/#UPHPU [+o macnewbold] by ChanServ |
| 14:49.01 | *** join/#uphpu mgeary (~mgeary@75-144-8-210-utah.hfc.comcastbusiness.net) |
| 14:49.01 | *** mode/#UPHPU [+v mgeary] by ChanServ |
| 14:57.53 | *** join/#uphpu MikeHansenMe (~mike@c-67-186-219-152.hsd1.ut.comcast.net) |
| 15:04.57 | *** join/#uphpu UtahDave (~boucha@208.187.248.7) |
| 15:04.57 | *** mode/#UPHPU [+o UtahDave] by ChanServ |
| 15:15.49 | *** join/#uphpu _carbohydrate (~textual@192.150.9.201) |
| 15:17.04 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 15:17.04 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 15:29.48 | *** join/#uphpu romanovic_ (~romanovic@puritynine.com) |
| 15:30.27 | *** join/#uphpu DexterTheDragon_ (~DexterThe@68.69.164.76) |
| 15:31.32 | *** join/#uphpu Heartsbane_ (tsharpe@unaffiliated/heartsbane) |
| 15:31.32 | *** mode/#UPHPU [+v Heartsbane_] by ChanServ |
| 15:32.00 | *** join/#uphpu DexterTheDragon (~DexterThe@oalug/member/blackchaos) |
| 15:32.00 | *** mode/#UPHPU [+v DexterTheDragon] by ChanServ |
| 15:32.27 | *** join/#uphpu unum (~unum@truecomputing.biz) |
| 15:33.35 | *** join/#uphpu xuu (xuu@2001:470:f0ca:0:225:90ff:fe65:1cc2) |
| 15:46.09 | *** join/#uphpu stevecoug_ (~stevecoug@107.170.150.96) |
| 15:47.06 | *** join/#uphpu _carbohydrate (~textual@192.150.9.201) |
| 16:04.45 | *** join/#uphpu bbankes (~bbankes@50-200-100-242-static.hfc.comcastbusiness.net) |
| 16:05.55 | *** join/#uphpu bbankes_ (~bbankes@50-200-100-242-static.hfc.comcastbusiness.net) |
| 17:05.00 | *** join/#uphpu macnewbold (~macnewbol@inferno.apxalarm.com) |
| 17:05.00 | *** mode/#UPHPU [+o macnewbold] by ChanServ |
| 17:18.35 | *** join/#uphpu jtsnow (~jsnow@71-219-31-74.slkc.qwest.net) |
| 17:18.35 | *** mode/#UPHPU [+v jtsnow] by ChanServ |
| 17:54.33 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 17:54.33 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 18:09.18 | *** join/#uphpu stevecoug (~stevecoug@oalug/member/stevecoug) |
| 18:09.18 | *** mode/#UPHPU [+v stevecoug] by ChanServ |
| 18:17.24 | *** mode/#UPHPU [+v romanovic] by ChanServ |
| 18:27.18 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 18:27.18 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 18:50.02 | *** join/#uphpu marmarmar (~martin@199.30.68.138) |
| 20:01.10 | *** join/#uphpu TuxToaster_ (~TuxToaste@inferno.apxalarm.com) |
| 20:01.14 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 20:01.14 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 20:03.07 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 20:03.07 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |
| 20:13.30 | skinner | Anyone familiar with the apache server-status page? |
| 20:15.12 | skinner | fungus: You around? |
| 20:15.21 | fungus | yup |
| 20:15.53 | fungus | Ask your Q. |
| 20:15.55 | skinner | Do you have any advice for me in troubleshooting high load on my production servers? :) |
| 20:16.23 | skinner | Yesterday, all of a sudden at 10:50 am, our 4 production servers cpu's all maxed |
| 20:16.24 | fungus | high load as in load average reported by uptime, top, etc? |
| 20:16.36 | skinner | yes top says: |
| 20:16.36 | skinner | load average: 29.39, 25.73, 23.64 |
| 20:16.42 | fungus | ouch |
| 20:16.44 | skinner | yeah |
| 20:16.59 | skinner | unfortunately, i'm the best they have here as far as a server admin goes |
| 20:17.00 | fungus | where does the CPU usage come from? |
| 20:17.18 | skinner | i believe apache |
| 20:17.32 | fungus | top shows where it comes from, both at the top in terms of User, System, Idle, Wait, etc. |
| 20:17.39 | fungus | and the bottom as in which processes. |
| 20:18.28 | skinner | top - 14:18:20 up 80 days, 2:45, 2 users, load average: 26.37, 25.36, 23.73 |
| 20:18.28 | skinner | Tasks: 206 total, 37 running, 169 sleeping, 0 stopped, 0 zombie |
| 20:18.28 | skinner | %Cpu(s): 96.7 us, 3.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st |
| 20:18.28 | skinner | KiB Mem: 15375676 total, 14170244 used, 1205432 free, 347892 buffers |
| 20:18.28 | skinner | KiB Swap: 0 total, 0 used, 0 free, 10269640 cached |
| 20:18.33 | skinner | user |
| 20:18.37 | *** join/#uphpu itimber (~Adium@184.206.245.72) |
| 20:18.38 | *** mode/#UPHPU [+v itimber] by ChanServ |
| 20:18.44 | skinner | tons of apache processes |
| 20:19.08 | fungus | So high user CPU. That usually represents your application logic. |
| 20:19.44 | fungus | Perhaps your server can't handle running that many apache processes at once with your application. |
| 20:21.01 | skinner | The strange thing is this spike started yesterday morning and has been persistent ever since |
| 20:21.12 | skinner | the only change in our files was a few tweaks to a css file |
| 20:21.49 | skinner | hosting on AWS, we have a load balancer with 4 c3.2xlarge instances |
| 20:22.02 | jsmith | Can your run "iotop" on your system, to see if it's related to disk IO? |
| 20:22.23 | skinner | iotop: command not found |
| 20:22.29 | skinner | we be running debian 7 |
| 20:22.51 | jsmith | apt-get install iotop? |
| 20:22.51 | skinner | @jsmith: one sec...installing |
| 20:23.08 | jsmith | There's also another utility called "apachetop", but it's not quite as useful |
| 20:23.42 | skinner | disk read staying around 0 and write about the same |
| 20:23.52 | skinner | memory usage is minmal |
| 20:23.57 | skinner | just crazy high cpu |
| 20:24.17 | jsmith | OK, then the next thing is to install htop, and see what's chewing through your CPU |
| 20:24.25 | jsmith | (is it really apache processes, or something else) |
| 20:24.34 | jsmith | Another way to check would be to look at your server-status page |
| 20:24.44 | jsmith | Look for lines with "W" in the "M" column |
| 20:25.12 | jsmith | And then look at the CPU and SS columns for those rows |
| 20:27.34 | skinner | jsmith: I've install htop and running it now |
| 20:28.35 | skinner | yeah looks like apache is the culprit |
| 20:28.54 | skinner | some of the processes using cpu% 10 - 25% |
| 20:28.58 | skinner | per process :\ |
| 20:29.16 | skinner | server-status page.... |
| 20:29.47 | skinner | lots of the items show a request of: |
| 20:29.54 | skinner | ..reading.. |
| 20:30.22 | jsmith | Being hit by a DDoS attack? |
| 20:30.39 | skinner | I've had that thought |
| 20:31.45 | skinner | any good way to tell? |
| 20:31.47 | jsmith | It looks to me that Apache is just struggling to keep up with the requests |
| 20:31.56 | jsmith | Have you looked at your access logs? |
| 20:32.02 | jsmith | Are the requests hitting one particular page? |
| 20:32.12 | fungus | skinner, perhaps try reducing your max processes so the server doesn't fall to it's knees during high traffic periods. |
| 20:32.21 | jsmith | Are they coming from strange IPs? |
| 20:32.41 | fungus | Good idea look for a DoS attack and block what you can first. |
| 20:32.48 | skinner | it's hard to say because they are all behind our load balancer so most of the requests show the load balancer's ip |
| 20:33.00 | jsmith | (for example, if most of your traffic is typically from North America, are you seeing a high volume of traffic from odd countries?) |
| 20:35.04 | skinner | I'm not sure the best way to see that data, also we do have users world wide |
| 20:35.22 | skinner | a large chunk in the uk, australia, japan |
| 20:35.27 | skinner | i think a few more |
| 20:37.12 | jsmith | Well, I typically see attacks like that coming from China, Russia, Botswana... |
| 20:37.42 | *** join/#uphpu itimber1 (~Adium@184-206-152-137.pools.spcsdns.net) |
| 20:39.26 | skinner | ok using a 3rd party website looking up some ips |
| 20:39.41 | skinner | running this command: |
| 20:39.42 | skinner | <PROTECTED> |
| 20:41.08 | skinner | i've seen some ips from ukraine, china, |
| 20:41.16 | skinner | so....probably a dos attack |
| 20:42.45 | *** join/#uphpu itimber (~Adium@184-206-166-32.pools.spcsdns.net) |
| 20:42.46 | *** mode/#UPHPU [+v itimber] by ChanServ |
| 20:44.36 | jsmith | skinner: Can you put a CDN in front of your load balancers? |
| 20:45.28 | jsmith | skinner: I've seen CDNs like MaxCDN and CloudFlare handle blocking the DoS attacks quite well... |
| 20:45.49 | skinner | Yeah I just found a site talking about using CloudFlare |
| 20:46.08 | skinner | it's possible, except we have like 150+ domains pointed to our system and we only control about half of them |
| 20:46.28 | jsmith | Yeah, that's a problem |
| 20:46.43 | jsmith | Anyhoo -- wish I could stick around and help -- but I've gotta run |
| 20:49.49 | skinner | jsmith: Thanks for the help :) |
| 21:03.32 | *** part/#uphpu littleSkinner (~JSkinn@asa-2.fbp.ore.fiber.net) |
| 21:08.17 | *** join/#uphpu clintberry (~clintberr@205.197.215.90) |
| 21:08.17 | *** mode/#UPHPU [+v clintberry] by ChanServ |
| 21:18.51 | *** join/#uphpu UtahDave (~boucha@72.250.214.74) |
| 21:18.52 | *** mode/#UPHPU [+o UtahDave] by ChanServ |
| 22:05.21 | *** join/#uphpu kendsnyder (~Adium@67.214.245.39) |
| 22:05.35 | *** mode/#UPHPU [+v kendsnyder] by ChanServ |
| 22:59.05 | *** join/#uphpu _carbohydrate (~textual@206.251.47.14) |
| 23:18.08 | *** join/#uphpu _carbohydrate (~textual@206.251.47.14) |
| 23:22.43 | *** join/#uphpu TuxToaster (~TuxToaste@inferno.apxalarm.com) |
| 23:22.43 | *** mode/#UPHPU [+v TuxToaster] by ChanServ |