| 01:03.34 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 01:03.34 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 01:34.51 | *** join/#uphpu stevecoug (~stevecoug@oalug/member/stevecoug) |
| 01:34.52 | *** mode/#UPHPU [+v stevecoug] by ChanServ |
| 02:28.55 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 02:28.55 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 02:47.38 | *** join/#uphpu bbankes_ (~bbankes@2601:680:c402:3225:d147:8a9f:d044:1e8d) |
| 02:56.11 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 02:56.11 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 03:33.42 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 03:33.42 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 04:06.48 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 04:06.48 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 04:31.23 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 04:31.23 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 04:34.39 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 04:34.40 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 04:54.45 | *** join/#uphpu TodPunk (~Tod@67-2-45-172.slkc.qwest.net) |
| 05:19.47 | *** join/#uphpu jtsnow (~jtsnow@174-23-191-204.slkc.qwest.net) |
| 05:19.47 | *** mode/#UPHPU [+v jtsnow] by ChanServ |
| 05:53.04 | *** join/#uphpu stevecoug (~stevecoug@oalug/member/stevecoug) |
| 05:53.05 | *** mode/#UPHPU [+v stevecoug] by ChanServ |
| 15:08.48 | *** join/#uphpu eggyknap (~jtolley@50.109.215.136) |
| 15:08.48 | *** join/#uphpu eggyknap (~jtolley@unaffiliated/eggyknap) |
| 15:08.48 | *** mode/#UPHPU [+v eggyknap] by ChanServ |
| 15:10.31 | *** join/#uphpu bbankes_ (~bbankes@50-200-100-242-static.hfc.comcastbusiness.net) |
| 15:45.17 | *** join/#uphpu stevecoug (~stevecoug@oalug/member/stevecoug) |
| 15:45.17 | *** mode/#UPHPU [+v stevecoug] by ChanServ |
| 15:54.04 | *** join/#uphpu josephscott (~josephsco@192.0.80.201) |
| 16:11.34 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 16:11.34 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 16:59.53 | *** join/#uphpu ttbakiatwoam (~ttbakiatw@96.92.83.228) |
| 17:14.33 | *** join/#uphpu ninnypan_ (~ninnypant@unaffiliated/ninnypants) |
| 17:14.33 | *** mode/#UPHPU [+v ninnypan_] by ChanServ |
| 17:25.14 | *** join/#uphpu jtsnow (~jtsnow@199.16.150.186) |
| 17:25.15 | *** mode/#UPHPU [+v jtsnow] by ChanServ |
| 18:17.54 | *** join/#uphpu stevecoug (~stevecoug@oalug/member/stevecoug) |
| 18:17.54 | *** mode/#UPHPU [+v stevecoug] by ChanServ |
| 18:29.14 | *** join/#uphpu ttbakiat_ (~ttbakiatw@96.92.83.225) |
| 18:30.34 | *** join/#uphpu ttbakiatwoam (~ttbakiatw@96.92.83.228) |
| 18:43.16 | *** join/#uphpu ttbakiat_ (~ttbakiatw@96.92.83.228) |
| 18:44.45 | *** join/#uphpu ttbakiatwoam (~ttbakiatw@96.92.83.225) |
| 19:03.24 | *** join/#uphpu ttbakiatwoam (~ttbakiatw@96.92.83.228) |
| 19:48.31 | mgeary | hey folks, do people still chat here? Iâm trying to avoid the slack migration |
| 19:49.37 | josephscott | not much any more |
| 19:49.42 | mgeary | sigh |
| 19:49.46 | josephscott | though slack isn't exactly super busy either |
| 19:49.55 | mgeary | interesting |
| 19:50.51 | stevecoug | So... who's going to Ski PHP? There will be lots of discussion there. ;) |
| 19:51.18 | mgeary | iâm trying to get some tips on configuring a server so it better withstands a WP-centric attack. Weâre using iThemes security, and it is doing a decent job, but twice now the database (mysql) has been brought to a shutdown. No attack has âgotten inâ, but these database shutdowns are aggravating |
| 19:54.39 | mgeary | it looks like mysql is running out of memory |
| 19:54.52 | unum | I'll be there |
| 19:55.32 | mgeary | limit max_connections, maybe? |
| 19:56.07 | josephscott | stevecoug: did you get the registration for the 3 of us from Automattic? |
| 19:56.19 | unum | mgeary: do you know how many people are hitting the site at once |
| 19:56.35 | unum | if you limit max_connections you'll be preventing people from the loading the page |
| 19:56.43 | mgeary | unum: not a ton. By far the most traffic is coming from brute-force attacks |
| 19:56.54 | mgeary | most of which are being 403âd fairly quickly |
| 19:57.04 | mgeary | though it might still be putting too heavy a burden on the DB |
| 19:57.11 | stevecoug | josephscott: no, we don't have your registrations |
| 19:57.19 | unum | people trying to brute-force login to wp? |
| 19:57.32 | mgeary | yeah, but after 5 failures, theyâre being 403âd |
| 19:57.35 | *** part/#uphpu ttbakiatwoam (~ttbakiatw@96.92.83.228) |
| 19:57.41 | josephscott | stevecoug: k, Rebecca should be doing that, or at least that is what she said :-) |
| 19:58.03 | stevecoug | josephscott: She hasn't been responding to my emails, I'm not sure what's going on there |
| 19:58.23 | unum | you should be able to watch queries in top and see if there are any that are eating up an unexpected amount of ram |
| 19:58.42 | josephscott | stevecoug: ok, I'll check in again there |
| 19:59.03 | mgeary | itâs pretty intermittent. Iâd have to react fairly quickly to try to inspect things when an attack is happening. Itâs been late saturday night the last two weeks running |
| 20:00.29 | unum | no cron job at that time? |
| 20:00.49 | mgeary | not for root |
| 20:01.06 | mgeary | nor any other user created by us |
| 20:01.11 | josephscott | stevecoug: apparently she has been on vacation, and currently at a meetup, you should be getting something from her, you are welcome to ping me if you need a second avenue for getting an answer on something |
| 20:01.49 | mgeary | âmysqld invoked oom-killer" |
| 20:02.45 | mgeary | might try to reproduce it using ab |
| 20:15.40 | mgeary | whatâs weird, though, is that the DB went down at 20:59:21, according to syslog, but the brute force attempts were 12 hours earlier. Sounds like maybe iâve not got the right cause for my effect |
| 20:16.36 | mgeary | although some of the timestamps on the apache error log are weird. Check this out: http://dpaste.com/1K17NA4 |
| 20:17.09 | mgeary | same IP, within a fraction of a second of each other, but every few entries the server says happened at midnight, rather than 7am |
| 20:19.47 | unum | looks like a UTC/Local issue |
| 20:19.54 | unum | or it's just plain weird |
| 20:20.14 | mgeary | yeah, in the access log it is a bit clearer" |
| 20:20.20 | mgeary | 46.235.47.28 - - [09/Jan/2016:07:41:08 +0000] "POST /wp-login.php HTTP/1.0" 403 474 "-" "-" |
| 20:20.24 | mgeary | 46.235.47.28 - - [09/Jan/2016:00:41:08 -0700] "POST /wp-login.php HTTP/1.0" 403 474 "-" "-" |
| 20:20.36 | mgeary | some are UTC and some are -0700 |
| 20:20.47 | mgeary | not sure what would cause that |
| 20:21.03 | utahcon | IRC > Slack |
| 20:52.52 | mgeary | so, i was able to get the server to hit some OOM errors using AB with 100 concurrent requests. Whatâs the best way to configure apache to throttle requests for this? Iâd rather have some users wait for a connection that hit an out of memory error |
| 20:58.03 | mgeary | messing around with MaxClients doesnât seem to have an impact |
| 21:00.38 | utahcon | Apache + PHP? Are you using FPM? |
| 21:03.05 | mgeary | yes and no |
| 21:05.36 | *** join/#uphpu jtsnow_ (~jtsnow@199.16.150.186) |
| 21:05.36 | *** mode/#UPHPU [+v jtsnow_] by ChanServ |
| 21:09.57 | DexterTheDragon | if it's mysql that's getting OOM wouldn't it be better to tune mysql's connection limit so that apache waits on it instead? |
| 21:14.18 | mgeary | well, from what i can tell, the monitoring agent there is saying, âaak. memory is low! What should i kill? Hmm⦠letâs try mysql" |
| 21:14.27 | mgeary | because sometimes itâs killed apache processes instead |
| 21:15.08 | mgeary | what i donât understand is why i can set all the max values to something pretty low, then fire off apache bench, and still see a ton of apache processes stack up in âtop" |
| 21:15.23 | mgeary | itâs like iâm being overridden somewhere, but i donât think i am |
| 21:16.33 | DexterTheDragon | MaxClients is the main one you'll probably want to adjust for apache |
| 21:16.47 | mgeary | did that. Set it to 2, and got no results |
| 23:51.43 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 23:51.43 | *** mode/#UPHPU [+v ninnypants] by ChanServ |
| 23:54.41 | *** join/#uphpu ninnypants (~ninnypant@unaffiliated/ninnypants) |
| 23:54.41 | *** mode/#UPHPU [+v ninnypants] by ChanServ |