Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout

Reindl Harald
on httpd this is just a single config line

https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500

while we have ratelimiting and max-connection per ip/subnet to solve
that problems on the firewall instead in the attacked application it
makes me tired in case of external security audits that i have to
explain every single time that this is because of rate-control
whitelists for the scanner IP

Severity
Medium

Type
Configuration
Reported by module Slow_HTTP_DOS

Description
Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.

Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
protocol, by design, requires requests to be completely received by the
server before they are processed. If an HTTP request is not complete, or
if the transfer rate is very low, the server keeps its resources busy
waiting for the rest of the data. If the server keeps too many resources
busy, this creates a denial of service.

Impact
A single machine can take down another machine's web server with minimal
bandwidth and side effects on unrelated
services and ports.
Reply | Threaded
Open this post in threaded view
|

Re: Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout

Reindl Harald
the current timeout configs are terrible

16:04:00 - request start
16:05:00 - still no repsone while expected
16:09:00 - Proxy: Inactivity Timeout

WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
minutes to close the backend connection and so for whatever reason
"proxy.config.http.transaction_no_activity_timeout_out" get triggerd

20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
'http://example.com/timeout.php' (setting last failure time)
20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://example.com/timeout.php'

and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
follow up requests which would hahve been served promptly (at least only
for that domain and not the other 200 on the same origin IP)

<?php
$start = time();
header('Content-Type: text/plain');
sleep(60);
echo "OK: ", (time() - $start), "\n";
?>

CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0


Am 21.08.2017 um 15:33 schrieb Reindl Harald:

> on httpd this is just a single config line
>
> https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
> RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
>
> while we have ratelimiting and max-connection per ip/subnet to solve
> that problems on the firewall instead in the attacked application it
> makes me tired in case of external security audits that i have to
> explain every single time that this is because of rate-control
> whitelists for the scanner IP
>
> Severity
> Medium
>
> Type
> Configuration
> Reported by module Slow_HTTP_DOS
>
> Description
> Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.
>
> Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
> protocol, by design, requires requests to be completely received by the
> server before they are processed. If an HTTP request is not complete, or
> if the transfer rate is very low, the server keeps its resources busy
> waiting for the rest of the data. If the server keeps too many resources
> busy, this creates a denial of service.
>
> Impact
> A single machine can take down another machine's web server with minimal
> bandwidth and side effects on unrelated
> services and ports.
Reply | Threaded
Open this post in threaded view
|

Re: Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout

Reindl Harald
[root@arrakis:~]$ tail -f php_error.log
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60


WTF - log on the Backend-Server
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:32:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:33:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:34:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:35:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60

and now what - trafficserver is trying every 60 seconds for
"proxy.config.http.transaction_no_activity_timeout_out" and don't say
anything to the client?

WTF - log on the trafficserver
20170821.16h30m39s CONNECT:[0] could not connect [INACTIVE_TIMEOUT] to
*.*.*.* for 'http://www.example.com/timeout.php?x=1'
20170821.16h35m41s CONNECT: could not connect to *.*.*.* for
'http://www.example.com/timeout.php?x=1' (setting last failure time)
20170821.16h35m41s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://www.example.com/timeout.php?x=1'

yeah it can take some time on a output-buffering backend application to
respond and hence "CONFIG
proxy.config.http.transaction_no_activity_timeout_out INT 300" so i have
no idea why the script below leads to 5 log-entries and trafficserver
let the client sit and wait all the time without any response

<?php
ob_start();
$start = time();
sleep(60);
echo "OK: ", (time() - $start), "\n";
error_log('TEST PROXY/ORIGIN TIMOUT: 60');
?>


Am 21.08.2017 um 16:15 schrieb Reindl Harald:

> the current timeout configs are terrible
>
> 16:04:00 - request start
> 16:05:00 - still no repsone while expected
> 16:09:00 - Proxy: Inactivity Timeout
>
> WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
> minutes to close the backend connection and so for whatever reason
> "proxy.config.http.transaction_no_activity_timeout_out" get triggerd
>
> 20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
> 'http://example.com/timeout.php' (setting last failure time)
> 20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
> Out) for 'http://example.com/timeout.php'
>
> and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
> follow up requests which would hahve been served promptly (at least only
> for that domain and not the other 200 on the same origin IP)
>
> <?php
> $start = time();
> header('Content-Type: text/plain');
> sleep(60);
> echo "OK: ", (time() - $start), "\n";
> ?>
>
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
> CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
> CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
> CONFIG proxy.config.http.transaction_active_timeout_in INT 900
> CONFIG proxy.config.http.transaction_active_timeout_out INT 0
> CONFIG proxy.config.http.accept_no_activity_timeout INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0
>
>
> Am 21.08.2017 um 15:33 schrieb Reindl Harald:
>> on httpd this is just a single config line
>>
>> https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
>> RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
>>
>> while we have ratelimiting and max-connection per ip/subnet to solve
>> that problems on the firewall instead in the attacked application it
>> makes me tired in case of external security audits that i have to
>> explain every single time that this is because of rate-control
>> whitelists for the scanner IP
>>
>> Severity
>> Medium
>>
>> Type
>> Configuration
>> Reported by module Slow_HTTP_DOS
>>
>> Description
>> Your web server is vulnerable to Slow HTTP DoS (Denial of Service)
>> attacks.
>>
>> Slowloris and Slow HTTP POST DoS attacks rely on the fact that the
>> HTTP protocol, by design, requires requests to be completely received
>> by the server before they are processed. If an HTTP request is not
>> complete, or if the transfer rate is very low, the server keeps its
>> resources busy waiting for the rest of the data. If the server keeps
>> too many resources busy, this creates a denial of service.
>>
>> Impact
>> A single machine can take down another machine's web server with
>> minimal bandwidth and side effects on unrelated
>> services and ports.