2009年8月31日星期一

Re: memcached problems...

i think so, yes

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,5434,5455#msg-5455

Re: Weird Memleak problem

Oh I also notice another issue, when ram usage is high like this, when i
reload nginx it does this:
root 5776 0.0 0.1 19528 4876 ? Ss Aug30 0:00 nginx:
master process /usr/local/nginx/sbin/nginx -c conf/ng1.conf
www 5777 2.3 2.3 184060 94364 ? S Aug30 28:53 nginx:
worker process is shutting down
www 5778 2.3 2.3 315500 95496 ? S Aug30 29:23 nginx:
worker process is shutting down
www 5781 2.2 3.8 342904 154144 ? S Aug30 28:08 nginx:
worker process is shutting down
www 5784 2.4 4.9 508552 199036 ? S Aug30 30:17 nginx:
worker process is shutting down
www 5785 2.4 3.0 349364 122764 ? S Aug30 30:18 nginx:
worker process is shutting down
www 5786 2.3 5.8 511264 234928 ? R Aug30 29:27 nginx:
worker process is shutting down
www 5787 2.3 2.7 209608 111840 ? S Aug30 29:29 nginx:
worker process is shutting down
www 5788 2.3 6.5 406672 263352 ? S Aug30 29:24 nginx:
worker process is shutting down
www 5789 2.3 2.4 275580 98532 ? S Aug30 29:25 nginx:
worker process is shutting down
www 5790 2.3 3.5 316852 145448 ? S Aug30 29:10 nginx:
worker process is shutting down
www 5791 2.3 6.5 505276 265980 ? S Aug30 29:13 nginx:
worker process is shutting down
www 5792 2.4 1.5 255980 62228 ? S Aug30 30:05 nginx:
worker process is shutting down
www 5793 2.3 5.4 546796 221996 ? S Aug30 29:34 nginx:
worker process is shutting down
www 5794 2.3 2.8 250940 116300 ? S Aug30 28:39 nginx:
worker process is shutting down
www 5795 2.3 2.1 246528 86908 ? S Aug30 29:08 nginx:
worker process is shutting down
www 5796 2.4 3.9 340724 161644 ? S Aug30 30:20 nginx:
worker process is shutting down
www 6347 2.8 0.7 43356 28640 ? S 17:10 0:13 nginx:
worker process
www 6348 2.7 0.6 42472 27636 ? S 17:10 0:13 nginx:
worker process
www 6349 2.4 0.6 40776 25672 ? S 17:10 0:11 nginx:
worker process
www 6350 2.3 0.6 41864 27104 ? S 17:10 0:11 nginx:
worker process
www 6351 2.3 0.6 42728 28028 ? S 17:10 0:11 nginx:
worker process
www 6352 2.3 0.6 43040 27828 ? S 17:10 0:11 nginx:
worker process
www 6353 2.6 0.7 43972 29360 ? S 17:10 0:12 nginx:
worker process
www 6354 2.4 0.7 46104 30892 ? S 17:10 0:11 nginx:
worker process
www 6355 2.0 0.6 40252 25776 ? S 17:10 0:09 nginx:
worker process
www 6356 2.9 0.7 46408 31396 ? S 17:10 0:14 nginx:
worker process
www 6357 2.2 0.6 41600 26816 ? S 17:10 0:10 nginx:
worker process
www 6358 2.3 0.6 42536 27960 ? S 17:10 0:11 nginx:
worker process
www 6359 2.4 0.6 40564 26120 ? S 17:10 0:11 nginx:
worker process
www 6360 2.4 0.6 42272 27656 ? S 17:10 0:11 nginx:
worker process
www 6361 2.8 0.7 46764 30336 ? S 17:10 0:14 nginx:
worker process
www 6362 2.1 0.6 40224 25636 ? S 17:10 0:10 nginx:
worker process


and the processes stay in 'process is shutting down' for a LONG time..
I've waited quite a while before
doing a restart instead of reload to clear them out. Not sure what
would cause this?
Most of my connection timeouts are set between 60-120seconds. Except SSL
session one is 10m, but i've waited
hours for the old processes to shut down.

Thanks again


Cliff Wells wrote:
> On Mon, 2009-08-31 at 16:52 -0400, Paul wrote:
>
>> Had some issues with people uploading/downloading files before on 0.6
>> and the buffers fixed it..
>>
>
> What sort of issues? Timeouts or "client body too large"? I regularly
> upload 10MB+ CSV files to a proxied application with the following:
>
> client_max_body_size 20m;
> client_body_buffer_size 128k;
> proxy_connect_timeout 90;
> proxy_send_timeout 90;
> proxy_read_timeout 90;
> proxy_buffer_size 4k;
> proxy_buffers 4 32k;
> proxy_busy_buffers_size 64k;
> proxy_temp_file_write_size 64k;
>
>
> Cliff
>
>
>> What would you suggest I change the config to? I will try and let you
>> know if I still see the problem.
>> Thank you.
>>
>>
>> Igor Sysoev wrote:
>>
>>> On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
>>>
>>>
>>>
>>>> I know, but this problem has never occured until recently.. Once the
>>>> request is done, it should remove the memory allocation, but it looks
>>>> like maybe it isn't?
>>>>
>>>>
>>> No. These workers run since Aug 7 and handle up to 3,800r/s:
>>>
>>> ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
>>> PID PPID %CPU VSZ WCHAN STARTED COMMAND
>>> 42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
>>> 51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng
>>>
>>>
>>>
>>>> The only difference in a month ago and now is that we have more server
>>>> entries, and more requests per second. It used to not even use a gig of
>>>> ram doing 200 requests/sec and now it keeps using more and more ram
>>>> until it fills the entire ram and swap and errors out 8gb+ ..
>>>> What would you suggest?
>>>>
>>>>
>>> Just 250 simultaneous proxied connections take 8G. And 250 connections
>>> are too little in modern world. Why at all you have set up so huge buffers ?
>>>
>>>
>>>
>>>> Igor Sysoev wrote:
>>>>
>>>>
>>>>> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
>>>>>> memleak issue.
>>>>>> What happens is as soon as I start nginx, it starts using ram and
>>>>>> continues to use more and more and more over a 16 hour period is
>>>>>> consumes 8gb ram and all the swap and then errors out because it cant
>>>>>> use any more.
>>>>>> This never used to happen until recently and the only difference at all
>>>>>> in the config is more server entries.
>>>>>>
>>>>>> here's the config:
>>>>>>
>>>>>> user www www;
>>>>>>
>>>>>> worker_processes 16;
>>>>>> error_log logs/error.log;
>>>>>> worker_rlimit_nofile 65000;
>>>>>>
>>>>>> events
>>>>>> {
>>>>>>
>>>>>> worker_connections 40000;
>>>>>> }
>>>>>>
>>>>>> ####### HTTP SETTING
>>>>>> http
>>>>>> {
>>>>>> access_log off;
>>>>>> log_format alot '$remote_addr - $remote_user [$time_local] '
>>>>>> '"$request" $status $body_bytes_sent '
>>>>>> '"$http_referer" "$http_user_agent" "$http_accept"
>>>>>> $connection';
>>>>>> sendfile on;
>>>>>> tcp_nopush on;
>>>>>> tcp_nodelay on;
>>>>>> keepalive_timeout 0;
>>>>>> output_buffers 16 128k;
>>>>>> server_tokens off;
>>>>>> ssl_verify_client off;
>>>>>> ssl_session_timeout 10m;
>>>>>> # ssl_session_cache shared:SSL:500000;
>>>>>> include /usr/local/nginx/conf/mime.types;
>>>>>> default_type application/octet-stream;
>>>>>>
>>>>>> # cache_max_size 24;
>>>>>>
>>>>>> gzip on;
>>>>>> gzip_min_length 512;
>>>>>> gzip_buffers 64 32k;
>>>>>> gzip_types text/plain text/html text/xhtml text/css text/js;
>>>>>>
>>>>>> proxy_buffering on;
>>>>>> proxy_buffer_size 32m;
>>>>>> proxy_buffers 16 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> These settings allocate 32M buffer for each proxied request.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> proxy_busy_buffers_size 64m;
>>>>>> proxy_temp_file_write_size 2048m;
>>>>>> proxy_intercept_errors on;
>>>>>> proxy_ssl_session_reuse off;
>>>>>> proxy_read_timeout 120;
>>>>>> proxy_connect_timeout 60;
>>>>>> proxy_send_timeout 120;
>>>>>> client_body_buffer_size 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> This setting allocates 32M buffer for each request with body.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> client_header_buffer_size 64k;
>>>>>> large_client_header_buffers 16 64k;
>>>>>> client_max_body_size 16m;
>>>>>>
>>>>>>
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:80;
>>>>>> location /
>>>>>> {
>>>>>>
>>>>>> proxy_pass http://3.4.5.6;
>>>>>> proxy_redirect http://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> }
>>>>>> }
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:443;
>>>>>>
>>>>>> ssl on;
>>>>>> ssl_certificate
>>>>>> /usr/local/nginx/conf/whatever.com.crt;
>>>>>> ssl_certificate_key
>>>>>> /usr/local/nginx/conf/whatever.com.key;
>>>>>> ssl_protocols SSLv3;
>>>>>> ssl_ciphers HIGH:!ADH;
>>>>>> ssl_prefer_server_ciphers on;
>>>>>> location /
>>>>>> {
>>>>>> proxy_pass https://3.4.5.6;
>>>>>> proxy_redirect https://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-FORWARDED_PROTO https;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> And these server entries repeated about 60 or so times and that's it.
>>>>>> When it was around 40 we never had a memleak issue.
>>>>>>
>>>>>> This is on Linux, kernel is 2.6.25
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> GloboTech Communications
>>>>>> Phone: 1-514-907-0050 x 215
>>>>>> Toll Free: 1-(888)-GTCOMM1
>>>>>> Fax: 1-(514)-907-0750
>>>>>> paul@gtcomm.net
>>>>>> http://www.gtcomm.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> GloboTech Communications
>>>> Phone: 1-514-907-0050 x 215
>>>> Toll Free: 1-(888)-GTCOMM1
>>>> Fax: 1-(514)-907-0750
>>>> paul@gtcomm.net
>>>> http://www.gtcomm.net
>>>>
>>>>
>>>>
>>>
>>>

--
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul@gtcomm.net
http://www.gtcomm.net

Re: Weird Memleak problem

Yes, it was timeouts, but that could have been fixed in the newer nginx
versions. I'm just wondering why
the ram usage hasn't occured before and and is now.. It never used to
use much at all and now it's using an
exorbitant amount of ram with nothing major changed.. That's why I
figured something was wrong.
It takes it about 12 hours or so to use 6-7gb of ram. That is another
reason why I thought something might be wrong.
I'll try changing the buffers and watching the RAM usage again and see
what happens.
Thank you, and Igor, for your inputs.

Paul

Cliff Wells wrote:
> On Mon, 2009-08-31 at 16:52 -0400, Paul wrote:
>
>> Had some issues with people uploading/downloading files before on 0.6
>> and the buffers fixed it..
>>
>
> What sort of issues? Timeouts or "client body too large"? I regularly
> upload 10MB+ CSV files to a proxied application with the following:
>
> client_max_body_size 20m;
> client_body_buffer_size 128k;
> proxy_connect_timeout 90;
> proxy_send_timeout 90;
> proxy_read_timeout 90;
> proxy_buffer_size 4k;
> proxy_buffers 4 32k;
> proxy_busy_buffers_size 64k;
> proxy_temp_file_write_size 64k;
>
>
> Cliff
>
>
>> What would you suggest I change the config to? I will try and let you
>> know if I still see the problem.
>> Thank you.
>>
>>
>> Igor Sysoev wrote:
>>
>>> On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
>>>
>>>
>>>
>>>> I know, but this problem has never occured until recently.. Once the
>>>> request is done, it should remove the memory allocation, but it looks
>>>> like maybe it isn't?
>>>>
>>>>
>>> No. These workers run since Aug 7 and handle up to 3,800r/s:
>>>
>>> ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
>>> PID PPID %CPU VSZ WCHAN STARTED COMMAND
>>> 42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
>>> 42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
>>> 51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng
>>>
>>>
>>>
>>>> The only difference in a month ago and now is that we have more server
>>>> entries, and more requests per second. It used to not even use a gig of
>>>> ram doing 200 requests/sec and now it keeps using more and more ram
>>>> until it fills the entire ram and swap and errors out 8gb+ ..
>>>> What would you suggest?
>>>>
>>>>
>>> Just 250 simultaneous proxied connections take 8G. And 250 connections
>>> are too little in modern world. Why at all you have set up so huge buffers ?
>>>
>>>
>>>
>>>> Igor Sysoev wrote:
>>>>
>>>>
>>>>> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
>>>>>> memleak issue.
>>>>>> What happens is as soon as I start nginx, it starts using ram and
>>>>>> continues to use more and more and more over a 16 hour period is
>>>>>> consumes 8gb ram and all the swap and then errors out because it cant
>>>>>> use any more.
>>>>>> This never used to happen until recently and the only difference at all
>>>>>> in the config is more server entries.
>>>>>>
>>>>>> here's the config:
>>>>>>
>>>>>> user www www;
>>>>>>
>>>>>> worker_processes 16;
>>>>>> error_log logs/error.log;
>>>>>> worker_rlimit_nofile 65000;
>>>>>>
>>>>>> events
>>>>>> {
>>>>>>
>>>>>> worker_connections 40000;
>>>>>> }
>>>>>>
>>>>>> ####### HTTP SETTING
>>>>>> http
>>>>>> {
>>>>>> access_log off;
>>>>>> log_format alot '$remote_addr - $remote_user [$time_local] '
>>>>>> '"$request" $status $body_bytes_sent '
>>>>>> '"$http_referer" "$http_user_agent" "$http_accept"
>>>>>> $connection';
>>>>>> sendfile on;
>>>>>> tcp_nopush on;
>>>>>> tcp_nodelay on;
>>>>>> keepalive_timeout 0;
>>>>>> output_buffers 16 128k;
>>>>>> server_tokens off;
>>>>>> ssl_verify_client off;
>>>>>> ssl_session_timeout 10m;
>>>>>> # ssl_session_cache shared:SSL:500000;
>>>>>> include /usr/local/nginx/conf/mime.types;
>>>>>> default_type application/octet-stream;
>>>>>>
>>>>>> # cache_max_size 24;
>>>>>>
>>>>>> gzip on;
>>>>>> gzip_min_length 512;
>>>>>> gzip_buffers 64 32k;
>>>>>> gzip_types text/plain text/html text/xhtml text/css text/js;
>>>>>>
>>>>>> proxy_buffering on;
>>>>>> proxy_buffer_size 32m;
>>>>>> proxy_buffers 16 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> These settings allocate 32M buffer for each proxied request.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> proxy_busy_buffers_size 64m;
>>>>>> proxy_temp_file_write_size 2048m;
>>>>>> proxy_intercept_errors on;
>>>>>> proxy_ssl_session_reuse off;
>>>>>> proxy_read_timeout 120;
>>>>>> proxy_connect_timeout 60;
>>>>>> proxy_send_timeout 120;
>>>>>> client_body_buffer_size 32m;
>>>>>>
>>>>>>
>>>>>>
>>>>> This setting allocates 32M buffer for each request with body.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> client_header_buffer_size 64k;
>>>>>> large_client_header_buffers 16 64k;
>>>>>> client_max_body_size 16m;
>>>>>>
>>>>>>
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:80;
>>>>>> location /
>>>>>> {
>>>>>>
>>>>>> proxy_pass http://3.4.5.6;
>>>>>> proxy_redirect http://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> }
>>>>>> }
>>>>>> server
>>>>>> {
>>>>>> listen 1.2.3.4:443;
>>>>>>
>>>>>> ssl on;
>>>>>> ssl_certificate
>>>>>> /usr/local/nginx/conf/whatever.com.crt;
>>>>>> ssl_certificate_key
>>>>>> /usr/local/nginx/conf/whatever.com.key;
>>>>>> ssl_protocols SSLv3;
>>>>>> ssl_ciphers HIGH:!ADH;
>>>>>> ssl_prefer_server_ciphers on;
>>>>>> location /
>>>>>> {
>>>>>> proxy_pass https://3.4.5.6;
>>>>>> proxy_redirect https://3.4.5.6/
>>>>>> http://$http_host/;
>>>>>> proxy_redirect default;
>>>>>> proxy_set_header Host
>>>>>> $host; ##Forwards host along
>>>>>> proxy_set_header X-Real-IP
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-Forwarded-For
>>>>>> $remote_addr; ##Sends realip to customer svr
>>>>>> proxy_set_header X-FORWARDED_PROTO https;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> And these server entries repeated about 60 or so times and that's it.
>>>>>> When it was around 40 we never had a memleak issue.
>>>>>>
>>>>>> This is on Linux, kernel is 2.6.25
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> GloboTech Communications
>>>>>> Phone: 1-514-907-0050 x 215
>>>>>> Toll Free: 1-(888)-GTCOMM1
>>>>>> Fax: 1-(514)-907-0750
>>>>>> paul@gtcomm.net
>>>>>> http://www.gtcomm.net
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> GloboTech Communications
>>>> Phone: 1-514-907-0050 x 215
>>>> Toll Free: 1-(888)-GTCOMM1
>>>> Fax: 1-(514)-907-0750
>>>> paul@gtcomm.net
>>>> http://www.gtcomm.net
>>>>
>>>>
>>>>
>>>
>>>

--
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul@gtcomm.net
http://www.gtcomm.net

Re: memcached problems...

"swf and flv files into memcached which should be possible for my understanding."

are you sure?

On Mon, Aug 31, 2009 at 10:34 PM, JG <nginx-forum@nginx.us> wrote:
normally, i whould agree. but in this special case the system has trouble to handle the high io load due to really lots of files (about 8 million.... i know... crazy, not my content).

to get a workaround for the hd io load, i just wanted to get nginx to load the swf and flv files into memcached which should be possible for my understanding.

i think i found one of my problems, the missing fallback server if the content isnt already in memcached.

but now i get 502 bad gateway error messages, and such strange error log entries

2009/08/31 23:21:17  30612#0: 8192 worker_connections is not enough while accepting new connection on 0.0.0.0:81

? no connections yet ... no production

but anyway, i trimmed down my config to a minimum for testing


   server {
       listen       81;
       server_name  _;

       #access_log  logs/host.access.log  main;

       location / {
           set $memcached_key  $uri;
           memcached_pass      127.0.0.1:11211;
           error_page 404 502  = /fallback;
#           default_type        text/html;

           root   /usr/share/nginx/html;
           index  index.html index.htm;
       }

       location /fallback {
           proxy_pass http://127.0.0.1:81;
       }

       location ~ \.php$ {
           set $memcached_key  $uri;
           memcached_pass      127.0.0.1:11211;

           root           /home/www/htdocs;
           fastcgi_pass   127.0.0.1:9000;
           fastcgi_index  index.php;
           fastcgi_param  SCRIPT_FILENAME  /home/www/htdocs$fastcgi_script_name;
           include        fastcgi_params;
       }



PHP works well, but it doesnt seem to get stored in memcached, why? and, for example if i try to download a zip file i get this in the error log

2009/08/31 23:30:16  30815#0: *12285 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /koch/_banner/paldauer/paldauer.zip HTTP/1.0", upstream: "http://127.0.0.1:81/yes/_this/doesnt/work.zip", host: "127.0.0.1:81", referrer: "http://ftp.nastyhost.de:81/koch/_banner/paldauer/index.php"

please igor, anyhow if you got an idea how i can get nginx to use memcached to store content...

greetings & thanks

juergen

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,5434,5452#msg-5452





--
the sun shines for all

Re: memcached problems...

normally, i whould agree. but in this special case the system has trouble to handle the high io load due to really lots of files (about 8 million.... i know... crazy, not my content).

to get a workaround for the hd io load, i just wanted to get nginx to load the swf and flv files into memcached which should be possible for my understanding.

i think i found one of my problems, the missing fallback server if the content isnt already in memcached.

but now i get 502 bad gateway error messages, and such strange error log entries

2009/08/31 23:21:17 30612#0: 8192 worker_connections is not enough while accepting new connection on 0.0.0.0:81

? no connections yet ... no production

but anyway, i trimmed down my config to a minimum for testing


server {
listen 81;
server_name _;

#access_log logs/host.access.log main;

location / {
set $memcached_key $uri;
memcached_pass 127.0.0.1:11211;
error_page 404 502 = /fallback;
# default_type text/html;

root /usr/share/nginx/html;
index index.html index.htm;
}

location /fallback {
proxy_pass http://127.0.0.1:81;
}

location ~ \.php$ {
set $memcached_key $uri;
memcached_pass 127.0.0.1:11211;

root /home/www/htdocs;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /home/www/htdocs$fastcgi_script_name;
include fastcgi_params;
}

PHP works well, but it doesnt seem to get stored in memcached, why? and, for example if i try to download a zip file i get this in the error log

2009/08/31 23:30:16 30815#0: *12285 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /koch/_banner/paldauer/paldauer.zip HTTP/1.0", upstream: "http://127.0.0.1:81/yes/_this/doesnt/work.zip", host: "127.0.0.1:81", referrer: "http://ftp.nastyhost.de:81/koch/_banner/paldauer/index.php"

please igor, anyhow if you got an idea how i can get nginx to use memcached to store content...

greetings & thanks

juergen

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,5434,5452#msg-5452

Re: php, $_SERVER & server_name

On Mon, 2009-08-31 at 14:07 -0700, Michael Shadle wrote:
> $host is the same as $http_host?

Yes, because $http_HEADER is available for *all* headers, so even though
there's already a $host variable, it's also available via the
generalized $http_host variable:

http://wiki.nginx.org/NginxHttpCoreModule#.24host
http://wiki.nginx.org/NginxHttpCoreModule#.24http_HEADER


> fastcgi_param SERVER_NAME $http_host;
>
> i've had that in there forever.
>
> just like $uri is short for $request_uri right?

No, $uri and $request_uri are not always equal:

http://wiki.nginx.org/NginxHttpCoreModule#.24request_uri
http://wiki.nginx.org/NginxHttpCoreModule#.24uri

In short, $uri is the *current* URI (after rewrites, redirects, etc),
$request_uri is the *original* requested URI.

Regards,
Cliff

> this should be documented better somewhere
>
> 2009/8/31 Igor Sysoev <is@rambler-co.ru>:
> > On Mon, Aug 31, 2009 at 03:58:28PM -0500, AMP Admin wrote:
> >
> >> I'm trying to list a bunch of domains under server_name. I'm doing this
> >> because these domains generate dynamic content based on their name. it
> >> seems to work fine except for one thing... $_SERVER['SERVER_NAME'] just
> >> displays the first server_name entry.
> >>
> >> So say we have the following:
> >>
> >> server_name *.example1.com *.example2.com *.example3.com;
> >>
> >> echo $_SERVER['SERVER_NAME']; displays *.example1.com regardless which
> >> domain was entered into the browser.
> >
> > You need to change
> >
> > -fastcgi_param SERVER_NAME $server_name;
> > +fastcgi_param SERVER_NAME $host;
> >
> > in fastcgi_params.
> >
> >
> > --
> > Igor Sysoev
> > http://sysoev.ru/en/
> >
> >
>
--
http://www.google.com/search?q=vonage+sucks

Re: php, $_SERVER & server_name

$host is the same as $http_host?

fastcgi_param SERVER_NAME $http_host;

i've had that in there forever.

just like $uri is short for $request_uri right?

this should be documented better somewhere

2009/8/31 Igor Sysoev <is@rambler-co.ru>:
> On Mon, Aug 31, 2009 at 03:58:28PM -0500, AMP Admin wrote:
>
>> I'm trying to list a bunch of domains under server_name.  I'm doing this
>> because these domains generate dynamic content based on their name.  it
>> seems to work fine except for one thing... $_SERVER['SERVER_NAME'] just
>> displays the first server_name entry.
>>
>> So say we have the following:
>>
>> server_name   *.example1.com *.example2.com *.example3.com;
>>
>> echo $_SERVER['SERVER_NAME']; displays *.example1.com regardless which
>> domain was entered into the browser.
>
> You need to change
>
> -fastcgi_param  SERVER_NAME        $server_name;
> +fastcgi_param  SERVER_NAME        $host;
>
> in fastcgi_params.
>
>
> --
> Igor Sysoev
> http://sysoev.ru/en/
>
>

Re: Weird Memleak problem

On Mon, 2009-08-31 at 16:52 -0400, Paul wrote:
> Had some issues with people uploading/downloading files before on 0.6
> and the buffers fixed it..

What sort of issues? Timeouts or "client body too large"? I regularly
upload 10MB+ CSV files to a proxied application with the following:

client_max_body_size 20m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;


Cliff

> What would you suggest I change the config to? I will try and let you
> know if I still see the problem.
> Thank you.
>
>
> Igor Sysoev wrote:
> > On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
> >
> >
> >> I know, but this problem has never occured until recently.. Once the
> >> request is done, it should remove the memory allocation, but it looks
> >> like maybe it isn't?
> >>
> >
> > No. These workers run since Aug 7 and handle up to 3,800r/s:
> >
> > ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
> > PID PPID %CPU VSZ WCHAN STARTED COMMAND
> > 42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
> > 42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
> > 42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
> > 51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng
> >
> >
> >> The only difference in a month ago and now is that we have more server
> >> entries, and more requests per second. It used to not even use a gig of
> >> ram doing 200 requests/sec and now it keeps using more and more ram
> >> until it fills the entire ram and swap and errors out 8gb+ ..
> >> What would you suggest?
> >>
> >
> > Just 250 simultaneous proxied connections take 8G. And 250 connections
> > are too little in modern world. Why at all you have set up so huge buffers ?
> >
> >
> >> Igor Sysoev wrote:
> >>
> >>> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
> >>>
> >>>
> >>>
> >>>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
> >>>> memleak issue.
> >>>> What happens is as soon as I start nginx, it starts using ram and
> >>>> continues to use more and more and more over a 16 hour period is
> >>>> consumes 8gb ram and all the swap and then errors out because it cant
> >>>> use any more.
> >>>> This never used to happen until recently and the only difference at all
> >>>> in the config is more server entries.
> >>>>
> >>>> here's the config:
> >>>>
> >>>> user www www;
> >>>>
> >>>> worker_processes 16;
> >>>> error_log logs/error.log;
> >>>> worker_rlimit_nofile 65000;
> >>>>
> >>>> events
> >>>> {
> >>>>
> >>>> worker_connections 40000;
> >>>> }
> >>>>
> >>>> ####### HTTP SETTING
> >>>> http
> >>>> {
> >>>> access_log off;
> >>>> log_format alot '$remote_addr - $remote_user [$time_local] '
> >>>> '"$request" $status $body_bytes_sent '
> >>>> '"$http_referer" "$http_user_agent" "$http_accept"
> >>>> $connection';
> >>>> sendfile on;
> >>>> tcp_nopush on;
> >>>> tcp_nodelay on;
> >>>> keepalive_timeout 0;
> >>>> output_buffers 16 128k;
> >>>> server_tokens off;
> >>>> ssl_verify_client off;
> >>>> ssl_session_timeout 10m;
> >>>> # ssl_session_cache shared:SSL:500000;
> >>>> include /usr/local/nginx/conf/mime.types;
> >>>> default_type application/octet-stream;
> >>>>
> >>>> # cache_max_size 24;
> >>>>
> >>>> gzip on;
> >>>> gzip_min_length 512;
> >>>> gzip_buffers 64 32k;
> >>>> gzip_types text/plain text/html text/xhtml text/css text/js;
> >>>>
> >>>> proxy_buffering on;
> >>>> proxy_buffer_size 32m;
> >>>> proxy_buffers 16 32m;
> >>>>
> >>>>
> >>> These settings allocate 32M buffer for each proxied request.
> >>>
> >>>
> >>>
> >>>> proxy_busy_buffers_size 64m;
> >>>> proxy_temp_file_write_size 2048m;
> >>>> proxy_intercept_errors on;
> >>>> proxy_ssl_session_reuse off;
> >>>> proxy_read_timeout 120;
> >>>> proxy_connect_timeout 60;
> >>>> proxy_send_timeout 120;
> >>>> client_body_buffer_size 32m;
> >>>>
> >>>>
> >>> This setting allocates 32M buffer for each request with body.
> >>>
> >>>
> >>>
> >>>> client_header_buffer_size 64k;
> >>>> large_client_header_buffers 16 64k;
> >>>> client_max_body_size 16m;
> >>>>
> >>>>
> >>>> server
> >>>> {
> >>>> listen 1.2.3.4:80;
> >>>> location /
> >>>> {
> >>>>
> >>>> proxy_pass http://3.4.5.6;
> >>>> proxy_redirect http://3.4.5.6/
> >>>> http://$http_host/;
> >>>> proxy_redirect default;
> >>>> proxy_set_header Host
> >>>> $host; ##Forwards host along
> >>>> proxy_set_header X-Real-IP
> >>>> $remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-Forwarded-For
> >>>> $remote_addr; ##Sends realip to customer svr
> >>>> }
> >>>> }
> >>>> server
> >>>> {
> >>>> listen 1.2.3.4:443;
> >>>>
> >>>> ssl on;
> >>>> ssl_certificate
> >>>> /usr/local/nginx/conf/whatever.com.crt;
> >>>> ssl_certificate_key
> >>>> /usr/local/nginx/conf/whatever.com.key;
> >>>> ssl_protocols SSLv3;
> >>>> ssl_ciphers HIGH:!ADH;
> >>>> ssl_prefer_server_ciphers on;
> >>>> location /
> >>>> {
> >>>> proxy_pass https://3.4.5.6;
> >>>> proxy_redirect https://3.4.5.6/
> >>>> http://$http_host/;
> >>>> proxy_redirect default;
> >>>> proxy_set_header Host
> >>>> $host; ##Forwards host along
> >>>> proxy_set_header X-Real-IP
> >>>> $remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-Forwarded-For
> >>>> $remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-FORWARDED_PROTO https;
> >>>> }
> >>>> }
> >>>>
> >>>> And these server entries repeated about 60 or so times and that's it.
> >>>> When it was around 40 we never had a memleak issue.
> >>>>
> >>>> This is on Linux, kernel is 2.6.25
> >>>>
> >>>> Thanks
> >>>>
> >>>>
> >>>> --
> >>>> GloboTech Communications
> >>>> Phone: 1-514-907-0050 x 215
> >>>> Toll Free: 1-(888)-GTCOMM1
> >>>> Fax: 1-(514)-907-0750
> >>>> paul@gtcomm.net
> >>>> http://www.gtcomm.net
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >> --
> >> GloboTech Communications
> >> Phone: 1-514-907-0050 x 215
> >> Toll Free: 1-(888)-GTCOMM1
> >> Fax: 1-(514)-907-0750
> >> paul@gtcomm.net
> >> http://www.gtcomm.net
> >>
> >>
> >
> >
>
--
http://www.google.com/search?q=vonage+sucks

RE: php, $_SERVER & server_name

Shoot! I sent that a second too soon. Found the following and it worked...

fastcgi_param SERVER_NAME $host;

Regards,

-Team AMP
http://www.ampprod.com


-----Original Message-----
From: AMP Admin [mailto:admin@ampprod.com]
Sent: Monday, August 31, 2009 3:58 PM
To: 'nginx@sysoev.ru'
Subject: php, $_SERVER & server_name

I'm trying to list a bunch of domains under server_name. I'm doing this
because these domains generate dynamic content based on their name. it
seems to work fine except for one thing... $_SERVER['SERVER_NAME'] just
displays the first server_name entry.


So say we have the following:

server_name *.example1.com *.example2.com *.example3.com;

echo $_SERVER['SERVER_NAME']; displays *.example1.com regardless which
domain was entered into the browser.

Regards,

-Team AMP
http://www.ampprod.com

Re: php, $_SERVER & server_name

On Mon, Aug 31, 2009 at 03:58:28PM -0500, AMP Admin wrote:

> I'm trying to list a bunch of domains under server_name. I'm doing this
> because these domains generate dynamic content based on their name. it
> seems to work fine except for one thing... $_SERVER['SERVER_NAME'] just
> displays the first server_name entry.
>
> So say we have the following:
>
> server_name *.example1.com *.example2.com *.example3.com;
>
> echo $_SERVER['SERVER_NAME']; displays *.example1.com regardless which
> domain was entered into the browser.

You need to change

-fastcgi_param SERVER_NAME $server_name;
+fastcgi_param SERVER_NAME $host;

in fastcgi_params.


--
Igor Sysoev
http://sysoev.ru/en/

Re: Weird Memleak problem

On Mon, 2009-08-31 at 16:17 -0400, Paul wrote:
> I know, but this problem has never occured until recently.. Once the
> request is done, it should remove the memory allocation, but it looks
> like maybe it isn't?
> The only difference in a month ago and now is that we have more server
> entries, and more requests per second. It used to not even use a gig of
> ram doing 200 requests/sec and now it keeps using more and more ram
> until it fills the entire ram and swap and errors out 8gb+ ..
> What would you suggest?

There's many more factors than simple requests per second and number of
server entries:

1) the speed of the proxied backends: if the backend is slow, then that
memory will be held onto for much longer.

2) the speed of clients: slow clients can cause the request processing
cycle to be slowed down, causing memory to be held onto for longer.

3) resource contention: more backends means slower overall performance
as the OS can't cache as efficiently (both CPU and disk), must handle
more context switches, more disk seeks, etc.

In short, one resource shortage (say CPU or I/O) can cause systemic
resource shortages as other resources can't be freed in a timely fashion
(in turn causing even more resource shortages), so the problem quickly
spirals out of control.

These problems are pretty much unavoidable on a single server, the only
question is at what point will you encounter them. Basically there's a
breaking point in the scalability of any particular system. Tuning is
the art of pushing that breaking point out as far as possible. Because
you've got excessively large buffers, you've almost certainly brought
that breaking point upon yourself much earlier than need be.

As Igor mentions, 32MB seems really, really excessive (an application
that generates responses of that size calls for one or more dedicated
servers and brain surgery for the developer). Maybe you should try
something closer to 8 *kilobytes* and see if that addresses your issues.

Regards,
Cliff


>
>
> Igor Sysoev wrote:
> > On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
> >
> >
> >> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
> >> memleak issue.
> >> What happens is as soon as I start nginx, it starts using ram and
> >> continues to use more and more and more over a 16 hour period is
> >> consumes 8gb ram and all the swap and then errors out because it cant
> >> use any more.
> >> This never used to happen until recently and the only difference at all
> >> in the config is more server entries.
> >>
> >> here's the config:
> >>
> >> user www www;
> >>
> >> worker_processes 16;
> >> error_log logs/error.log;
> >> worker_rlimit_nofile 65000;
> >>
> >> events
> >> {
> >>
> >> worker_connections 40000;
> >> }
> >>
> >> ####### HTTP SETTING
> >> http
> >> {
> >> access_log off;
> >> log_format alot '$remote_addr - $remote_user [$time_local] '
> >> '"$request" $status $body_bytes_sent '
> >> '"$http_referer" "$http_user_agent" "$http_accept"
> >> $connection';
> >> sendfile on;
> >> tcp_nopush on;
> >> tcp_nodelay on;
> >> keepalive_timeout 0;
> >> output_buffers 16 128k;
> >> server_tokens off;
> >> ssl_verify_client off;
> >> ssl_session_timeout 10m;
> >> # ssl_session_cache shared:SSL:500000;
> >> include /usr/local/nginx/conf/mime.types;
> >> default_type application/octet-stream;
> >>
> >> # cache_max_size 24;
> >>
> >> gzip on;
> >> gzip_min_length 512;
> >> gzip_buffers 64 32k;
> >> gzip_types text/plain text/html text/xhtml text/css text/js;
> >>
> >> proxy_buffering on;
> >> proxy_buffer_size 32m;
> >> proxy_buffers 16 32m;
> >>
> >
> > These settings allocate 32M buffer for each proxied request.
> >
> >
> >> proxy_busy_buffers_size 64m;
> >> proxy_temp_file_write_size 2048m;
> >> proxy_intercept_errors on;
> >> proxy_ssl_session_reuse off;
> >> proxy_read_timeout 120;
> >> proxy_connect_timeout 60;
> >> proxy_send_timeout 120;
> >> client_body_buffer_size 32m;
> >>
> >
> > This setting allocates 32M buffer for each request with body.
> >
> >
> >> client_header_buffer_size 64k;
> >> large_client_header_buffers 16 64k;
> >> client_max_body_size 16m;
> >>
> >>
> >> server
> >> {
> >> listen 1.2.3.4:80;
> >> location /
> >> {
> >>
> >> proxy_pass http://3.4.5.6;
> >> proxy_redirect http://3.4.5.6/
> >> http://$http_host/;
> >> proxy_redirect default;
> >> proxy_set_header Host
> >> $host; ##Forwards host along
> >> proxy_set_header X-Real-IP
> >> $remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-Forwarded-For
> >> $remote_addr; ##Sends realip to customer svr
> >> }
> >> }
> >> server
> >> {
> >> listen 1.2.3.4:443;
> >>
> >> ssl on;
> >> ssl_certificate
> >> /usr/local/nginx/conf/whatever.com.crt;
> >> ssl_certificate_key
> >> /usr/local/nginx/conf/whatever.com.key;
> >> ssl_protocols SSLv3;
> >> ssl_ciphers HIGH:!ADH;
> >> ssl_prefer_server_ciphers on;
> >> location /
> >> {
> >> proxy_pass https://3.4.5.6;
> >> proxy_redirect https://3.4.5.6/
> >> http://$http_host/;
> >> proxy_redirect default;
> >> proxy_set_header Host
> >> $host; ##Forwards host along
> >> proxy_set_header X-Real-IP
> >> $remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-Forwarded-For
> >> $remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-FORWARDED_PROTO https;
> >> }
> >> }
> >>
> >> And these server entries repeated about 60 or so times and that's it.
> >> When it was around 40 we never had a memleak issue.
> >>
> >> This is on Linux, kernel is 2.6.25
> >>
> >> Thanks
> >>
> >>
> >> --
> >> GloboTech Communications
> >> Phone: 1-514-907-0050 x 215
> >> Toll Free: 1-(888)-GTCOMM1
> >> Fax: 1-(514)-907-0750
> >> paul@gtcomm.net
> >> http://www.gtcomm.net
> >>
> >>
> >
> >
>
--
http://www.google.com/search?q=vonage+sucks

Re: Weird Memleak problem

On Mon, Aug 31, 2009 at 04:52:51PM -0400, Paul wrote:

> Had some issues with people uploading/downloading files before on 0.6
> and the buffers fixed it..

These issues should be fixed in other way.

> What would you suggest I change the config to? I will try and let you
> know if I still see the problem.
> Thank you.

- proxy_buffer_size 32m;
- proxy_buffers 16 32m;
- proxy_busy_buffers_size 64m;
- proxy_temp_file_write_size 2048m;
- client_body_buffer_size 32m;

+ proxy_buffer_size 32k;
+ proxy_buffers 16 32k;
+ client_body_buffer_size 32k;

> Igor Sysoev wrote:
> >On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
> >
> >
> >>I know, but this problem has never occured until recently.. Once the
> >>request is done, it should remove the memory allocation, but it looks
> >>like maybe it isn't?
> >>
> >
> >No. These workers run since Aug 7 and handle up to 3,800r/s:
> >
> >ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
> > PID PPID %CPU VSZ WCHAN STARTED COMMAND
> >42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
> >42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
> >42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
> >51429 1 0.0 291556 pause 28Jul09 nginx: master process
> >/usr/local/nginx/ng
> >
> >
> >>The only difference in a month ago and now is that we have more server
> >>entries, and more requests per second. It used to not even use a gig of
> >>ram doing 200 requests/sec and now it keeps using more and more ram
> >>until it fills the entire ram and swap and errors out 8gb+ ..
> >>What would you suggest?
> >>
> >
> >Just 250 simultaneous proxied connections take 8G. And 250 connections
> >are too little in modern world. Why at all you have set up so huge buffers
> >?
> >
> >
> >>Igor Sysoev wrote:
> >>
> >>>On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
> >>>
> >>>
> >>>
> >>>>I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
> >>>>memleak issue.
> >>>>What happens is as soon as I start nginx, it starts using ram and
> >>>>continues to use more and more and more over a 16 hour period is
> >>>>consumes 8gb ram and all the swap and then errors out because it cant
> >>>>use any more.
> >>>>This never used to happen until recently and the only difference at all
> >>>>in the config is more server entries.
> >>>>
> >>>>here's the config:
> >>>>
> >>>>user www www;
> >>>>
> >>>>worker_processes 16;
> >>>>error_log logs/error.log;
> >>>>worker_rlimit_nofile 65000;
> >>>>
> >>>>events
> >>>>{
> >>>>
> >>>> worker_connections 40000;
> >>>>}
> >>>>
> >>>>####### HTTP SETTING
> >>>>http
> >>>>{
> >>>> access_log off;
> >>>> log_format alot '$remote_addr - $remote_user [$time_local] '
> >>>> '"$request" $status $body_bytes_sent '
> >>>> '"$http_referer" "$http_user_agent" "$http_accept"
> >>>>$connection';
> >>>> sendfile on;
> >>>> tcp_nopush on;
> >>>> tcp_nodelay on;
> >>>> keepalive_timeout 0;
> >>>> output_buffers 16 128k;
> >>>> server_tokens off;
> >>>> ssl_verify_client off;
> >>>> ssl_session_timeout 10m;
> >>>># ssl_session_cache shared:SSL:500000;
> >>>>include /usr/local/nginx/conf/mime.types;
> >>>>default_type application/octet-stream;
> >>>>
> >>>># cache_max_size 24;
> >>>>
> >>>> gzip on;
> >>>> gzip_min_length 512;
> >>>> gzip_buffers 64 32k;
> >>>> gzip_types text/plain text/html text/xhtml text/css text/js;
> >>>>
> >>>> proxy_buffering on;
> >>>> proxy_buffer_size 32m;
> >>>> proxy_buffers 16 32m;
> >>>>
> >>>>
> >>>These settings allocate 32M buffer for each proxied request.
> >>>
> >>>
> >>>
> >>>> proxy_busy_buffers_size 64m;
> >>>> proxy_temp_file_write_size 2048m;
> >>>> proxy_intercept_errors on;
> >>>> proxy_ssl_session_reuse off;
> >>>> proxy_read_timeout 120;
> >>>> proxy_connect_timeout 60;
> >>>> proxy_send_timeout 120;
> >>>> client_body_buffer_size 32m;
> >>>>
> >>>>
> >>>This setting allocates 32M buffer for each request with body.
> >>>
> >>>
> >>>
> >>>> client_header_buffer_size 64k;
> >>>> large_client_header_buffers 16 64k;
> >>>> client_max_body_size 16m;
> >>>>
> >>>>
> >>>> server
> >>>> {
> >>>> listen 1.2.3.4:80;
> >>>> location /
> >>>> {
> >>>>
> >>>> proxy_pass http://3.4.5.6;
> >>>> proxy_redirect http://3.4.5.6/
> >>>>http://$http_host/;
> >>>> proxy_redirect default;
> >>>> proxy_set_header Host
> >>>>$host; ##Forwards host along
> >>>> proxy_set_header X-Real-IP
> >>>>$remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-Forwarded-For
> >>>>$remote_addr; ##Sends realip to customer svr
> >>>> }
> >>>> }
> >>>> server
> >>>> {
> >>>> listen 1.2.3.4:443;
> >>>>
> >>>> ssl on;
> >>>> ssl_certificate
> >>>>/usr/local/nginx/conf/whatever.com.crt;
> >>>> ssl_certificate_key
> >>>>/usr/local/nginx/conf/whatever.com.key;
> >>>> ssl_protocols SSLv3;
> >>>> ssl_ciphers HIGH:!ADH;
> >>>> ssl_prefer_server_ciphers on;
> >>>> location /
> >>>> {
> >>>> proxy_pass https://3.4.5.6;
> >>>> proxy_redirect https://3.4.5.6/
> >>>>http://$http_host/;
> >>>> proxy_redirect default;
> >>>> proxy_set_header Host
> >>>>$host; ##Forwards host along
> >>>> proxy_set_header X-Real-IP
> >>>>$remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-Forwarded-For
> >>>>$remote_addr; ##Sends realip to customer svr
> >>>> proxy_set_header X-FORWARDED_PROTO https;
> >>>> }
> >>>> }
> >>>>
> >>>>And these server entries repeated about 60 or so times and that's it.
> >>>>When it was around 40 we never had a memleak issue.
> >>>>
> >>>>This is on Linux, kernel is 2.6.25
> >>>>
> >>>>Thanks
> >>>>
> >>>>
> >>>>--
> >>>>GloboTech Communications
> >>>>Phone: 1-514-907-0050 x 215
> >>>>Toll Free: 1-(888)-GTCOMM1
> >>>>Fax: 1-(514)-907-0750
> >>>>paul@gtcomm.net
> >>>>http://www.gtcomm.net
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>--
> >>GloboTech Communications
> >>Phone: 1-514-907-0050 x 215
> >>Toll Free: 1-(888)-GTCOMM1
> >>Fax: 1-(514)-907-0750
> >>paul@gtcomm.net
> >>http://www.gtcomm.net
> >>
> >>
> >
> >
>
> --
> GloboTech Communications
> Phone: 1-514-907-0050 x 215
> Toll Free: 1-(888)-GTCOMM1
> Fax: 1-(514)-907-0750
> paul@gtcomm.net
> http://www.gtcomm.net
>

--
Igor Sysoev
http://sysoev.ru/en/

php, $_SERVER & server_name

I'm trying to list a bunch of domains under server_name. I'm doing this
because these domains generate dynamic content based on their name. it
seems to work fine except for one thing... $_SERVER['SERVER_NAME'] just
displays the first server_name entry.

So say we have the following:

server_name *.example1.com *.example2.com *.example3.com;

echo $_SERVER['SERVER_NAME']; displays *.example1.com regardless which
domain was entered into the browser.

Regards,

-Team AMP
http://www.ampprod.com

Re: Weird Memleak problem

Had some issues with people uploading/downloading files before on 0.6
and the buffers fixed it..
What would you suggest I change the config to? I will try and let you
know if I still see the problem.
Thank you.


Igor Sysoev wrote:
> On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:
>
>
>> I know, but this problem has never occured until recently.. Once the
>> request is done, it should remove the memory allocation, but it looks
>> like maybe it isn't?
>>
>
> No. These workers run since Aug 7 and handle up to 3,800r/s:
>
> ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
> PID PPID %CPU VSZ WCHAN STARTED COMMAND
> 42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
> 42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
> 42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
> 51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng
>
>
>> The only difference in a month ago and now is that we have more server
>> entries, and more requests per second. It used to not even use a gig of
>> ram doing 200 requests/sec and now it keeps using more and more ram
>> until it fills the entire ram and swap and errors out 8gb+ ..
>> What would you suggest?
>>
>
> Just 250 simultaneous proxied connections take 8G. And 250 connections
> are too little in modern world. Why at all you have set up so huge buffers ?
>
>
>> Igor Sysoev wrote:
>>
>>> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
>>>
>>>
>>>
>>>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
>>>> memleak issue.
>>>> What happens is as soon as I start nginx, it starts using ram and
>>>> continues to use more and more and more over a 16 hour period is
>>>> consumes 8gb ram and all the swap and then errors out because it cant
>>>> use any more.
>>>> This never used to happen until recently and the only difference at all
>>>> in the config is more server entries.
>>>>
>>>> here's the config:
>>>>
>>>> user www www;
>>>>
>>>> worker_processes 16;
>>>> error_log logs/error.log;
>>>> worker_rlimit_nofile 65000;
>>>>
>>>> events
>>>> {
>>>>
>>>> worker_connections 40000;
>>>> }
>>>>
>>>> ####### HTTP SETTING
>>>> http
>>>> {
>>>> access_log off;
>>>> log_format alot '$remote_addr - $remote_user [$time_local] '
>>>> '"$request" $status $body_bytes_sent '
>>>> '"$http_referer" "$http_user_agent" "$http_accept"
>>>> $connection';
>>>> sendfile on;
>>>> tcp_nopush on;
>>>> tcp_nodelay on;
>>>> keepalive_timeout 0;
>>>> output_buffers 16 128k;
>>>> server_tokens off;
>>>> ssl_verify_client off;
>>>> ssl_session_timeout 10m;
>>>> # ssl_session_cache shared:SSL:500000;
>>>> include /usr/local/nginx/conf/mime.types;
>>>> default_type application/octet-stream;
>>>>
>>>> # cache_max_size 24;
>>>>
>>>> gzip on;
>>>> gzip_min_length 512;
>>>> gzip_buffers 64 32k;
>>>> gzip_types text/plain text/html text/xhtml text/css text/js;
>>>>
>>>> proxy_buffering on;
>>>> proxy_buffer_size 32m;
>>>> proxy_buffers 16 32m;
>>>>
>>>>
>>> These settings allocate 32M buffer for each proxied request.
>>>
>>>
>>>
>>>> proxy_busy_buffers_size 64m;
>>>> proxy_temp_file_write_size 2048m;
>>>> proxy_intercept_errors on;
>>>> proxy_ssl_session_reuse off;
>>>> proxy_read_timeout 120;
>>>> proxy_connect_timeout 60;
>>>> proxy_send_timeout 120;
>>>> client_body_buffer_size 32m;
>>>>
>>>>
>>> This setting allocates 32M buffer for each request with body.
>>>
>>>
>>>
>>>> client_header_buffer_size 64k;
>>>> large_client_header_buffers 16 64k;
>>>> client_max_body_size 16m;
>>>>
>>>>
>>>> server
>>>> {
>>>> listen 1.2.3.4:80;
>>>> location /
>>>> {
>>>>
>>>> proxy_pass http://3.4.5.6;
>>>> proxy_redirect http://3.4.5.6/
>>>> http://$http_host/;
>>>> proxy_redirect default;
>>>> proxy_set_header Host
>>>> $host; ##Forwards host along
>>>> proxy_set_header X-Real-IP
>>>> $remote_addr; ##Sends realip to customer svr
>>>> proxy_set_header X-Forwarded-For
>>>> $remote_addr; ##Sends realip to customer svr
>>>> }
>>>> }
>>>> server
>>>> {
>>>> listen 1.2.3.4:443;
>>>>
>>>> ssl on;
>>>> ssl_certificate
>>>> /usr/local/nginx/conf/whatever.com.crt;
>>>> ssl_certificate_key
>>>> /usr/local/nginx/conf/whatever.com.key;
>>>> ssl_protocols SSLv3;
>>>> ssl_ciphers HIGH:!ADH;
>>>> ssl_prefer_server_ciphers on;
>>>> location /
>>>> {
>>>> proxy_pass https://3.4.5.6;
>>>> proxy_redirect https://3.4.5.6/
>>>> http://$http_host/;
>>>> proxy_redirect default;
>>>> proxy_set_header Host
>>>> $host; ##Forwards host along
>>>> proxy_set_header X-Real-IP
>>>> $remote_addr; ##Sends realip to customer svr
>>>> proxy_set_header X-Forwarded-For
>>>> $remote_addr; ##Sends realip to customer svr
>>>> proxy_set_header X-FORWARDED_PROTO https;
>>>> }
>>>> }
>>>>
>>>> And these server entries repeated about 60 or so times and that's it.
>>>> When it was around 40 we never had a memleak issue.
>>>>
>>>> This is on Linux, kernel is 2.6.25
>>>>
>>>> Thanks
>>>>
>>>>
>>>> --
>>>> GloboTech Communications
>>>> Phone: 1-514-907-0050 x 215
>>>> Toll Free: 1-(888)-GTCOMM1
>>>> Fax: 1-(514)-907-0750
>>>> paul@gtcomm.net
>>>> http://www.gtcomm.net
>>>>
>>>>
>>>>
>>>
>>>
>> --
>> GloboTech Communications
>> Phone: 1-514-907-0050 x 215
>> Toll Free: 1-(888)-GTCOMM1
>> Fax: 1-(514)-907-0750
>> paul@gtcomm.net
>> http://www.gtcomm.net
>>
>>
>
>

--
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul@gtcomm.net
http://www.gtcomm.net

Re: Weird Memleak problem

On Mon, Aug 31, 2009 at 04:17:03PM -0400, Paul wrote:

> I know, but this problem has never occured until recently.. Once the
> request is done, it should remove the memory allocation, but it looks
> like maybe it isn't?

No. These workers run since Aug 7 and handle up to 3,800r/s:

ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'
PID PPID %CPU VSZ WCHAN STARTED COMMAND
42412 51429 16.7 372452 kqread 7Aug09 nginx: worker process (nginx)
42413 51429 18.2 372452 kqread 7Aug09 nginx: worker process (nginx)
42414 51429 0.0 291556 kqread 7Aug09 nginx: cache manager process (nginx)
51429 1 0.0 291556 pause 28Jul09 nginx: master process /usr/local/nginx/ng

> The only difference in a month ago and now is that we have more server
> entries, and more requests per second. It used to not even use a gig of
> ram doing 200 requests/sec and now it keeps using more and more ram
> until it fills the entire ram and swap and errors out 8gb+ ..
> What would you suggest?

Just 250 simultaneous proxied connections take 8G. And 250 connections
are too little in modern world. Why at all you have set up so huge buffers ?

> Igor Sysoev wrote:
> >On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
> >
> >
> >>I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
> >>memleak issue.
> >>What happens is as soon as I start nginx, it starts using ram and
> >>continues to use more and more and more over a 16 hour period is
> >>consumes 8gb ram and all the swap and then errors out because it cant
> >>use any more.
> >>This never used to happen until recently and the only difference at all
> >>in the config is more server entries.
> >>
> >>here's the config:
> >>
> >>user www www;
> >>
> >>worker_processes 16;
> >>error_log logs/error.log;
> >>worker_rlimit_nofile 65000;
> >>
> >>events
> >>{
> >>
> >> worker_connections 40000;
> >>}
> >>
> >>####### HTTP SETTING
> >>http
> >>{
> >> access_log off;
> >> log_format alot '$remote_addr - $remote_user [$time_local] '
> >> '"$request" $status $body_bytes_sent '
> >> '"$http_referer" "$http_user_agent" "$http_accept"
> >>$connection';
> >> sendfile on;
> >> tcp_nopush on;
> >> tcp_nodelay on;
> >> keepalive_timeout 0;
> >> output_buffers 16 128k;
> >> server_tokens off;
> >> ssl_verify_client off;
> >> ssl_session_timeout 10m;
> >># ssl_session_cache shared:SSL:500000;
> >>include /usr/local/nginx/conf/mime.types;
> >>default_type application/octet-stream;
> >>
> >># cache_max_size 24;
> >>
> >> gzip on;
> >> gzip_min_length 512;
> >> gzip_buffers 64 32k;
> >> gzip_types text/plain text/html text/xhtml text/css text/js;
> >>
> >> proxy_buffering on;
> >> proxy_buffer_size 32m;
> >> proxy_buffers 16 32m;
> >>
> >
> >These settings allocate 32M buffer for each proxied request.
> >
> >
> >> proxy_busy_buffers_size 64m;
> >> proxy_temp_file_write_size 2048m;
> >> proxy_intercept_errors on;
> >> proxy_ssl_session_reuse off;
> >> proxy_read_timeout 120;
> >> proxy_connect_timeout 60;
> >> proxy_send_timeout 120;
> >> client_body_buffer_size 32m;
> >>
> >
> >This setting allocates 32M buffer for each request with body.
> >
> >
> >> client_header_buffer_size 64k;
> >> large_client_header_buffers 16 64k;
> >> client_max_body_size 16m;
> >>
> >>
> >> server
> >> {
> >> listen 1.2.3.4:80;
> >> location /
> >> {
> >>
> >> proxy_pass http://3.4.5.6;
> >> proxy_redirect http://3.4.5.6/
> >>http://$http_host/;
> >> proxy_redirect default;
> >> proxy_set_header Host
> >>$host; ##Forwards host along
> >> proxy_set_header X-Real-IP
> >>$remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-Forwarded-For
> >>$remote_addr; ##Sends realip to customer svr
> >> }
> >> }
> >> server
> >> {
> >> listen 1.2.3.4:443;
> >>
> >> ssl on;
> >> ssl_certificate
> >>/usr/local/nginx/conf/whatever.com.crt;
> >> ssl_certificate_key
> >>/usr/local/nginx/conf/whatever.com.key;
> >> ssl_protocols SSLv3;
> >> ssl_ciphers HIGH:!ADH;
> >> ssl_prefer_server_ciphers on;
> >> location /
> >> {
> >> proxy_pass https://3.4.5.6;
> >> proxy_redirect https://3.4.5.6/
> >>http://$http_host/;
> >> proxy_redirect default;
> >> proxy_set_header Host
> >>$host; ##Forwards host along
> >> proxy_set_header X-Real-IP
> >>$remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-Forwarded-For
> >>$remote_addr; ##Sends realip to customer svr
> >> proxy_set_header X-FORWARDED_PROTO https;
> >> }
> >> }
> >>
> >>And these server entries repeated about 60 or so times and that's it.
> >>When it was around 40 we never had a memleak issue.
> >>
> >>This is on Linux, kernel is 2.6.25
> >>
> >>Thanks
> >>
> >>
> >>--
> >>GloboTech Communications
> >>Phone: 1-514-907-0050 x 215
> >>Toll Free: 1-(888)-GTCOMM1
> >>Fax: 1-(514)-907-0750
> >>paul@gtcomm.net
> >>http://www.gtcomm.net
> >>
> >>
> >
> >
>
> --
> GloboTech Communications
> Phone: 1-514-907-0050 x 215
> Toll Free: 1-(888)-GTCOMM1
> Fax: 1-(514)-907-0750
> paul@gtcomm.net
> http://www.gtcomm.net
>

--
Igor Sysoev
http://sysoev.ru/en/

Re: Weird Memleak problem

I know, but this problem has never occured until recently.. Once the
request is done, it should remove the memory allocation, but it looks
like maybe it isn't?
The only difference in a month ago and now is that we have more server
entries, and more requests per second. It used to not even use a gig of
ram doing 200 requests/sec and now it keeps using more and more ram
until it fills the entire ram and swap and errors out 8gb+ ..
What would you suggest?

Igor Sysoev wrote:
> On Sun, Aug 30, 2009 at 09:45:55PM -0400, Paul wrote:
>
>
>> I had nginx 0.6.32 and just upgraded to 0.7.61 and still have the same
>> memleak issue.
>> What happens is as soon as I start nginx, it starts using ram and
>> continues to use more and more and more over a 16 hour period is
>> consumes 8gb ram and all the swap and then errors out because it cant
>> use any more.
>> This never used to happen until recently and the only difference at all
>> in the config is more server entries.
>>
>> here's the config:
>>
>> user www www;
>>
>> worker_processes 16;
>> error_log logs/error.log;
>> worker_rlimit_nofile 65000;
>>
>> events
>> {
>>
>> worker_connections 40000;
>> }
>>
>> ####### HTTP SETTING
>> http
>> {
>> access_log off;
>> log_format alot '$remote_addr - $remote_user [$time_local] '
>> '"$request" $status $body_bytes_sent '
>> '"$http_referer" "$http_user_agent" "$http_accept"
>> $connection';
>> sendfile on;
>> tcp_nopush on;
>> tcp_nodelay on;
>> keepalive_timeout 0;
>> output_buffers 16 128k;
>> server_tokens off;
>> ssl_verify_client off;
>> ssl_session_timeout 10m;
>> # ssl_session_cache shared:SSL:500000;
>> include /usr/local/nginx/conf/mime.types;
>> default_type application/octet-stream;
>>
>> # cache_max_size 24;
>>
>> gzip on;
>> gzip_min_length 512;
>> gzip_buffers 64 32k;
>> gzip_types text/plain text/html text/xhtml text/css text/js;
>>
>> proxy_buffering on;
>> proxy_buffer_size 32m;
>> proxy_buffers 16 32m;
>>
>
> These settings allocate 32M buffer for each proxied request.
>
>
>> proxy_busy_buffers_size 64m;
>> proxy_temp_file_write_size 2048m;
>> proxy_intercept_errors on;
>> proxy_ssl_session_reuse off;
>> proxy_read_timeout 120;
>> proxy_connect_timeout 60;
>> proxy_send_timeout 120;
>> client_body_buffer_size 32m;
>>
>
> This setting allocates 32M buffer for each request with body.
>
>
>> client_header_buffer_size 64k;
>> large_client_header_buffers 16 64k;
>> client_max_body_size 16m;
>>
>>
>> server
>> {
>> listen 1.2.3.4:80;
>> location /
>> {
>>
>> proxy_pass http://3.4.5.6;
>> proxy_redirect http://3.4.5.6/
>> http://$http_host/;
>> proxy_redirect default;
>> proxy_set_header Host
>> $host; ##Forwards host along
>> proxy_set_header X-Real-IP
>> $remote_addr; ##Sends realip to customer svr
>> proxy_set_header X-Forwarded-For
>> $remote_addr; ##Sends realip to customer svr
>> }
>> }
>> server
>> {
>> listen 1.2.3.4:443;
>>
>> ssl on;
>> ssl_certificate
>> /usr/local/nginx/conf/whatever.com.crt;
>> ssl_certificate_key
>> /usr/local/nginx/conf/whatever.com.key;
>> ssl_protocols SSLv3;
>> ssl_ciphers HIGH:!ADH;
>> ssl_prefer_server_ciphers on;
>> location /
>> {
>> proxy_pass https://3.4.5.6;
>> proxy_redirect https://3.4.5.6/
>> http://$http_host/;
>> proxy_redirect default;
>> proxy_set_header Host
>> $host; ##Forwards host along
>> proxy_set_header X-Real-IP
>> $remote_addr; ##Sends realip to customer svr
>> proxy_set_header X-Forwarded-For
>> $remote_addr; ##Sends realip to customer svr
>> proxy_set_header X-FORWARDED_PROTO https;
>> }
>> }
>>
>> And these server entries repeated about 60 or so times and that's it.
>> When it was around 40 we never had a memleak issue.
>>
>> This is on Linux, kernel is 2.6.25
>>
>> Thanks
>>
>>
>> --
>> GloboTech Communications
>> Phone: 1-514-907-0050 x 215
>> Toll Free: 1-(888)-GTCOMM1
>> Fax: 1-(514)-907-0750
>> paul@gtcomm.net
>> http://www.gtcomm.net
>>
>>
>
>

--
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul@gtcomm.net
http://www.gtcomm.net

Re: memcached problems...

On Mon, Aug 31, 2009 at 08:05:07PM +0200, Juergen Gotteswinter wrote:

> Hallo folks again,
>
> i try to configure nginx to deliver swf files etc, and wanted to cache
> them within memcached. but until now i didnt have much success with that.
> what i already did:
>
> include/excluded parts of the config (flv part, swf part & php part) and
> tested them one by one. no change... but if i try to open a sfv file via
> browser i see that nginx communicates with memcached and this error
> messages appears in the nginx error log (without the memcached part in
> the config i get the file as download, like expected)
>
> 2009/08/31 18:34:40 [info] 23808#0: *1 key:
> "/files/_somewhere/xxx/xxxx.flv" was not found by memcached while
> reading response header from upstream, client: 62.
> 116.129.3, server: media, request: "GET /files/_somewhere/xxx/xxxx.flv
> HTTP/1.1", upstream: "memcached://127.0.0.1:11211", host: "ftp.xxx.org:8
> 1"
>
> it seems that nginx looks in memcached for the file, but it has not yet
> read from the disk so nginx cant find it inside memcached and throws me
> a 404

Put your swf files on filesystem and forget about memcached: in modern OS
disk == memory, if content volume is comparable to physical memory (and
it'seems to be comparable if you want to put files in memcached).


--
Igor Sysoev
http://sysoev.ru/en/