Hi managers,
I asked the question below two weeks ago, without getting an
exect answer from this list. Sorry it took me so long to summarise,
basically I did some tweaking on the oracle web server and needed
the time to monitor it's behavior after that.
Well, first thanks to :
Arthur Darren Dunham, Olivier Grange-Labat, Dave Rhodes.
Their suggestions :
- CLOSE_WAIT is the state for the TCP connection after the remote side has
requested a shutdown, and the TCP connection is waiting for the local
application to close the socket. So the problem might be related to the
web server program -- oraweb.
- tune the tcp_close_wait_interval parameter.
- the only parameters that needed to be set were :
tcp_keepalive_interval
tcp_ip_abort_interval
tcp_close_wait_interval
I received me-toos from a few person as well.
Now what I did and had observed from the server :
1. We use Oracle Web Application Server to connect to our backend
oracle database for data retrieval. In OWAS's implementation, there
are 'cartridge' processes spawned by OWAS which in turn will create
oracle processes and interact with the database.
2. In my OWAS's settings, the number of such cartridge is set to a
minimum of 0, which mean the system will need to create/kill new
cartridge and oracle processes every time there's an incoming request.
3. The system is a little low on memory. So during create/kill of
such processes, there's disk swapping which slowed down especially
the creating of new processes.
4. So during this stage, if the user kills the web connection, somehow
this left oraweb with the CLOSE_WAIT state for that connection.
5. I increased the number of minimum cartridge OWAS should create to a
non-zero value, so that it by default will have this number of cartridge
and oracle processes running and doesn't spawn/kill processes for every
new request that comes in.
6. After this change and restarted OWAS, for the past 2 weeks, there's
very few CLOSE_WAIT connections, and oraweb managed to kill these
connections quite fast (within minutes) when they happened. So far
it's only one instance that the problem as described below re-occurred
and I had to kill and restart oraweb.
I'm buying more memory for this server and hope that with sufficient
memory for all the processes, I can avoid this problem.
So in summary, I still couldn't pin-point the exect cause of the problem
but seems that it's performance related.
For your info please. Thanks.
bh khoo
>X-Sender: boonhing@ncb.gov.sg
>X-Mailer: QUALCOMM Windows Eudora Light Version 3.0.6 (32)
>Date: Thu, 29 Jul 1999 20:41:55 +0800
>To: sun-managers@sunmanagers.ececs.uc.edu
>From: Khoo Boon Hing <boonhing@ncb.gov.sg>
>Subject: Oraweb server - CLOSE_WAIT
>Sender: owner-sun-managers@sunmanagers.ececs.uc.edu
>
>Hi managers,
>
>I have a UE450 Solaris 2.5.1 running Oracle RDBMS and Oracle Web
>v 3.0 (oraweb) with latest recommended Solaris 251 patches.
>
>os 102 > uname -a
>SunOS os 5.5.1 Generic_103640-26 sun4u sparc SUNW,Ultra-4
>
>The problem is, using PC browsers to access the oraweb at port 8080,
>if the user is to click on 'Stop' at the browser, this will create
>a CLOSE_WAIT connection at the server.
>
>Very often, the number of such CLOSE_WAIT connections accumulate and
>the oraweb server stop responding to new connections although these
>new connections are ESTABLISHED. A netstat -n output :
>
># netstat -n
>
>TCP
> Local Address Remote Address Swind Send-Q Rwind Recv-Q State
>-------------------- -------------------- ----- ------ ----- ------ -------
>222.22.111.207.23 222.22.111.164.1864 7349 0 8760 0 ESTABLISHED
>222.22.111.207.23 222.22.111.164.1870 8009 1 8760 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.162.3858 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.33177 222.22.111.207.8080 32768 0 8192 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.207.33177 8192 0 32768 0 ESTABLISHED
>222.22.111.207.8080 223.122.56.8.58740 8760 0 8760 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.168.1248 8760 0 8760 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.162.3859 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 223.122.56.8.58757 8760 0 8760 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.162.3860 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3861 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3874 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3875 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3911 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3912 8760 0 8760 0 ESTABLISHED
>222.22.111.207.8080 222.22.111.162.3913 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3914 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.164.1987 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 222.22.111.162.3915 8760 0 8760 0 ESTABLISHED
>222.22.111.207.8080 223.122.56.8.59202 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 223.122.56.8.59203 8760 0 8760 0 CLOSE_WAIT
>222.22.111.207.8080 223.122.56.8.59556 8760 0 8760 0 CLOSE_WAIT
>Active UNIX domain sockets
>Address Type Vnode Conn Addr
>60a71970 dgram 5 0 /tmp/psb_back_socket
>60a71ab0 dgram 4 0 /tmp/psb_front_socket
>608dc428 stream-ord 0 608dc6a8
>608dc6a8 stream-ord 0 608dc428
>608ddbe8 stream-ord 48946 0 /var/tmp/.oracle/s#229.1
>#
>
>The oraweb server will hang in this state for a 2-3 hours, then the server
>will somehow recovers and the CLOSE_WAIT connections are gone.
>
>Is there any way to kill the CLOSE_WAIT connections, or make these
>connections timeout very fast so that the web server does not hang ?
>
>I even tried tuning the /dev/tcp parameters, but they doesn't help
>(in /etc/rc2.d/S69inet) :
>
>ndd -set /dev/tcp tcp_keepalive_interval 900000
>ndd -set /dev/tcp tcp_close_wait_interval 60000
>ndd -set /dev/tcp tcp_ip_abort_interval 60000
>ndd -set /dev/tcp tcp_rexmit_interval_min 1500
>ndd -set /dev/tcp tcp_rexmit_interval_initial 2000
>ndd -set /dev/tcp tcp_rexmit_interval_max 10000
>ndd -set /dev/tcp tcp_conn_req_max_q0 4096
>ndd -set /dev/tcp tcp_conn_req_max_q 1024
>ndd -set /dev/tcp tcp_slow_start_initial 2
>ndd -set /dev/ip ip_path_mtu_discovery 0
># end static updates
>
>Please advise. Thanks in advance.
>
>bh khoo
>
>
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:24 CDT