Hi all, Many thanks to everyone who responded - Aleksander Pavic, francisco, and Darren Dunham. My original email is attached below, along with the replies I got - but to summarise : I was seeing a very high sysload on a Solaris 9 web server, and vmstat confirmed that a large number of system calls were being generated. I wanted to track these down and find out what was being called, but couldn't use Dtrace. Yet another argument for moving to Solaris 10 :) As Darren said in his response: "The limitations on existing tools like 'truss' are part of what drove dtrace, so I don't know that there's any magic out there." He then went on to suggest I analyse all the Apache processes with truss, send the output to a file and then analyse that. This was also the path suggested by Aleksander, who quite correctly pointed out that truss can be made to follow any child processes generated via forking, so I could therefore truss the main Apache process and follow all it's children. He also suggested I send the output to a file, and post-process it with awk or perl. Francisco also suggested the useful lsof tool to see what files are open, as my original hypothesis was that there were a large number of file handles being opened and closed. In the end, I trussed every "httpd" process, and generated a summary using "truss -c". I let this run for 20 seconds, and saw that there were a very large number of resolvepath() and open() calls being generated, roughly half of these calls returned with an error. I then narrowed my search down, and examined what was actually being passed as arguments to these calls. This is easily done with "truss -t open,resolvepath". It turns out that a huge number of the resolvepath()'s and open()'s were being generated by PHP scripts running under Apache. They were using an inefficient include_path, and so when most files were being included, PHP generated many resolvepath() and open() calls which returned in error before finally finding the correct location of the file. We fixed the PHP include_path and also modified some of the scripts to use an absolute path in include() or require() functions, and as expected, the number of syscalls being generated halved itself. There were a number of other code-related problems on that server as well, but these were unrelated to my original request for help. Once again, many thanks to those that responded. Problem resolved ! -Mark --- Original post : Hi all, Apologies in advance for the somewhat open-ended question here. I have a very busy web server, that is experiencing a large number of system calls - a snip from vmstat 1 produces : kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr m1 m1 m1 m2 in sy cs us sy id 0 0 0 6793080 5648456 14 123 0 8 0 0 0 0 0 0 0 1052 70799 1308 51 25 23 0 0 0 6800984 5650440 195 800 0 8 8 0 0 0 0 0 0 1334 49447 1476 38 23 39 0 0 0 6808168 5651576 63 238 0 8 8 0 0 1 1 1 0 1225 49429 1686 35 26 40 0 0 0 6807528 5652248 22 275 0 16 16 0 0 0 0 0 0 1040 41245 1252 46 16 38 0 0 0 6815480 5654432 50 306 0 16 16 0 0 0 0 0 0 1114 47343 1410 37 21 42 1 0 0 6825704 5657904 56 148 0 24 8 0 0 0 0 0 0 1388 68366 1648 53 34 13 7 0 0 6825648 5657784 50 170 0 0 0 0 0 0 0 0 0 1240 53751 1781 42 17 41 0 0 0 6825704 5657832 17 102 0 8 0 0 0 0 0 0 0 951 58447 1292 41 20 39 11 0 0 6825704 5657808 33 137 0 0 0 0 0 0 0 0 0 1011 53959 1219 54 23 23 6 0 0 6824168 5657184 41 173 0 32 8 0 0 1 1 1 0 1400 69368 1381 54 31 15 How can I find out what these calls are ? I have experimented with lockstat, but this doesn't seem to provide me with the information I'm after. I know there are many Dtrace scripts that would provide me with the information I'm after - but I am on Solaris 9/Sparc (V240, 2x1.2Ghz, 8Gb RAM) so that is sadly not an option. I'd ideally like something like a system-wide "truss"; In an ideal world, I'd like to be able to see most frequently called syscalls, and see what arguments are being passed to them. For instance, I suspect that our web application is opening a large number of files, but I have no way of proving this... Any input would be much appreciated. I will summarise any responses I get. Responses : [Darren Dunham] The limitations on existing tools like 'truss' are part of what drove dtrace, so I don't know that there's any magic out there. You can certainly write a shell script that runs 'truss' on all the server processes and sends output to files. Then you have to correlate the output files. For a large number of 'truss'es, you're going to affect performance on the box. For now, that's the path I'd take. Then you can post-process the output files to summarize them. [francisco] Not an answer to your overall question, but to see what files are open, lsof (not a standard Solaris command) comes in handy. You can use that per uid or process to see if a particular app has a lot of open files. [Aleksander Pavic] You can truss the main process of your WebServer. For example: If you're using apache, then you have something like a >master< process that spawns all other >slave< processes. Now truss the main process (with -f)and truss will follow all child processes. redirect STDERR to a file and analyse it with awk or perl. ----------------------------------------------------------------------- This E-mail is from IPC Media Ltd whose registered office is at Kings Reach Tower, Stamford Street, London SE1 9LS, registered number 53626. The contents and any attachments to it include information that is private and confidential and should only be read by those persons to whom they are addressed. IPC Media accepts no liability for any loss or damage suffered by any person arising from the use of this e-mail. Neither IPC Media nor the sender accepts any responsibility for viruses and it is your responsibility to check the email and attachments (if any). No contracts may be concluded on behalf of IPC Media by means of e-mail communications. If you have received this e-mail in error, please destroy and delete the message from your computer. For unbeatable savings on magazine subscriptions and great gift ideas visit www.giftmags.co.uk _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagersReceived on Wed Sep 14 05:57:47 2005
This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:43:51 EST