SUMMARY: How to determine if we were hacked

SUMMARY: "how to determine if we were hacked" (and how to respond)

Thanks to all who gave advice--sorry for the delay posting a summary, but 
as you can imagine, we went right off and did a bunch of stuff.

Advice ranged from "you've been compromised, get the box off the network 
NOW and reinstall everything" to "how important is the machine, do you 
trust Tripwire, but at least get it off the network until you can think all 
the nuances through." Pretty much what I would have said myself, but the 
confirmation from all directions was great.

My bosses decided we trust Tripwire. There is not much on our machines that 
would be classified as highly sensitive--we're in an academic environment 
and serve the general faculty/staff/student population, NOT registrar or 
financial data. The affected box was a staging server, not a production 
machine. So we're not playing by quite as paranoid a rule set as *I* might 
have if it was just up to me. It's primarily been a matter of 
rebuilding/replacing the binaries reported by Tripwire (i.e. all of ssh--we 
were running some pretty old versions--but we've taken the opportunity to 
upgrade on ALL our boxes), and making sure we weren't hit anywhere else. 
We've blown away our old host keys and generated new ones, changed root and 
shell passwords for everything (before AND AFTER ssh reinstallation), and 
so on.

Other suggestions people came up with--which are all going into my 
intrusion recovery file, with many thanks:

a. --running a port scanner (such as nmap, snoop, etc) to insure that no 
unusual services or ports are suddenly open.
b. --getting known good copies of ls/netstat/ps onto the system in case the 
originals were compromised by a rootkit and using the good versions to look 
for other things.
c. --running "strings" against the compromised binary and looking for 
anything strange (tried this ... nothing obvious, but the suggestions about 
specific kinds of things to look for, such as  are great)
d. --use Sun's "patchdiag" utility to look for other patches we might need 
(I installed this on the affected machine just last week and was about to 
propagate it to the rest when this crisis hit--it looks like a good tool).
e.--go through the advice from CERT at 
http://www.cert.org/tech_tips/intruder_detection_checklist.html (thanks!!)
f.--Checksum your binaries and compare with /var/sadm/install/contents, 
preferably a copy of that file restored from backup from before the problem 
started.
g.--setting the following in /etc/system:

set noexec_user_stack=1
set noexec_user_stack_log=1

"This will prevent the kernel for running any code from the heap."

I think that's most of it. Again, thanks--this is the first security breach 
I've dealt with as a primary responsible sysadmin and it was great to have 
my suspicions and initial ideas about how to respond confirmed, plus some 
stuff I definitely hadn't thought of.

Thanks to:
"chris"; Eric; David Stern; Nick Hindley; Alex (alexp@----); David J Evans; 
Chris Josephes; Sean Quaint; David Meissner; Jim Lang; Blake; "chrish"; 
Michael DeSimone; Scott; Justin Stringfellow; Al Hopper; Casper; Johannes J 
Hartzenberg ...
and anyone else I forgot--

Pete

Peter A. Chvany, Ph.D.
Systems Administrator, Harvard Law School
617-496-9025; pchvany@law.harvard.edu