SUMMARY: software or script that makes patching system easier.

From: Mark Lin <sv650_at_macysadv.com>
Date: Mon Dec 17 2001 - 16:28:53 EST
Again, this list proves to be the best resource I've yet encounter online.

List of people to thank:
Randy Romero <splat@fury.to>
Ken_Germann@bluecrossmn.com
Hindley Nick <nick.hindley@lbhf.gov.uk>
Jeff Putsch <putsch@hf.mxim.com>
Tim Chipman <chipman@ecopiabio.com>
David Foster <foster@dim.ucsd.edu>
Thomas Knox <Thomas_Knox@cch.com>
Adam.Kupsta@ca.cgeyc.com
Rikard Stemland Skjelsvik <rskjels@pogostick.net>
Rayen Riedel <RRiedel@upc.nl
Michael Sullivan <mps@discomsys.com>

Original Question:
------------------------------------------------------------
Hi, I want to ask sun admins about some centralize managment softwares.  We
have nine sun servers, running OS from 2.5 to 2.8.  Everytime I do a
recommended patch update, I have to download patches for three different OS,
and install them separately and individualy.  Cumbersome.

I would like to know if there is any software or script out there that can
download the newest patch and install on all the servers by issuing a few
commands.  I probably dont even need the patch to be downloaded, I just need
something to ease up the pain of patching so many systems.  I never use any
central management softwares like SMC or Tivoli, so I dont know if they can
do such jobs(I dont even think we can afford such software).  So, please let
me know if you have some solution for this.

yeah, you can call me lazy, but isn't laziness behind all the great
inventions. :)

thanks for your reply in advance.
Mark
--------------------------------------------------------------

Links to this summary:
http://www.sun.com/bigadmin  (for patchck.pl, checking necessary patch)
cfEngine:
http://www.iu.hio.no/cfengine/
http://astro.uchicago.edu/~davidr/cfengine-tools
Wget crawler:
http://www.geocrawler.com/archives/3/409/1999/6/0/2223890/
Misc:
http://www.gnu.org/software/wget/wget.html"
http://ist.uwaterloo.ca/security/howto/2000-12-04/  (very cool sun info
site! thanx Rikard)
http://www.boran.com/security/sp/Solaris_hardening4.html#Patches (another
one from Rikard, very informative)
SecurityFocus Vulnerability calculator
http://SecurityFocus.com/focus/sun/form.html
Casper Dik's FastPatch
ftp://www.wins.uva.nl/pub/solaris/auto-install/
Joe Shambin's Patchreport
ftp://x86.cs.duke.edu/pub/PatchReport/index.html
wget command
http://sunsolve.sun.com/private-cgi/show.pl?target=wget
checkpatch
http://www.boran.com/security/sp/solaris/CheckPatches.tar


Summary:
    There are lots of different approachs to my question.  Out of eleven
replies, one person express negative experience.  One suggests that since in
order to apply Recommended Patch, system has to be brought down to single
user mode, script should only goes as far as distributing the Patch Cluster
to each machine, and no further than that.  He also suggests that "it is
possible to cook up a temporary /etc/rc0.d script which would fire up the
cluster pactch install script and pass it "yes" to the first "proceed now"
question".  Couple replies suggested wget, which should ease up the pain of
downloading patch cluster manually.
    Two people suggest using Cfengine(check the link I provide above), but I
havn't got time to read more about it, but from the look of it, it seems to
be capable of doing more than just automated patching.
    People has suggested solutions that aim at different parts of the
patching automation: patch downloading(wget), installation(by Michael
Sulliva), or patch cluster distribution(by Thomas Knox, his script does
almost everything).  Some scripts use patchdiag and its reference file for
downloading patches, some require you to input patch numbers in a file.  You
should use whatever you see fit.
    As Tim points out in his reply, it's generally not a good idea to
blindly install patchs on production systems without reviewing the readme
file or test it on a non-production system first.  You might not need the
patch to certain video card because you dont have it on your server.  You
might need to reboot the system after patching.  So, be sure to know what
the patches are before you install them.


p.s. original replies are pasted below(some full blown scripts are
included).

Thank you for the help again,
Mark

---------------------------------------------------
Original Replies:
Thomas Knox:
=============================================
This i being published in the next issue of
SysAdmin magazine. Enjoy.

Tom
Background

At my company we have many Solaris servers, and maintaining the
patchlevels on them was turning into a full time job. We also have one
server directly connected to the Internet with no firewall or proxy, so
keeping that system up to date is very important.

With the worldwide explosion of "script-kiddies" and the easy availability
of hacking programs, keeping exposed systems up to date is a requirement.
It is not a question of if an exposed system with known vulnerabilities
will get hacked, it is a question of when.  The mailing lists offered by
SANS http://www.sans.org/ and CERT http://www.cert.org/ are useful in
keeping on top of any new security holes that have been discovered.

Keeping the latest Solaris patches installed is a big step towards keeping
your machine out of the hands of unfriendlies, and to that end I have
written a few scripts to automate the patching process as much as
possible. These scripts have been tested on Solaris 2.6, 2.7 and 2.8.

The Automation Setup

NOTE: Please substitute the URL for your local SunSolve mirror in place of
sunsolve.sun.com. See

http://sunsolve.Sun.COM/private-cgi/show.pl?target=link for the list of
mirrors.

The first thing to do is to install the Sun patchdiag tool onto your
server(s). I like to install it into /usr/local/patchdiag so I always know
where it is, no matter what system I might be on. The patchdiag tool can
be found at

http://sunsolve.Sun.COM/private-cgi/show.pl?target=resources/patchdiag and
the most recent version as of this writing is 1.0.4. After you have
downloaded the patchdiag tool, install it into a uniform place. All of my
scripts assume /usr/local/patchdiag, change
yours accordingly.

cd /usr/local
zcat patchdiag-1.0.4.tar.Z | tar -xvf -
ln -s patchdiag-1.0.4 patchdiag
cd patchdiag
./patchdiag_setup

I also make a user on each machine called patches that owns the patchdiag
directory. This account is used to automate pushing the patchdiag.xref
file to all of the servers.

cd /usr/local
chown -R patches patchdiag-1.0.4
chmod 700 patchdiag-1.0.4

The Automation Process

The first script will go out to the SunSolve FTP site and download the
current patchdiag.xref file for system analysis. After downloading it, it
will push it to all of your other servers.

--- Start of script ---

#!/usr/bin/ksh

PATH=/usr/bin; export PATH
SUNSOLVE=sunsolve.sun.com

cd /tmp
rm -f patchdiag.xref >/dev/null 2>&1

ftp -n << EOF
    open ${SUNSOLVE}
    user anonymous my_email@company.com
    binary
    cd /pub/patches
    get patchdiag.xref
    bye
EOF

rm -f /usr/local/patchdiag/patchdiag.xref
>/dev/null 2>&1
cp patchdiag.xref /usr/local/patchdiag

while read SYSTEM ACCOUNT PASSWD DIR
do
    ftp -n << EOF2
        open ${SYSTEM}
        user ${ACCOUNT} ${PASSWD}
        binary
        cd ${DIR}
        put patchdiag.xref
        bye
EOF2
    done << SYSEOF
host1 login_id password /usr/local/patchdiag
...
hostX login_id password /opt/sun/patchdiag
SYSEOF

--- End of script ---

Replace "host1 login_id password" ... "hostX login_id password" with your
server names and the login information, for example:
"sunbox1 patches patchpw /usr/local/patchdiag". Since this script will have
live account info, it is a good idea to keep it owned by root with
permissions 700, and in a private directory.

Initially I used ncftpget to FTP the patchdiag.xref file, but Sun changed
how the file was stored (it is now listed as a 0 byte file) and ncftpget
will no longer retrieve this file, even with command line arguments to
"force" a RETR.

This script was designed to be run as a cron entry. Depending on how often
you check your patch levels should determine for you how often to run this
script. Running it at off-peak hours will endear you to the Sun
administrators.

The next phase of our automation involves determining which patches need
to be downloaded, getting them, and preping them for installation.

This script uses wget, available from

"http://www.gnu.org/software/wget/wget.html", or precompiled from
"http://www.sunfreeware.com/.
Follow the instructions provided by your download of wget and install it.
Replace "my_login_id" with your SunSolve login ID, and "my_passwd" with
your SunSolve password. Again, since this script contains live passwords,
keep it in a private directory with permissions 700.

--- Start of script ---

#!/usr/bin/ksh

PD=/usr/local/patchdiag/patchdiag

PATH=/usr/bin:/usr/local/bin; export PATH
SUN_MIRROR=sunsolve.sun.com
let MY_VER=`uname -r | cut -f2 -d.`; export
MY_VER
SUNSOLVE_ID=my_login_id
SUNSOLVE_PW=my_passwd
PATCH_DIR=/tmp/patches

PATCHES=`${PD} | grep ^1 | grep -v CURRENT |
sed '/^[0-9\-]*\ *[0-9]*\*$/d' \ |
    grep -vi obsoleted | nawk '{ print $1 }'
| sort -u`

${PD} | grep -v CURRENT | sed '/^[0-9\-]*\
*[0-9]*\*$/d' \ |
    grep -iv obsoleted | mailx -s "Patches
retrieved" my_email@company.com

undo_patch() {
    if [ "$MY_VER" -ge "7" ]; then
        unzip -o ${1}*.zip && rm -f
${1}*.zip >/dev/null 2>&1
    else
        uncompress ${1}*.tar.Z && tar -xf
${1}*.tar && \
            rm -f ${1}*.tar >/dev/null 2>&1
    fi
}

mkdir -p ${PATCH_DIR} >/dev/null 2>&1
cd ${PATCH_DIR}
touch patch.ignore

for i in ${PATCHES}
do
    grep -s ${i} patch.ignore >/dev/null
2>&1
    if [ "$?" -ne "0" ]; then
        wget -nd -l2 -r -A "${i}*"
--http-user=${SUNSOLVE_ID} \
            --http-passwd=${SUNSOLVE_PW} \

"http://${SUN_MIRROR}/private-cgi/pls.pl?arg=${i}*" >/dev/null 2>&1
        rm -f pls.pl*

        undo_patch ${i}
    fi
done

--- End of script ---

patch.ignore is a list of patch ID's that you do _not_ want to get. For
example, if you're running a headless Solaris 8 server, you probably do
not want patch 108576 to support Expert3D IFB Graphics. List the patches
without revision numbers. A patch.ignore file that contained the
following:

108569
108576
108864

Would not download patches 108569, 108576 or
108864.

If your server is behind a proxy, add the flags "--proxy=on
--proxy-user=my_id --proxy-passwd=my_passwd" to the wget statement above,
supplying your correct proxy user id and password. Be sure to add the line
http_proxy = http://my.company.proxy:port/ to your ~/.wgetrc file, or
define the environment variable http_proxy in the script, e.g.

http_proxy=http://proxy.company.com:8080/;
export http_proxy

This script will get all current patches for your system that were not
explicitly excluded by the patch.ignore file, and their associated readme
files. It will also expand the patches for easy installation.

It would be easy to automate the patch installation as well. A simple
"i=`ls -d *-*`; for j in $i;do;patchadd $j;done" would work
perfectly well. However, it is highly recommended not to do so, but rather
review each patch's .readme and PATCH-ID/README files to determine
applicability and special requirements, as well as if a specific order
is needed for installation.

This script can also be run from cron, preferably after the first one.

Using these scripts on a regular basis on my servers has enabled me to be
much more proactive in keeping my systems up to date and preventing problems
before they become major issues. It has also reduced the usual hassle in
finding new patches and getting
them, saving my time for other tasks.
==============================================



Tim Chipman:
==============================================
Given that recommended patch clusters are meant to be applied in
single-user mode, your options are a bit limited. Possibly you could
cook up some script to automate the file distribution process (ie, while
all machines are still in multi-user mode - to copy and extract them to
standardized locations on all boxes).

However you still will need downtime and then console access in single
user mode to apply the things.

Personally I wouldn't be interested in cutting corners harder than this.
It is possible I guess that you could cook up a temporary /etc/rc0.d
script which would fire up the cluster pactch install script and pass it
"yes" to the first "proceed now" question.

IMHO this is not really the greatest idea but..

anyhow. Maybe you shall get more optimistic feedback from others. I'll
be interested to hear the concensus in a summary :-)

--Tim Chipman
==============================================



Adam Kupsta:
==============================================
What you ask for can be developed in a simple shell script. I don't have
any samples but I can give you an idea of what this script can do.

1. have a variable for all Solaris 2.6, 7 and 8 servers
eg.
SOLARIS26="server1 server2 server3 etc.."
SOLARIS7="server1 server2 server3 etc.."
SOLARIS8="server1 server2 server3 etc.."

2. A simple case / for loop / if statement comparison can determine which
environment the script is executed from.

3. install the approriate patch/cluster.

To automate this, you can put the patches/clusters on a NFS share, have it
mounted on all servers and schedule a job in cron to run this script
periodically. If a patch cluster exists say in
/<mnt_point>/solaris26/clusters/... and it's newer than one already applied
install it, if not, look for patches in a patches directory and grep the
output of showrev -p to see if a patch is already applied.

There, I've pretty much wrote the script for you, you just have to put ifs
and fis around the logic :) ..

Hope this helps,

Adam Kupsta
Cap Gemini Ernst & Young Canada Inc.
Senior Consultant
Critical Technologies, Network Infrastructure Solutions
==============================================



Michael Sulliva:
==============================================
Mark,

Here's something I wrote.  Not the most efficient way to do it, but it
makes sure things are put in the correct order (and I can tweak the
order in the patch list file), as well as avoids trying to apply the
patch twice and makes sure the most current version of the patch is
appied.

Hope it helps,

Mike


------------------------- BEGIN INCLUDED FILE -------------------------
#!/bin/sh

patchlist="/tmp/patches.$$"
newpatchlist="/tmp/newpatches.$$"
installedlist="/tmp/installed.$$"

# Get a list of all the latest patch numbers and revisions
echo ""
echo "Gathgering the latest patches from the directory\c"
for patchname in `/bin/ls -d *-* | awk -F'-' '{print $1}'`
do
        echo ".\c"
        patches="`/bin/ls -d ${patchname}-* | tail -1`\n${patches}"
done
echo ${patches} | sort -u > ${newpatchlist}

# Build a list of the latest patches in the patch cluster "patch_order"
# file.
echo ""
echo "Updating the Sun patch cluster patch_order list with latest patches\c"
for patchname in `awk -F'-' '{print $1}' < patch_order`
do
        echo ".\c"
        grep ${patchname} ${newpatchlist} >> ${patchlist}
done

# Now append the patches not in the "patch_order" file.
echo ""
echo "Appending pathes not included in the patch cluster\c"
for patchname in `cat ${newpatchlist}`
do
        echo ".\c"
        if [ -z "`grep -s ${patchname} ${patchlist}`" ]
        then
                echo "${patchname}" >> ${patchlist}
        fi
done

# Now install the patch depending on whether ot not it's already been
# installed.
echo ""
echo "Starting installation phase."
showrev -p | awk '{print $2}' > ${installedlist}
for patchname in `cat ${patchlist}`
do
        if [ -z "`grep -s ${patchname} ${installedlist}`" ]
        then
                echo "Installing patch: ${patchname}"
                cd ${patchname}
                patchadd -d .
                cd ..
        else
                echo "Skipping patch: ${patchname}"
        fi
done

cp ${patchlist} patchorder.`uname -n | awk -F"." '{print $1}'`

rm ${patchlist} ${newpatchlist} ${installedlist}
-------------------------- END INCLUDED FILE --------------------------
==============================================
Received on Mon Dec 17 21:28:53 2001

This archive was generated by hypermail 2.1.8 : Wed Mar 23 2016 - 16:32:37 EDT