ITSS Information
Security Services ITSS
Security Alerts > Multiple
UNIX compromises on campus -- 10 April 2004
On this page: Summary Technical
Details Countermeasures References
Summary
Stanford, along with a large number of
research institutions and high performance computing
centers, has become a target for some sophisticated
Linux and Solaris attacks. An unknown attacker (or
group) has compromised numerous multi-user Solaris and
Linux computers on Stanford's campus using a variety of
mechanisms. In most cases, the attacker gets access to a
machine by cracking or sniffing passwords. Local user
accounts are escalated to root privileges by triggering
a variety of local exploits, including the do_brk() and mremap() exploits on Linux and the sadmind, arbitrary kernel loading modules and
passwd vulnerabilities on Solaris.
If you believe
your Unix computer has been affected by these
intrusions, please contact the Information Security
Services office (650-723-2911 or http://securecomputing/alerts/security@stanford.edu).
Please include the name or IP address of the affected
machine, as well as any compromised
userIDs.
The attacker appears to be deliberately
targetting machines in academic and high performance
computing environments, rather than attacking systems
indiscriminately.
It's not always obvious that a machine has
been compromised, but they've been discovered at
Stanford because:
- a user notices that their last login (as reported
by many computers when the user logs in) is not
consistent with the last time she actually logged in
- a user complains that performance on a particular
computer has severely degraded
- a system administrator notices that users are
logging in from unusual locations -- in particular, a
sys admin notices multiple failed logins, sometimes
for more than one user ID, coming from another
university or research location
- unexpected errors are generated when a computer
reboots, because a rootkit has been installed on a
compromised machine, but doesn't work with that
particular machine's operating system or kernel
version
If you are responsible for Solaris or Linux computers
at Stanford, please be sure that your systems are
running the most recent versions of their kernels, and
all security relevant patches. These are described in
the security alerts referenced above.
If for some reason you cannot upgrade to the latest
Linux kernel, please consider taking other precautions
to defend these computers from malicious use. Possible
precautions are discussed in the Countermeasures
section below.
Technical
Details
Stanford system administrators and the
Information Security Services office are in contact with
a number of academic and research organizations
involving ongoing Linux and Solaris compromises. The
perpetrators regularly gain access to an unprivileged
local user account, presumably by sniffing passwords,
cracking passwords from other compromised systems, or by
triggering vulnerabilities in remotely accessible
services.
In cases where the victim machine is
running up to date versions of the kernel and privileged
applications, the compromised user account is typically
used to run a password decoding application called John
the Ripper. This program is what usually causes
system performance to degrade. The attacker is
knowledgeable about Kerberos as well as other
authentication systems, and has been observed running
dictionary attacks against Kerberos passwords, as well
as local password databases.
In cases where the target machine is
running known vulnerable versions of an operating system
or an application, or where there are insecure trust
relationships between multiple machines (one of which is
compromised already), the attacker is able to "get
root." With root privileges, he can replace core
utilities and applications on the victim machine,
usually with the intention of capturing more usernames
and passwords, and making it easier for himself to
access the machine at a later time.
Some compromised systems have rootkits
installed. A rootkit modifies the behavior of a
compromised machine by hiding malicious activity (like
keylogging or network sniffing); replacing system
binaries with malicious versions that enable the
attacker to collect information from local users; etc.
Unfortunately not all rootkits leave evidence on the
disk; truff's Phrack article on loadable kernel
modules describes mechanisms for changing the
behavior of a running machine by modifying the image in
memory. Even without support for loadable kernel modules
enabled, which might be expected to provide some
protection against kernel-based malware, some rootkits
are able to modify system behavior -- the SucKIT rootkit is perhaps the
best-known of these tools. SucKIT has been found on
several of the Linux boxes involved in the current set
of compromises.
An introduction to Linux Kernel
Rootkits
Given the sophistication of these attacks,
and the difficulty involved in removing rootkits and
illicit access mechanisms, we strongly recommend that
compromised hosts be taken off line and completely
rebuilt, including a fresh install of the operating
system and application of all relevant patches.
How are they
compromised?
The attacks start with the compromise of
an unprivileged local user account. Usually this is
because the attacker's captured the password from
somewhere else: it's been sniffed off the network
(through the use of insecure protocols like
telnet), it's been collected when the user
signs on to or from another compromised machine, it's
been harvested from the password file on a compromised
system.
If the target machine is behind on its
patches, the attacker then uses one of a number of
public exploits to elevate the unprivileged account to
root
status. Exploits target the Linux mremap()
vulnerabilities, the Solaris kernel module loading
vulnerability (for which an attack was made public on 8
Apr), and a Solaris priocntl()
issue.
Configuration
mistakes with the Network File Service
(NFS): NFS is a mechanism for sharing file
systems and directories over a network. In its most
conservative configuration, the amount of privileged
access provided to a shared directory is strictly
limited, and filesystems are marked nosuid*.
Unfortunately, there are many circumstances in which
more permissive access is granted, usually to facilitate
the distribution of system management and data
processing tasks -- in particular, programmers and
system administrators often make programs setuid root in
order to run them without requiring that the
user running them have root privileges.
In several cases of compromises at other
institutions, the attacker has taken advantage of an NFS
server exporting a filesystem to the world (ie. with no
access restrictions), and allowing root=0 NFS
access. With unrestricted NFS access to the target
system from another compromised machine, the attacker is
able to install a setuid root
version of the UNIX shell program /bin/sh. This
enables him to operate on the system with superuser
privileges.
* nosuid is an
filesystem mounting option that prevents any program or
script on that filesystem from running with superuser
privileges.
Evidence of
compromise
This following list summarizes the ways in
which compromised machines have been detected, both at
Stanford and at other organizations involved in these
attacks. Not all compromised
machines have all these symptoms. This list
may not be complete.
Any Linux or
Solaris system that is behind on security updates is
subject to compromise by the attacker. Any system
running NFS or maintaining trust relationships with
known-compromised machines are also at
risk.
- chkrootkit
detects the SucKIT rootkit or other malware -- but do
not assume that a clean bill of health from chkrootkit
means that your system is genuinely clean. Rootkits are
notoriously hard to find (that's kind of the point). The
SK rootkit replaces /sbin/init with
itself, and moves the real system init file to /sbin/initXXX,
where XXX is a special suffix that causes the file to be
hidden. Some victim machines have had the .d rootkit
installed. This one replaces /usr/lib/syslogd
and /bin/netstat -- it is sometimes detected by looking
for the file /etc/rc3.d/S20xfst.
[Note: in order to identify modified and/or hidden files
on a compromised system, you'll need to use versions of
your normal system tools (like ls and netstat) that
are guaranteed to be unaffected -- use known-good copies
on a CD or floppy. For more information on preparing a
forensics toolkit, check out
Incident Response Tools for Unix, Part
One: System Tools Forensic Analysis of a Live Linux System
Part I
and the references therein.]
- OpenSSH has been replaced by a malicious version.
This affects the SSH server and client as well as scp. The SSH
client is modified to capture usernames, passwords,
private keys and private key passphrases and send them
to a destination specified by the attacker, enabling him
to masquerade as a legitimate system user.
- in.ftpd and the
ftp client have been replaced by malicious versions.
These programs are modified to capture usernames and
passwords and send them to the attacker's specified
destination.
- ettercap is
installed and used to sniff usernames and passwords from
the network, as well as SSH v1 credentials which can be
extracted
- the user news, list, or smmsp is
enabled for interactive login (and is not a legitimate
system user) -- be on the lookout for any new user
accounts, especially those with root privileges or sudo access
- successful login from an "unexpected" remote host
without any preceding failures (indicating that the user
account being compromised had not yet been associated
with these attacks)
- multiple failed logins in sequence from a remote
system, followed by a successful login for a different
user (showing the attacker trying different combinations
of userIDs and passwords):
Apr 3
04:41:00 victim-host sshd[14598]: Bad protocol version
identification '' from
remote-unexpected-host-1
Apr 3
04:41:27 victim-host sshd(pam_unix)[14599]:
authentication failure; logname= uid=0 euid=0
tty=NODEVssh ruser= rhost=remote-unexpected-host-2
user=loginID-no-go-1
Apr 3
04:41:29 victim-host sshd[14599]: Failed password for
loginID-no-go-1 from remote-unexpected-host-2 port
36554 ssh2
Apr 3
04:41:37 victim-host sshd(pam_unix)[14601]:
authentication failure; logname= uid=0 euid=0
tty=NODEVssh ruser= rhost=remote-unexpected-host-2
user=loginID-no-go-2
Apr 3
04:41:39 victim-host sshd[14601]: Failed password for
loginID-no-go-2 from remote-unexpected-host-2 port
36555 ssh2
Apr 3
04:41:46 victim-host sshd[14603]: Accepted password
for loginID-bingo from remote-unexpected-host-2 port
36556 ssh2
Apr 3
04:41:46 victim-host sshd(pam_unix)[14605]: session
opened for loginID-bingo from remote-unexpected-host-2
port 36556 ssh2
(the failed logins are attempts to use loginIDs that
have already been identified as compromised, and are
therefore disabled).
- the root user
account executes the su command to
switch into an unprivileged user account:
2004-04-06
16:44:25 PDT auth.info local@victim-host su: [ID
366847 auth.info] su loginID-bingo succeeded for root
on /dev/pts/5
(there are some circumstances in which this is a
legitimate command, but in most "legitimate" situations
the root
user can accomplish their tasks without moving into a
less-privileged user context).
- The system programs /usr/bin/pppd,
/usr/bin/ypwatch
and/or /etc/inet/inetd.conf
are replaced
- The attacker's SSH key is added to the authorized
keys file, allowing key-based remote logins
- If rsh is enabled, the .rhosts file is
modified to allow universal access.
Detection
tools:
chkrootkit - checks local machine
for rootkit infestation Rootkit
Hunter - another detection tool swatch or grep or other log monitoring
programs to look for unusual login
patterns
CIAC relates these tests for rootkits on Linux:
- The SucKIT rootkit allows an attacker to hide
malicious files by giving them a particular ending. The
current attacker is hiding code that ends in xrk or mem. To test
for the presence of the rootkit, create a file whose
name ends in xrk or mem, then
execute an "ls
-l". If the files you just created are not shown
in the output of ls, it means
that the rootkit is hiding them, ie. your system is
compromised and needs to be rebuilt.
- Change directories to /sbin and
execute an "ls
-l init" -- the link count should be 1. Create a
hard link to init using ln, and then
execute the "ls
-l init" again. If the link count is still 1, the
SK rootkit is installed.
- Rooted systems send usernames and passwords to
other compromised machines using TCP port 55, so if you
keep records of network connections, traffic to
destination port TCP/55 merits further
investigation. Countermeasures
Note: all of the
countermeasures discussed below involve some amount of
work and inconvenience, both for you and your users.
As you're considering how to protect
your machines, your research and your co-workers from
these aggressive and sophisticated attackers, please
remember that although it's disruptive, the time you
spend today in changing passwords, upgrading kernels,
applying patches, and instituting remote access
restrictions is far less than the time you'll spend
rebuilding your system if it's
compromised!
No single protection
mechanism will prevent computer compromises -- but
implementing a few of these suggestions will minimize
your chances of being successfully attacked.
Prevention:
Install all
security-relevant patches for your multi-user UNIX
systems:
Sun releases patches for two privilege
escalation vulnerabilities Solaris vulnerability involving the
passwd command Solaris security vulnerability involving
the priocntl system call
Two vulnerabilities in Linux
distributions Linux kernel patch fixes memory
management vulnerabilities
Be sure your users are picking
hard-to-crack passwords:
Pluggable password strength checking for
your servers
Use different passwords on all your UNIX
accounts.
Use SSH with RSA keys or Kerberos for user
authentication.. [Note: these mechanisms will not
prevent an account from being compromised, especially on
a machine with a rootkit installed, but they
may make it harder for an attacker to access
unprivileged user accounts.]
Limit remote access to unpatched
multi-user systems, using host-based firewalls, TCP-wrappers or other mechanisms. This
is a potentially management-intensive process,
especially for machines with a lot of remote users --
but even relatively permissive rules (allow
connections from aol.com; deny
all else) provide some protection against remote
attacks.
References
[1] Computer Emergency Response Team: Steps
for Responding to a UNIX or Windows Compromise
[2] Securing a UNIX desktop
Last
modified Wednesday, 14-Apr-2004 21:05:03
PDT
© 2003-2004, Stanford University. All
rights reserved. Comments about this document?
Use
the HelpSU submission
form. Need computing help? Visit HelpSU or call 5-HELP (650-725-4357). |
|