a load of distributed Linux servers

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen.  Find a real news server.

NOTE: Please don't multipost the same message to multiple groups. If
you MUST post to multiple groups, include them all in the Newsgroups:
header, and set a Followup-To: header as I have done here.

Are you indicating that someone already _HAS_ gotten in?  If so, you
are screwed - as you don't know what that person has done. You mention
identifying compromised binaries - that implies the person has root -
and think you can identify things by modification date.  Boy, are you
ever wrong:

And the bad guy is getting in exactly how?   You're posting from an
address of a company in the medical business. If there is even the
remote possibility that this is subject to HIPAA, disconnect the network
from the Internet RIGHT NOW.

as it's a waste of time if the bad guy has gotten in once already. How
do you know that the tools you are using haven't been 0wn3d already?

You're posting from what is indicating Ubunto 8.04.  If that is
representative of your Linux, have you been keeping things up to date?
If so, the bad guy may be getting in through a compromised account.
Which one?

In the Usenet newsgroup comp.os.linux.security, Bit Twister has
suggested aide and samhain (and ossec which I have no experience with).
A problem with these tools is that they are best used starting with a
"clean" system, and allowing them to report on changes.  In both Usenet
newsgroups, others have recommended a reinstall. I TOTALLY agree with
that recommendation. If you believe you've been compromised, the _only_
solution is a wipe and reinstall, followed by a full update before
allowing access from the world.

Hmmm, I wonder if your SSH setup is one of the compromised ones. From
the US-CERT Technical Cyber Security Alert TA08-137A from last May:

Once I'm not swamped with other things I'll check this out.

Are you indicating that someone already _HAS_ gotten in? If so, you
are screwed - as you don't know what that person has done.

I am aware of that, and yes, I'm pretty sure that he did get root.

The modification stamps on /bin/bash and /bin/grep didn't get there by
anyone else and I didn't do it personally. It does tell a little bit
about the sophistication level of the person that I'm dealing with,
though.

Obviously it either indicates that the person thought we were
incompetent (or did not have enough resources available as a target)
enough to not bother with the time for that, or they do not posses the
level of sophistication to implement that.

And the bad guy is getting in exactly how?

Almost certainly looks to be from an apache or webmin hole.

I've dealt with network security from the other side of the spectrum
when I was an angsty teenager and learned how exploitations occur. So
I may not have the knowledge to deal with this on the administration
side just yet, but I do know what I'm looking for, what he's capable
of, and have an idea of what we're playing with.

In the Usenet newsgroup comp.os.linux.security, Bit Twister has
suggested aide and samhain (and ossec which I have no experience
with). A problem with these tools is that they are best used
starting with a "clean" system, and allowing them to report on
changes.

Thank you for the advice. I'll look into the tools. I'm doing what I
can to advocate what you've mentioned, but I might be hamstrung from
higher up in being able to do a full cleaning.

Hmmm, I wonder if your SSH setup is one of the compromised ones. From
the US-CERT Technical Cyber Security Alert TA08-137A from last May:

On the old machine it certainly was. I was aware of this ssl
vulnerability but didn't think of it because I didn't know about
Asterisk's usage of SSL. I'll look into it more now.

I don't have a whole lot of resources to work with right now and a
pointing in the right direction saves me a lot of time and enables me
to do what I can over the weekend when all of our users are off of
the main servers, so quick action is a necessity.

NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen. Find a real news server.

Are you indicating that someone already _HAS_ gotten in? If so, you
are screwed - as you don't know what that person has done.

I am aware of that, and yes, I'm pretty sure that he did get root.

The only safe solution is to wipe and reinstall from known clean media.
But you know that too.

Almost certainly looks to be from an apache or webmin hole.

Yeah, there are plenty of those. Two points: PHP is generally a walking
disaster area - looking at Bugtraq shows more PHP exploits than
anything else, followed fairly closely by SQL.

Second point, if one of your users has been 0wn3d _elsewhere_ and they
had their SSH authentication tokens on "that" system, there is an
exploit out there that uses stolen authentication to get "in" to your
system, uses any number of local exploits to escalate privilege to
root, and then installs the 'Phalanx2' root kit that hides itself
pretty well, and searches for authentication data on your system (which
it mails out to a drop-box). CERT reported this about five months ago.

The only safe solution is the wipe/reinstall/update. There are a
number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
(http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
to look for many rootkits. By in large, these are trivial to defeat or
bypass. As one example, all look for the existence of a file named
'/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
sniffer worm from 2003). That's all well and good, but what if the
mal-ware author does something terribly complicated, such as changing
the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
anti-mal-ware tools won't find it.

Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
strong point, but I'm much less thrilled by it now.

Many of us suffer through the same interference. Best advice remains
a wipe, reinstall from trusted media, and updates. In this case, I'd
also strongly recommend new passwords for ALL accounts. Then, take a
snapshot of the system using 'aide' (or the fore-runner 'tripwire').
This is what the system looks like "now" - and you can then compare
this snapshot to "later" and look for changes. Big caution: the
aide binary and database go on removable media that is kept in a safe
place when not being used. Also, you have to retake the snapshot each
time you intentionally change something (such as security updates). It
is a pain in the a$$, but less of a pain than what you are going
through now.

In comp.os.linux.security Moe Trin <ibuprofin at (no spam) painkiller.example.tld> wrote:
NOTE: Posting from groups.google.com (or some web-forums) dramatically
reduces the chance of your post being seen. Find a real news server.

Obviously I've gotten to that step now, something I had been neglecting
previously. Thanks.

Are you indicating that someone already _HAS_ gotten in? If so, you
are screwed - as you don't know what that person has done.
I am aware of that, and yes, I'm pretty sure that he did get root.

At this stage I know that he got root on one of our externally facing
machines. This machine has been wiped and reinstalled but is still not
up to the standards that I would like it to be at. It also had an
internal interface which was not securely firewalled or DMZed. Yes, I
understand the horror of that situation, but it was out of my control.
Thanks to that issue I now have the ability to make suggestions that
will be taken much more seriously regarding our handling of network
security in the future. This does not help the fact that I'm still
learning this as I go, however, at least at this level of
sophistication.

The only safe solution is to wipe and reinstall from known clean media.
But you know that too.

As a starting point I have wiped and reinstalled my own workstation; I
am not totally sure that I had it locked down before I reconnected it to
the WAN, however, so I may just end up doing that again. I am also
using a startpoint of my own personal laptop which, at this point, is
disconnected from the WAN and I am fairly certain is not compromised at
the moment. It is running a significantly different kernel from the
other machines, as well.

Almost certainly looks to be from an apache or webmin hole.
Yeah, there are plenty of those. Two points: PHP is generally a walking
disaster area - looking at Bugtraq shows more PHP exploits than
anything else, followed fairly closely by SQL.

Thank you. As we get downtime where our users are not needing vital
network services I will make sure that we concentrate on these areas and
not providing them where they aren't needed, and maintaining the best I
can find to lock them down where they absolutely are needed.

Second point, if one of your users has been 0wn3d _elsewhere_ and they
had their SSH authentication tokens on "that" system, there is an
exploit out there that uses stolen authentication to get "in" to your
system, uses any number of local exploits to escalate privilege to
root, and then installs the 'Phalanx2' root kit that hides itself
pretty well, and searches for authentication data on your system (which
it mails out to a drop-box). CERT reported this about five months ago.

Yep. I remember similar .rhost exploits and disasters surrounding them
over a decade ago.

The only safe solution is the wipe/reinstall/update. There are a
number of windoze wanna-be anti-mal-ware tools, such as 'chkrootkit'
(http://www.chkrootkit.org), 'rkhunter' (http://www.rootkit.nl), and
from a cursory look, 'ossec' (http://www.ossec.net/) that can be used
to look for many rootkits. By in large, these are trivial to defeat or
bypass. As one example, all look for the existence of a file named
'/tmp/.../a' or '/tmp/.../r' as evidence of the '55808 worm' (a port
sniffer worm from 2003). That's all well and good, but what if the
mal-ware author does something terribly complicated, such as changing
the filename to '/tmp/.../A' or '/tmp/.../b' or something... yup, the
anti-mal-ware tools won't find it.

I've actually been able to start going through some OSSEC logs at this
point that are showing things of this sort. Of course, I do not know
enough about the specifics of OSSEC administration just yet to be able
to eliminate all of the false positives, however there are a few entries
in them that are leading me to believe that we have significant
intrusion on some if not all of our primary internal servers within the
WAN. IE:

OSSEC HIDS Notification.
2009 Jan 20 15:29:47

Received From: (agentname) 192.168.1.NOU->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event
(rootcheck)."
Portion of the log(s):

Port '41861'(tcp) hidden. Kernel-level rootkit or trojaned version of
netstat.



--END OF NOTIFICATION


Comment: I've only done a cursory look at 'ossec' and 'C' isn't my
strong point, but I'm much less thrilled by it now.

I'm also working on getting these alternate tools installed on a
completely clean system right now to get some decent monitoring in
place.

Many of us suffer through the same interference. Best advice remains
a wipe, reinstall from trusted media, and updates. In this case, I'd
also strongly recommend new passwords for ALL accounts. Then, take a
snapshot of the system using 'aide' (or the fore-runner 'tripwire').
This is what the system looks like "now" - and you can then compare
this snapshot to "later" and look for changes. Big caution: the
aide binary and database go on removable media that is kept in a safe
place when not being used. Also, you have to retake the snapshot each
time you intentionally change something (such as security updates). It
is a pain in the a$$, but less of a pain than what you are going
through now.

Gotcha. Your recommendations have given me an excellent starting point
and I thank you much. Any further feedback is greatly appreciated.

-Damon Getsman