Overview
cPanel’s Technical Support department detected the following security issues:
- Compromised RPMs in the OpenSSH binaries.
- Compromised
libkeyutilsdirectories. root-level compromises.
In these cases, Trojan horses (Trojans) affected files that these directories and binaries contain. We strongly recommend that hosting providers and system administrators use this document to determine the status of their systems.
libkeyutils directories produce similar network traffic.
If you experience any issues while you perform these checks or you suspect that someone has compromised your server, contact your hosting provider for assistance.
-
To check your system for compromises unrelated to the OpenSSH Binaries and
libkeyutilsdirectories, use WHM’s Security Advisor interface (WHM >> Home >> Security Center >> Security Advisor). -
We strongly recommend that you read our Tips to Make Your Server More Secure documentation.
Compromised systems
Open SSH rpms
On CentOS and Red Hat® Enterprise Linux® systems, we determined that the following OpenSSH binaries contain the Ebury Trojan:
- The
sshdbinary. - The
sshbinary. - The
ssh-keygenbinary. - The
ssh-askpassbinary.
This Trojan’s code collects authentication credentials for inbound and outbound network connections, as well as the SSH keys that these binaries generate.
Use the following checks to determine whether the Trojan compromised your system’s OpenSSH binaries.
Check for malicious processes
To check for malicious processes on your server, run the following command:
netstat -plan | grep atdThis command does not return output on non-compromised systems. On compromised systems, this command returns output that resembles the following example:
unix 2 [ ACC ] STREAM LISTENING 103713 8119/atd @/tmp/dbus-ZP7tFO4xsLCheck the RPMs’ change logs.
To check an RPM version’s change log, run the following command:
rpm -q --changelog openssh-5.3p1-209.el6 | head -10In this example, openssh-5.3p1-209.el6 represents the OpenSSH RPM binary that you wish to check.
Non-compromised RPMs contain a signature. A compromised RPM’s change log contains no entries.
Check your system’s yum logs.
Open your system’s /var/log/yum.log file with a text editor and search for the OpenSSH binaries. The file’s contents resemble the following example:
|
|
If your system’s /var/log/yum.log file does not contain the OpenSSH RPMs, the server may exist in a compromised state.
The libkeysutils utilities
To check your libkeysutils utility for the Ebury Trojan, perform the following steps:
-
We recommend that you perform each of the following steps to fully determine your server’s status.
-
Your server’s architecture determines the output that you receive when you run the commands in this section. 32-bit systems use
lib. 64-bit systems use the/lib64library. The/lib64library may also contain alibdirectory with 32-bit libraries for compatibility purposes. To find your server’s architecture, run thearchcommand:[user@host ~]$ archThe output will resemble one of the following examples:
- 64-bit server —
x86_64 - 32-bit server —
i386
- 64-bit server —
Verify the keyutils-libs package.
Changes should not occur on your system without your consent. To check for changes on your system, run either of the following commands:
|
|
This command does not return output on non-compromised systems. On compromised systems, this command returns the following output:
….L… /lib64/libkeyutils.so.1libkeyutils1 package to match the old packages, so an MD5 check may not reveal compromised packages.
Check the files’ installation source.
If the command in Step 1 returns output, verify whether an RPM provided the files. To do this, run the following command:
rpm -qf /lib64/libkeyutils.so.1/lib64/libkeyutils.so.1 represents the file to verify.
On non-compromised systems, the output resembles the following example:
keyutils-libs-1.4-5.el6.i686On compromised systems, the output resembles the following example:
file /lib64/tls/libkeyutils.so.1.5 is not owned by any packageVerify the file that links to the libkeyutils.so.1 file.
To see which file links to the libkeyutils.so.1 file, run the following command:
ls -l /lib64/libkeyutils.so.1On non-compromised systems, the output will resemble the following example:
lrwxrwxrwx 1 root root 18 Feb 20 12:15 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*On compromised systems, the output will return one or more of the following files:
|
|
Check the strings for all libkeyutils.* libraries for items that relate to networking.
The default libkeyutils.so.1.3 file should not contain the following strings:
|
|
To check your system for these strings, run the following command:
strings /lib64/libkeyutils.so.1.3 | egrep 'connect|socket|inet_ntoa|gethostbyname'This command does not return output on non-compromised systems. On compromised systems, this command returns the following output:
|
|
Check sshd processes.
On compromised servers, sshd processes use shared memory segments.
To check whether any sshd processes currently use shared memory segments, run the following command:
ipcs -mp
|
|
On a compromised server, the output will resemble the following example:
|
|
If any programs use shared memory segments, run the ps command, for example:
ps aux | grep 1975This command checks whether any of the items in the cpid and lpid columns correspond to the sshd processes. The output will resemble the following example:
|
|
Monitor outbound UDP port 53 traffic.
We recommend that you use a network data capture tool, such as the tcpdump utility, to monitor outbound User Datagram Protocol (UDP) on port 53. This utility returns DNS traffic between your server and the local resolvers that reside in your /etc/resolv.conf file.
To monitor outbound UDP on port 53, run the following command:
[root@host ~]# tcpdump -Annvvs 1500 -i any udp and dst port 53On a non-compromised server, the output will resemble the following example:
|
|
On a compromised server, the output will resemble the following example:
|
|
The example of the packet above shows that the cPanel server at 1.2.3.4 sends a UDP packet on port 53 to the host at 72.156.139.154. You should notice the following false query from the example above:
6196g8f43a4facd3561de4gec736fb.5.5.5.5{- The IP address that you used to log in to the server.
- The login password.
Verify that each library that sshd links against belongs to a known package.
To check the libraries that the sshd daemon links against, run the following command
ldd /usr/sbin/sshdThe output will resemble the following example:
|
|
To verify that these libraries belong to a valid package, run the following command:
rpm -qf /lib64/libfipscheck.so.1The output will resemble the following example:
fipscheck-lib-1.2.0-7.el6.x86_64Perform an LD_DEBUG check
To check the libraries with the LD_DEBUG program, which searches for Indicators of Compromise (IOC), run the following command
LD_DEBUG=symbols /bin/true 2>&1 | egrep '/lib(keyutils|ns[25]|pw[35]|s[bl]r).'The program will return any indicators of a compromise.
Perform an objdump check
The objdump command reports information about object files. You can use this command to determine if a needed file displays suspicious characteristics or behavior
To check the libraries with the LD_DEBUG program, run the following command, where /path/to/libkeyutils.so.1 represents the full path to the suspected file:
objdump -x /path/to/libkeyutils.so.1 | grep NEEDED | grep -v -F -e libdl.so -e libc.soThe program will return the phrase NEEDED if any indicators of a compromise exist.
root-level compromises
Check for suspicious library files
For example, the /lib64/libpw5.so file is part of the Ebury Rootkit, and clean systems do not usually contain this file. That file may use other names which the We Live Security analysis of the Ebury Rootkit document lists.
Llibrary files normally use less than 10Kb. Any library file that uses over 10Kb strongly indicates a compromise.
To check for the existence of the /lib64/libpw5.so library file, run the following command:
ls /lib64/libpw5.soThis command returns the following output when the file does not exist:
/bin/ls: cannot access /lib64/libpw5.so: No such file or directoryOn compromised systems, this command returns output that resembles the following example:
/lib64/libpw5.soAdditionally, the system uses yum or rpm to install most library files. If we query the RPM database for the /lib64/libpw5.so file and it returns a Not Owned By Any Package error, then the server may be compromised.
|
|
Check the file hash.
Run the sha256 hash of the file through the VirusTotal website to check it against pre-existing scans of the same file.
For example, to generate the sha256 hash for the /lib64/libpw5.so file, run the following command:
|
|
Then, navigate to the https://www.virtustotal.com/en/file/hash website, where hash represents the hash of the file. In this example, you would navigate to the following website:
https://www.virustotal.com/en/file/970b49c16eebd558ac8984643f3763e76a52c9be4118f9e5830b8f5c406414fc/analysis/
The website should return similar results to the following output:
|
|
These results show that at least 3 out of 5 antivirus systems detect this file as a compromise.
Check for intertwined binaries
Check binaries on your server to detect whether they intertwine with the suspicious file.
For example, the following command detects whether binaries intertwine with the /lib64/libpw5.so file.
# lsof | grep /lib64/libpw5.soThe command will produce results similar to the following output:
|
|
In this example, the first line shows that the /lib64/libpw5.so file intertwines with the auditd program. The auditd program, or Linux Auditing System, writes audit records to disk.
If you run the lsof -p 1577 command, which checks the PID of the auditd program, you will see results similar to the following:
|
|
The last line represents a socket connection in a temporary directory, which is highly suspicious. If we check for that name with the netstat command, we see that it listens for connections:
|
|
No reason exists for the auditd program to maintain a socket connection to the outside world. Likely in this case, hackers modified this binary enough to search for SSH keys or root passwords, which the binary then sends to a Command and Control server for later use.
FAQ
How does a root compromise differ from a site compromise?
A site compromise occurs on a specific website and malicious attackers can only steal information from that website, not from any others. A root compromise exposes the entire server to theft, such as SSH keys and passwords. It can also cause enough damage to the server so that it can no longer boot.
But what about symlink hacks that can hack multiple websites?
Most experts do not consider symlink hacks themselves to be root compromises, even though they look like they have hacked multiple accounts. The server itself is usually not in any danger.
For more information on how to prevent symlink hacks, read our Symlink Race Condition Protection documentation.
I’ve seen symlinks that can grab the /etc/passwd file. Isn’t that a root compromise?
No. The /etc/passwd file must be readable, but this does not represent a root compromise. The system stores password hashes in the /etc/shadow file, which you cannot simply view with a symlink hack. A further definition of a root-level compromise is one where an unauthorized user gains access to it. You may not lose much from a particular attack, but any unauthorized access has the potential for further breaches.
Every rootkit has at lease 2 purposes:
- Hides the attacker.
- Grants access to the attacker.
A root-level compromise exposes the entire server, and you should consider everything a loss. You can no longer trust any data, any configuration information, and probably any connectivity information and passwords. An attacker will likely want to continue to access to the server, and they will try to configure the system as to perform normally. This makes rootkits difficult to detect, but it is not impossible to detect them.
