The other day, I had a client contact me with an issue: His Linux ColdFusion server had been hacked, and his web site defaced. This was a CentOS system, and they were running ColdFusion and PHP on Apache for a number of virtual hosts.
After SSH’ing into the system, the 1st thing I wanted to do was make sure the hacker was not still connected to the system. At a command line, enter ‘w’ and press enter. You will see something like this:
[root@host user]# w 07:39:00 up 15:57, 1 user, load average: 0.04, 0.07, 0.09 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT user pts/0 00-000-000-000.l 07:38 0.00s 0.02s 0.01s sshd: user [priv]
The output above shows that only one user is connected (myself), but if you had someone SSH’d into your system, you would see multiple lines here, as well as their source IP address.
At first my approach was to review system logs. This server was hosting many many sites, some of the log files were in the 2 GB range, so easily getting log idea’s from the console was less than ideal. ( I know that there are great tools like grep and awk and what not, I am just not a wizard with those tools ). I zipped up the apache log files and downloaded them locally, then wrote a quick ColdFusion script to parse the important bits into SQL.
<cfloop file="/Users/justice/Downloads/access_log" index="line"> <cfset templine = replace(line, " """, ",", "ALL") /> <cfset templine = replace(templine, " -", ", ", "ALL") /> <cfset templine = replace(templine, """ ", ",", "ALL") /> <cfset templine = replace(templine, " [", ",[", "ALL") /> <cfset templine = replace(templine, "] ", "],", "ALL") /> <cfset k = 1 /> <cfset record = structNew() /> <cfloop list="#templine#" delimiters="," index="i"> <cfswitch expression="#k#" > <cfcase value="1"> <cfset record.ip = replace(replace(i, '-', '', 'ALL'), ' ', '', 'ALL') /> </cfcase> <cfcase value="2"> <cfset record.date = replace( replace(i, '[', '', 'ALL'), ']', '', 'ALL') /> </cfcase> <cfcase value="3"> <cfset record.url = right(i, len(i)-3) /> </cfcase> <cfcase value="5"> <cfset record.refer = i /> </cfcase> </cfswitch> <cfset k++ /> </cfloop> <!---// Insert record into the database //---> <cfquery name="insert" datasource="localTest"> INSERT INTO apacheLogs( ip, reqdate, url, refer ) VALUES ( <cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#record.ip#" />, <cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#record.date#" />, <cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#record.url#" />, <cfqueryparam cfsqltype="CF_SQL_VARCHAR" value="#record.refer#" /> ) </cfquery> </cfloop>
This database was invaluable in recent request analysis, but it was still *massive*, and was very tough to isolate any one thing that looked suspicious. Back on the linux server, I used a command to look at recently changed files within the web root(s):
find . -mtime -1 -print
This command showed me any files in a subfolder of my current location, that had been edited in the last 1 day. I scanned down the list to find anything suspicious. One thing that stood out was a set of files owned by the apache // apache user, one called ss.php I switched back to my MS SQL query editor, and searched through the requests for any going to ss.php. One suspicious IP address made multiple requests to this file, and a quick lookup on http://dnstools.com/ showed that this was coming from Saudia Arabia (most likely a compromised host there, bouncing through from another location).
After finding this, I queried my imported apache database for all requests from that IP. This showed me several requests to a PHP admin tool that one of the customers loaded into their web site, with a referrer of google. Yahh, now I know the entry point to this server – a PHP admin tool with no authentication on it, found by a random user doing a google search for just such a windfall. This IP query also showed all url parameters, so I was able to review each thing this hacker did on the system, and remove all effected files one by one.
Of course, this has led us to further securing the server, restricting PHP access, locking down SSH access, and more. I hope this general strategy helps someone figure out what is going on when they get hacked!