With the abundance of malware and ransomware it’s absolutely necessary that we take the time to examine our virtualization host and backup structures.
- Volume Shadow Copies
- Obviously not a “backup”
- Most ransomware kills all shadow copies available on a system
- Backup to Disk/NAS/SAN
- Disk Rotated
- NAS/SAN needs to be streamed off-site
- Most ransomware is backup product aware and hunts the backups down
- Current, off-site 1, off-site 2, 6 Month, 12 Month, ETC…
- GFS: Grandfather, Father, Son
- Cloud Tier
- Air-gap(s) and/or Cloud-gap(s)
- Management Console
- Backup Files via Server
- Backup Files via UNC
- Veeam and ShadowProtect backup files are independent of the backup software
- Veeam has the freebie Veeam Explorer to mount backup files (not tied to the software being installed ever)
- ShadowProtect requires their software to be installed before a backup can be mounted (freebie download though)
- Both require a decryption password (we always set this up)
With our last mile issue up here in Canada we are mindful about anything Cloud since most upload speeds are not capable enough. The download speeds are mostly not capable of a decent recovery time either.
Now, what is _the most important_ aspect to our backup setup?
It must be isolated and air-gapped!
Isolation and Air-Gaps
What does that mean?
First, that means that at no point in the backup structure can anyone on the production network have access to the backup files without at least a credentials prompt. The credentials for READ access should be closely managed with no MOD access via anywhere beyond the local console.
The backup software console should be available via a Privileged Access Workstation or jump server only. While inconvenient, there needs to be segmentation between the production network(s) and the backup setup. Period.
Second, the backup files need a protection policy set up to not be deleted for a period of time. In the DAT/DLT tape days we would set a retention period that blacked out overwrites to tapes to preserve the data on them just-in-case. From there, a periodic FULL backup should be permanently archived allowing for future access.
Some pointers for isolating things:
- Veeam/ShadowProtect user with unique pass phrase and MOD on the repository root folder
- Other than the NAS/SAN backup folder Admin account no other user account is set up with access
- Turn on the NAS/SAN Recycle Bin if it has one!
- Most ransomware creates a new file then deletes the old one
- Create a separate username and folder structure for user facing resources!
- Veeam/ShadowProtect user with unique pass phrase and MOD on the repository root folder
- Network destination set up with a unique user account
- Recovery user subset with READ only access to the backup files
- Veeam/ShadowProtect On-Premises Backup Destinations
- Encrypted AES 256-bit with a long pass phrase
- Rotated on at least a weekly basis
- Periodic FULL backup for GFS
- USB HDD
- Root folder share given MOD to backup user and Domain\Users REMOVED
- Whether domain user on a cluster or local user on standalone Hyper-V
- Blog post: Folder Permissions: How To Properly Disinherit Permissions
- Rotated Frequently
- Cloud Backup
- Must provide a no delete window!
- Some Veeam CloudConnect services offer a protection window
- Backup Console
- All managed backups are set up to be accessed via dedicated backup admin account only
- No repository, whether NAS/SAN or USB HDD has Users MOD rights
- All repositories must be protected with a username and password I.e. no anonymous access
Essentially, we have a series of hops to get to the backup files for management: Tenant User Admin –> PAW/Jump Station –> Backup Files.
Or, for recovery: Tenant Recovery User –> UNC –> Credentials –> Backup File Mount via Recovery/Encryption Password –> READ Access.
Hyper-V Standalone and Cluster Node Protection
Protecting the virtualization hosts can be a bit of a challenge depending on the setup. We always try to operate under the KISS principle. Keep It Simple S_____. 😉
To do so is not that difficult, but in some cases requires a cultural shift in mindset to make it happen.
Recently, we know of a domain joined standalone Hyper-V server get hit by ransomware thus the entire setup was held hostage.
It has been our long standing practice to leave standalone Hyper-V hosts in workgroup mode. This provides a layer of protection from an encryption event at the guest level. So, in the above case our host would not have been touched by the ransomware from the guests.
Cluster Node Security
In a cluster setting we’ve started setting up the cluster nodes with enough local storage to host the Hyper-V host operating system and a local guest.
Lately, that means that at least two nodes have a pair of Intel SSD D3-S4610 series 240GB RAID 1 set up with two partitions:
- 110GB for host OS
- 110GB for guest a VM
The guest VM is set up as an Active Directory domain controller with its own DNS dedicated to that cluster. In multi-cluster settings we would have a cluster DC running on at least two of the clusters.
Essentially, we are treating all virtual machines running on the clusters as tenants thus providing a layer of protection for the cluster itself.
Segmenting the network via VLAN is another option to carve off the cluster network completely from the tenant/production network.
We find that Veeam facilitates backing up in cluster settings best as we can set up a dedicated username and password on the cluster domain and on the “tenant” network to allow for in-guest backup acquiescence (Exchange and SQL log consolidation triggers). We can then run the Veeam setup in a workgroup setting, also as a tenant on the same cluster, to provide yet another layer of protection for the backups.
And finally, some of the more obvious aspects around backups and domain operation in general:
- Users are Standard Users on the domain
- If they absolutely need local admin then use Microsoft’s Local Administrator Password Solution (LAPS)
- Domain User accounts have _NO_ access to any aspect of the backup structure
- None, Nada, Zippo, Zilch! 😉
Domain Admin accounts should have no access to any aspect of the backup system
- Use KeePass or other password vault to protect all credentials
- Client sites should have one or two users that know these credentials
- Access via UNC
- Use a READ account to access and _do not save_ the credentials!
- Backups are managed by us, spot recovered by us, and quarterly bare metal/hypervisor restored by us
- No client intervention other than perhaps the off-site rotation (we do this too)
- Cloud backups are managed by us or even served by us (Veeam CloudConnect)
One bit of information that’s _really important_ to ask any backup vendor: Can the backup files be restored if the original backup server(s) got wiped?
We know that Veeam and StorageCraft files can be mounted for access but what about the other vendor’s products?
Is Restore no longer possible if the original backup servers are gone? There may be a few products out there that may indeed have this fault!
So, what spawned this blog post?
Besides the above Hyper-V host getting encrypted?
Hearing of a ShadowProtect destination NAS getting wiped out by ransomware. This should not be possible on our managed networks ever!
Why protect the virtualization hosts and the backups from the guests?
If we do end up in a situation where the guest systems get encrypted or wiped we have a layer of protection in place that would allow us to fully recover the network.
GFS (Grandfather, Father, and Son) backups give us access to a backed up domain controller if the need exists to step-back in time for recovery purposes.
- Backing up at least the PDCe right?
- Maersk Notoptera Horror Story
- This story is a reminder of the RAID is not a backup conversations back in the day!
- Wolters Kluwer/CCH Outage
- Krebs on Security: What’s Behind the Wolters Kluwer Tax Outage?
- This one was bad. They had most services back on May 12th but ended up taking it all back down again on the 13th!
- One has to wonder whether their DCs were backed up?
- This Removeddit thread has some interesting posts on it (neat tool too)
- Veeam: How to follow the 3-2-1 backup rule with Veeam Backup & Replication
- IT Security Related
- Krebs on Security: Experts: Breach at IT Outsourcing Giant Wipro
- Krebs on Security: Payroll Provider Gives Extortionists a Payday
- Vice: Hackers Hijacked ASUS Software Updates to Install Backdoors on Thousands of Computers
- Ars Technica: Hackers breached 3 US antivirus companies, researchers reveal
- Subsequent posts found about reveal McAfee, Symantec, and Trend
- Our Cloud Security Guides
- Architecture references
- Segmentation recommendations
There’s no lack of stories today of all sorts of malware and encryption events in companies of all sizes.
What’s becoming apparent is larger cloud vendors, such as Wolters Kluwer/CCH, that architect flat network designs putting cloud services systems on their production network are destined to fail.
What’s scary, is hearing about companies that have, “given up trying to eradicate the virus infections as it looks like it’s not doing anything bad anyway.”
What’s even sadder is Emotet using EternalBlue (SMBv1 vulnerability) to travel laterally in networks when the patch has been available for over two years now.
TIP: Domain trust gone with no recovery possible from backup? Side-by-side migrate using ForensiT’s User Profile Wizard