With the abundance of malware and ransomware it’s absolutely necessary that we take the time to examine our virtualization host and backup structures.

Volume Shadow Copies
- Obviously not a “backup”
- Most ransomware kills all shadow copies available on a system
Backup to Disk/NAS/SAN

Disk Rotated
NAS/SAN needs to be streamed off-site
Most ransomware is backup product aware and hunts the backups down

Backup Structure

Current, off-site 1, off-site 2, 6 Month, 12 Month, ETC…
GFS: Grandfather, Father, Son
Cloud Tier
Air-gap(s) and/or Cloud-gap(s)

Backup Access

Management Console
Backup Files via Server
Backup Files via UNC

Backup Software Resilience

Veeam and ShadowProtect backup files are independent of the backup software
Veeam has the freebie Veeam Explorer to mount backup files (not tied to the software being installed ever)
ShadowProtect requires their software to be installed before a backup can be mounted (freebie download though)
Both require a decryption password (we always set this up)

With our last mile issue up here in Canada we are mindful about anything Cloud since most upload speeds are not capable enough. The download speeds are mostly not capable of a decent recovery time either.

Now, what is _the most important_ aspect to our backup setup?

Anyone?

It must be isolated and air-gapped!

Isolation and Air-Gaps

What does that mean?

First, that means that at no point in the backup structure can anyone on the production network have access to the backup files without at least a credentials prompt. The credentials for READ access should be closely managed with no MOD access via anywhere beyond the local console.

The backup software console should be available via a Privileged Access Workstation or jump server only. While inconvenient, there needs to be segmentation between the production network(s) and the backup setup. Period.

Second, the backup files need a protection policy set up to not be deleted for a period of time. In the DAT/DLT tape days we would set a retention period that blacked out overwrites to tapes to preserve the data on them just-in-case. From there, a periodic FULL backup should be permanently archived allowing for future access.

Some pointers for isolating things:

NAS/SAN
- Veeam/ShadowProtect user with unique pass phrase and MOD on the repository root folder
  - Other than the NAS/SAN backup folder Admin account no other user account is set up with access
  - Turn on the NAS/SAN Recycle Bin if it has one!
    - Most ransomware creates a new file then deletes the old one
    - Create a separate username and folder structure for user facing resources!
Veeam/ShadowProtect
- Network destination set up with a unique user account
- Recovery user subset with READ only access to the backup files
Veeam/ShadowProtect On-Premises Backup Destinations

Encrypted AES 256-bit with a long pass phrase
Rotated on at least a weekly basis
Periodic FULL backup for GFS

USB HDD

Root folder share given MOD to backup user and Domain\Users REMOVED

Whether domain user on a cluster or local user on standalone Hyper-V
Blog post: Folder Permissions: How To Properly Disinherit Permissions

Rotated Frequently

Cloud Backup

Must provide a no delete window!

Some Veeam CloudConnect services offer a protection window

Backup Console

All managed backups are set up to be accessed via dedicated backup admin account only

No repository, whether NAS/SAN or USB HDD has Users MOD rights
All repositories must be protected with a username and password I.e. no anonymous access

Essentially, we have a series of hops to get to the backup files for management: Tenant User Admin –> PAW/Jump Station –> Backup Files.

Or, for recovery: Tenant Recovery User –> UNC –> Credentials –> Backup File Mount via Recovery/Encryption Password –> READ Access.

Hyper-V Standalone and Cluster Node Protection

Protecting the virtualization hosts can be a bit of a challenge depending on the setup. We always try to operate under the KISS principle. Keep It Simple S_____. 😉

To do so is not that difficult, but in some cases requires a cultural shift in mindset to make it happen.

Hyper-V Standalone

Recently, we know of a domain joined standalone Hyper-V server get hit by ransomware thus the entire setup was held hostage.

It has been our long standing practice to leave standalone Hyper-V hosts in workgroup mode. This provides a layer of protection from an encryption event at the guest level. So, in the above case our host would not have been touched by the ransomware from the guests.

Cluster Node Security

In a cluster setting we’ve started setting up the cluster nodes with enough local storage to host the Hyper-V host operating system and a local guest.

Lately, that means that at least two nodes have a pair of Intel SSD D3-S4610 series 240GB RAID 1 set up with two partitions:

110GB for host OS
110GB for guest a VM

The guest VM is set up as an Active Directory domain controller with its own DNS dedicated to that cluster. In multi-cluster settings we would have a cluster DC running on at least two of the clusters.

Essentially, we are treating all virtual machines running on the clusters as tenants thus providing a layer of protection for the cluster itself.

Segmenting the network via VLAN is another option to carve off the cluster network completely from the tenant/production network.

We find that Veeam facilitates backing up in cluster settings best as we can set up a dedicated username and password on the cluster domain and on the “tenant” network to allow for in-guest backup acquiescence (Exchange and SQL log consolidation triggers). We can then run the Veeam setup in a workgroup setting, also as a tenant on the same cluster, to provide yet another layer of protection for the backups.

Backup Management

And finally, some of the more obvious aspects around backups and domain operation in general:

Users are Standard Users on the domain

If they absolutely need local admin then use Microsoft’s Local Administrator Password Solution (LAPS)
Domain User accounts have _NO_ access to any aspect of the backup structure
- None, Nada, Zippo, Zilch! 😉

Domain Admin accounts should have no access to any aspect of the backup system
- Use KeePass or other password vault to protect all credentials
- Client sites should have one or two users that know these credentials

Access via UNC
- Use a READ account to access and _do not save_ the credentials!

Backups are managed by us, spot recovered by us, and quarterly bare metal/hypervisor restored by us

No client intervention other than perhaps the off-site rotation (we do this too)
Cloud backups are managed by us or even served by us (Veeam CloudConnect)

One bit of information that’s _really important_ to ask any backup vendor: Can the backup files be restored if the original backup server(s) got wiped?

We know that Veeam and StorageCraft files can be mounted for access but what about the other vendor’s products?

Is Restore no longer possible if the original backup servers are gone? There may be a few products out there that may indeed have this fault!

Protection Rationale

So, what spawned this blog post?

Besides the above Hyper-V host getting encrypted?

Hearing of a ShadowProtect destination NAS getting wiped out by ransomware. This should not be possible on our managed networks ever!

Why protect the virtualization hosts and the backups from the guests?

If we do end up in a situation where the guest systems get encrypted or wiped we have a layer of protection in place that would allow us to fully recover the network.

GFS (Grandfather, Father, and Son) backups give us access to a backed up domain controller if the need exists to step-back in time for recovery purposes.

Protecting a Hyper-V Host and Backup Repository from Malware and Ransomware

Isolation and Air-Gaps

Hyper-V Standalone and Cluster Node Protection

Hyper-V Standalone

Cluster Node Security

Backup Management

Protection Rationale

Further reading

Leave a comment Cancel reply

2 thoughts on “Protecting a Hyper-V Host and Backup Repository from Malware and Ransomware”