In May of 2020, Blackbaud discovered they were the victims of a ransomware attack. Among the items stolen by the hacker included unencrypted database backup files. The subsequent data breach impacted thousands of nonprofits and countless donors.
That event should serve as a wake-up call for nonprofits everywhere to the business risk of unencrypted database backup files.
If your organization uses Blackbaud CRM, you should be especially concerned about your risk of an identical data breach just like the one that hit Blackbaud.
Most organizations focus security resources on protecting one thing - the production instance of the primary database. However, the very same data that resides in the well guarded database also resides in backups of that database.
Hackers tend to go for easy targets. Unencrypted database backup files amount to treasure troves of data for hackers. Even if they don't directly use the data, hackers use the threat of exposure to extort huge ransoms.
Allow me to offer some perspective on security and risk. Risk is a business and legal matter, not a technical one. Risk says, "What's the likelihood this can happen, how vulnerable are we, and what would be the cost if it were to happen?"
Let's say there are 10,000 possible security risks in a complex system like that of Blackbaud. Separate out the single security risk of a file repository containing thousands of unencrypted database backup files, each with the personal data of thousands of donors.
That risk is like the Grand Canyon.
Next, combine the security risk of all the other 9,999 possible bad things that could happen.
That combined risk is like a sinkhole in Florida (hey, it’s a really, really big sinkhole!)
It’s one thing to get hacked. Organizations carry cyber insurance to help mitigate that risk. It’s quite another to maintain a dragon’s horde of unencrypted database backup files, teeming with PII, all sitting together in a file directory ripe for plunder.
For those of you doing risk analysis, once you discover a tremendous risk like that which was staring you in the face all along...well, trust me, it's the stuff of nightmares. Our real culprit in this caper was not the hacker, but rather the failure to properly assess the risk from those unencrypted database backup files.
Any organization that uses Blackbaud CRM should be asking themselves - where is our Grand Canyon? How many such troves of backup data do we also keep, even informally?
Organizations that use Blackbaud CRM are susceptible to a copycat hack for three key reasons:
Blackbaud CRM deployments typically have multiple secondary environments with a full installation of the software and a copy of the database. For example: Staging, Training, Development, and Production Sandbox. These secondary environments get refreshed with a database backup from Production.
If you were to take an inventory of all the unencrypted database backup files being kept informally across all these systems, what would you find?
For each restored database, it's likely your inventory would also uncover at least one copy of an unencrypted database backup file used to restore the database within each of these secondary environments. Once restored, the database backup file doesn't get deleted. Who knows when a backup from a particular point in time may need to be tested?
Backups, it turns out, are terribly handy.
Let's face it, getting database backup copies later on may not be practical or convenient from whomever is in charge of handing out backups. Best to hang onto a database backup copy once you get your hands on one. Don't be surprised if further digging unearths informal repositories kept by various team members, all filled with backups from different times for various needs.
Do you have any developers doing custom programming? Or perhaps team members building out your data warehouse or reporting system? Each team member may have their own database backup copies in any number of places. Now that everyone is working from home, who knows how many places people have squirreled away unencrypted database backup files.
And that's just with your own internal team.
It's not uncommon for backups of Blackbaud CRM databases to be shared with outside vendors and consultants. A copy would likely be found on the FTP server used to transfer the data - either your FTP or their FTP - or both.
Plus the vendor made at least one other copy of your database backup by copying the file from the FTP over to the server of its ultimate destination. Maybe the vendor also maintains an ad hoc file system of customer backups like Blackbaud does? Perhaps that file system uses cloud storage like Dropbox or One Drive? It's not hard to imagine how copies of your database backups could easily multiply into copies of copies on other systems completely outside your control.
Once shared, are those database backup copies ever deleted? Most likely, no. Without an explicit policy delineating expectations for deletion, copies can proliferate and exist for years. At what point does a data breach become an inevitability?
Organizations that use Blackbaud CRM face a sobering truth: they have lost control of their own data.
Since backups are handy and sharing has become routine, trying to stop that behavior is fighting a losing battle. Instead, the database backup files must be encrypted. If a hacker were to gain access to those same database backup files and they are encrypted, the hacker has nothing.
Backup encryption, a feature built into Microsoft SQL Server, would solve the Grand Canyon of risk. SQL backup encryption was added to Microsoft SQL in April 2014 with the release of SQL Server 2014 (12.x). Multiple encryption algorithms are supported, up to AES 256 bit, which is considered the strongest.
It's free, and very straightforward to setup. Why is encryption still not being used after so many years?
Encryption is inconvenient.
Technology managers tend to want to keep things simple. They care less about risks and more about efficiency. Even if it's free and easy, encryption is an extra step, and therefore perceived to be less efficient.
Presidents and board members oversee insurance and risk, yet rarely understand technology nuances. I'd wager that the vast majority don't even understand what happened in the Blackbaud hack, much less how to combat it.
The tech folks don't want to be bothered. And the leaders figure that if it were important, their fellow leaders at other nonprofits would be doing it already.
Nobody mandates encryption for database backups, much less advocates for it. Ergo, encryption must not be that important.
And here we have the dilemma.
So how do we overcome it?
Concourse subscribes to the “Never Nude” policy when it comes to SQL Server database backups for Blackbaud CRM. A nude SQL backup means a database backup file that is not encrypted.
It's not good enough to take an existing unencrypted database backup file then apply encryption to it after the fact. For example, using 7-Zip to encrypt a database backup. Such a method would require starting with an unencrypted nude backup to begin with. It also ends with another nude backup on the target system once you unzip it.
As you can see, using encryption after the fact does nothing to help reduce the risk of exposure. We still end up with two nude backups on two different systems.
Even one nude backup is too nude.
Never nude means just that - no unencrypted database backup files exist at any point in time.
If there are no nude backups, we worry less about data being stolen due to some kind of accidental exposure. If our data backups are always encrypted, the data remains protected even if the files are exposed in a data breach.
If your backup process generates a single uncompressed nude database backup file, you can and should do better. The method outlined below not only adds encryption, it speeds the overall transfer.
First, use compression to reduce the overall size of the backup. Next, break the backup into multiple files, as this allows the backup to be generated much faster than a single file. Plus, once the backup is ready, files may be transferred simultaneously in parallel, thus shortening the overall transfer window. In the end, you'll find that any existing process that requires regular copies of the database benefits from this approach.
And if you're migrating your Blackbaud CRM database to a new hosting environment, you can do so much faster, more efficiently and, of course, securely.
Step 1: Create a Database Master Key (probably not needed)
To start with, the SQL Server requires a Database Master Key for the master database. However, the OLTP SQL Server housing the Blackbaud CRM database should already have this key. Nevertheless, I'll show this step for the sake of completeness.
If a Database Master Key already exists (which it probably does), you'll see this error message.
Don't drop the Master Key! As expected, you've already got a Master Key. Hopefully somebody has the key's password saved someplace safe.
Step 2: Create a Certificate
Run this against the master database to create the Certificate that we'll use for encryption.
Step 3: Create an Encrypted Backup
Run this to create an encrypted backup file.
Note that the backup is split into multiple files to speed up the backup process. Other settings such as BufferCount, BlockSize, and MaxTransferSize further optimize the backup speed. Adding compression reduces the size of the backup files, but takes a fair bit of CPU resources.
You may wish to increase the number of files for very large databases. The maximum number of files is 64.
Step 4: Copy Certificate (once per year)
Once the Certificate is in place, it may be used over and over again to encrypt and decrypt database backups.
Should you wish to restore the database to a different SQL Server, export the Certificate and Private Key, and then import them on the target server. The Certificate expires after one year, so you'll only need to perform this step again after the Certificate expires a year later.
To protect the export of the Certificate and Private Key files, generate a random 32-character password. We prefer 1Password where we could also save the password.
Save the Certificate and Private Key to files that are protected by the password generated above.
Next, copy these files to the target server and grant SQL Server access to read the files by running the following command from an Administrator command prompt.
Finally, run this script against the target SQL Server to import the Certificate and Private Key. Of course, include the same password used to export the Certificate.
Step 5: Restore Database
Restore the database as you normally would using SSMS. If the Certificate was added properly, SSMS will automatically figure that out for you.
As demonstrated above, the process of adding encryption to Blackbaud CRM database backups is simple.
Organizations that use Blackbaud CRM tend to be large. Such organizations possess the capacity, the resources, and the wherewithal to implement encryption for database backups.
Nonprofits tend to look to one another for guidance on how to introduce and adopt new technologies. With a groundswell of awareness, new encryption policies and practices could be introduced fairly quickly.