Gav's Blog

I was catching up on Twitter today, and saw some references to optimising Adobe Lightroom catalogues in #Lightroom (I’ve now found the original¹ posts² and had a read of them – there is more good stuff to try out. I’ve incorporated the previews and caching as well in my cleanup today). This struck me as something I was due to do as I knew my catalogue database was well over 250MB, and when something gets that big it needs a little maintenance. This prompted me to poke around a little and have a good cleanout.

Here’s what I did – some for speed, others just freeing up space.

0. Complete a Time Machine backup first.

Naturally, I wanted to make sure a had a backup before I did anything destructive – such as deleting files – so I forced Apple’s Time Machine to complete a backup. With that out of the way, I could start tidying up. If you’re on Windows, you should do a backup first before you do anything else.

1. Check the Backups directory.

I have Lr setup to backup my catalogue weekly – this keeps a fairly good trail of backups in case something goes wrong. It hasn’t yet. However, after 2 years of Lightroom use, my directory of Lightroom catalogue backups had blown out to 13.52GB! There were a couple of things I did to trash some of the existing backups.

• I removed all Lr v1 backups. I knew the date that I upgraded (by the last modified date of my old v1 catalogue file). I deleted all the v1 backups and this freed up 8.71GB. Since the upgrade produced a new catalogue name (with a -2 appended), I was able to confirm that I was only deleting v1 backups.
• For all months bar the last one, I decided that I only really needed to keep the most recent backup in each month, rather than sometimes all 4 or 5 in a month. I deleted all but the most recent and freed up another 2.64GB of space.

This brought the Backups directly back to a rather svelte 2.42GB.

2. Optimise Catalogue

I hadn’t seen this option before today, so I thought I’d check it and see how it goes. It is accessible via Lightroom > Catalogue Settings, and is a simple ‘Relaunch and Optimise’ button. My catalogue was around 18k photos, and was 262.9MB before optimisation. My Previews file was 101.7MB. Note this figure. After expecting it to take say quarter of an hour to process, I was surprised at how little time it took.

The resulting catalogue saw ~20MB removed and the overall file size brought down to 241.5MB.

I did get a shock when I saw my Previews file though, it had blown out from a lowly 101.7MB to a massive 7.74GB! However, I think this may have been a case of the package (Previews are a package on OS X) under-reporting its true size. I don’t think the optimisation process was busy enough to create 7GB of previews in such a short time! Therefore I think the optimisation actually made the package report the correct size. So I’m not treating this as a ‘loss’ of 7GB

3. Deleting Cruft

In the same directory as my main catalogue, I noted I still had my unloved v1 catalogue, as well as a couple of ‘Temporary Folders’ Lightroom had created that contained a couple of images that I no longer needed. Since these were already backed-up, I just deleted the old catalogue and these temp directories.

4. Rendering Previews

Quite a few of my previews had already been rendered, but I decided to check my rendering settings, and force Lr to render the rest. Library > Previews > Render Standard-sized Previews. I expect I may be facing a 20GB+ Preview file by the time its finished!

5. Adobe Camera Raw cache

As per Lightroom Queen’s article, I went and configured the cache on a separate internal hard drive. I created a new directory called /Cache/Adobe Camera Raw in the root of a second ‘working’ drive and added this in the Lightroom > Preferences > File Handling dialog. I choose the new directory I had created, and upped the somewhat meagre 1GB cache to a more workable 10GB.

Summary

All up, it has been a good little cleaning effort that netted me another ~11.5GB of storage. Which I think I will probably lose to previews. Lightroom doesn’t seem noticeably faster on open, but it does seem quicker to quit. I haven’t spent any other time in it this evening yet to comment on general usage. I’ll report back once I have completed the preview rendering and had a chance to spend a little more time using the optimised Lightroom.

1. Relaunch and Optimize Your Catalog Today – Adobe Photoshop Lightroom Killer Tips [↩]
2. Hurry Up Lightroom – Lightroom Queen [↩]

After hearing about one of our GPS Society members losing their data in a computer malfunction tonight, I’ve decided to sit down and flesh out some thoughts on developing a good backup strategy for your computer(s). This is one of those get a round ‘tuit posts that I’ve been meaning to do after seeing people caught by HD failures on the Digital Photography School forums.

The topic of developing a good backup strategy for your computer surely makes most peoples eyes glaze over. It is decidedly unsexy until such time as you need it. Of course, by then it is too late. I’m hoping to combine some of my IT, risk and emergency knowledge to provide some insight into develop a suitably robust backup strategy.

If the consequence is lost data, what are the risks?

When developing a backup strategy, it is important to have a good understanding of how data can be lost – the risks – so that we can create a simple yet comprehensive plan to backup our data that accommodates the many different ways data can disappear.

So, lets pick a few. I’ve named them L1-L5 where ‘L’ is for loss.

• L1 Loss of computer (e.g. theft, smoke or water damage; electrical surge from computer power supply)
• L2 Filesystem accidents – formatting of filesystem, deletion of files, data corruption
• L3 Malicious software – formatting, deletion, or encryption of files with an unknown key (e.g. encrypt and extort)
• L4 Mechanical failure of the hard drive (the dreaded clunking sounds)
• L5 Loss of home containing computer (e.g. fire, earthquake, flood, landslide)

Whilst not comprehensive, this includes a good range of different issues we may face where a backup would be very handy and save us a lot of time, and potentially money. If we can come up with something that protects us from these losses, we should be doing pretty well.

What we need to do now, is look at various means available to backup data, and then create a quick matrix comparing each type of backup, and which losses it may/may not protect us from.

Firstly, let’s identify a number of backup solutions.

• S1 Backup to CD/DVD/HD and store on site
• S2 Backup to external HD on site
• S3 Backup to internal HD
• S4 Backup to other computer at home
• S5 Backup to Internet (service or web host)
• S6 Backup to CD/DVD and store off site
• S7 Backup to HD and store off site

Now, all these solutions are not equal. What we need to investigate now is which type of loss a given backup solution can protect against. I’ve created a sample table below to give you some idea of how it all comes together.

We start with a grid comparing types of loss and solutions. A green box means the solution generally prevents that type of loss, and red box that it generally doesn’t protect, and an orange one means that it may provide some protection.

Next, we compare solutions with various costs and constraints – in this grid green means it isn’t really a cost/constraint, red means it is a cost/constraint, and again, orange means it might be a cost/constraint.

In this example matrix, two points stand out.

• backing up to an internal hard drive does not provide much protection against data loss
• quite a few backup solutions do not protect against major losses such as the loss of a home from fire.

Additionally, every backup solution has a number of costs and/or constraints on its operation. The next step has been to add some cells that identify some of the more common costs and constraints associated with each solution.

What we can see is that there is no single perfect solution. We could extend this further and add a grid outlining some of the benefits of each backup solution – they all have some – and this would also further educate us in the development of our backup strategy.

Now, we’ll use this grid to look at selecting a couple of complementary backup solutions that avoid each others weakness.

Personally, I’m a fan of backing up my home computer using Time Machine on a Mac to an external USB hard drive (effectively S2). As you can see from the matrix, this protects me against most of the common losses, except the rather catastrophic loss-of-home. Clearly then, I can select an Internet or off-site solution as well that will provide me with more complete data protection than just backing up to an external hard drive.

Quite a few people will look at the Internet backup option (S5) and think that it looks pretty good, but be warned, there are some issues that you may face including – the speed of your internet connection when backing up files to remote servers, ongoing service fees, and potential privacy risks by storing you files on a remote server of a business.

I’d recommend selecting solutions so that you can meet the follow three four requirements.

1. You should have at least three copies of your data (source + two backups).
2. At least one backup must be reasonably current and disconnected from the computer most of the time (except when a backup is being made).
3. At least one backup file must be offsite.
4. At least one backup should be incremental.

Ian (in the GPS forums) made a good point about incorporating incremental backups into the process. Broadly speaking, there are two types of backups, full (where everything is copied at once) and incremental (where only the files that have changed since the last backup are copied). When doing incremental backups, the first backup is a full backup, and then incremental backups take place from there on. Time Machine is a good example of incremental backup software – every hour it backs-up any changed files.

As I’m not that keen personally on online backups, I’d recommend one of the following as the minimum. There is nothing wrong with making more copies on CD/DVD media to supplement the main backup solutions.

• external hard drive onsite + DVD media offsite (affordable setup)
• external hard drive onsite + external hard drive offsite (same sizes, switch them once a week or month, expensive setup)
• synchronise files between two home computers on network + external hard drive offsite (utilise existing hardware and provide backups of both computers)

There are three other tips to provide as well:

• If you use backup software, keep a copy of the install media (and licence key if appropriate) with the backups
• If you need quick access to data upon failure, make sure that at least one of your backups uses a very accessible filesystem on external hard drives (CD/DVDs are good as they generally use filesystems that are accessible in any computer). This means you can literally plug them in and access key files without having to perform a software installation and full restore
• AND TEST THAT YOU CAN ACCESS BACKED-UP DATA and/or RESTORE FROM BACKUPS

Finally, as you should have a GetAway Kit for natural disasters and the like, in addition to your other important paper information such as identification, policies and photos – you should also include a backup of your data in the kit. If you haven’t got a GetAway Kit, then now is a good time to learn about how to get ready!

As I write this, my home Mac has just about finished receiving a hard drive upgrade. I had approach the 250GB hard drive’s limit and was needing something far bigger – mainly for photos and movies from my camera.I had read online a few blogs that performed the upgrade by simply making sure that the Time Machine backup was up-to-date, and then removing the old hard drive, installing the new one, booting the OS X installer DVD, formatting the drive and restoring from backup. Sounded like a great option so I thought I’d give it a go to install a nice shiny 1TB hard drive.This blog post noted that the restore process was extremely slow. I had a quick restore with around 230GB being restored in around 2.5 hours. The expected time was a lot slower initially, I think this was because the system files were copied first, and this included a lot more small files that have more overhead associated with reading and writing than say transferring a digital photo or movie – e.g. one 10MB file is faster than transferring 1000 1KB files.It is just booting now, so I’ll know very soon whether it worked or not… stunning it has appeared to work perfectly. Now I just have to wait for Spotlight to re-index everything. But other than that, it was a painless upgrade.It also served an excellent dual purpose of actually testing that the restore process works. Very useful!Now I need to look at setting Time Machine to backup to multiple drives. It sounds like it can be done.