New cameras, raw image support and Adobe software

In yesterday’s post about my Nikon Coolpix P7100, I mentioned that I’d had to invest in new software when I bought a new camera (as if a new camera wasn’t a big enough expense). As I’m reading about Adobe’s beta of Lightroom 4, I thought it was probably worth eleborating on this, as once of my friends also had a similar experience last year – and it’s something that pretty much all Adobe users will come across if they buy new cameras and shoot raw images.

Whilst some might argue that there is no noticable difference between a fine JPEG image and something generated from a raw file, the simple fact is that multiple edits on compressed files will lead to a gradual degradation in quality. I prefer to capture in the highest possible quality, work on that, and only save to .JPG at the end of my workflow (typically before uploading to the web, or sending to a lab for printing).

So, when I bought the P7100, I found that I needed the latest version of Adobe Camera Raw to read the .NRW (raw) images that it created. That wouldn’t have been a problem, except that Adobe Camera Raw 6.x doesn’t work with the software I was using at the time – Adobe Lightroom 2.x and Photoshop CS4. So I purchased Lightroom 3, although I have to make do without editing my P7100’s images in Photoshop – it’s just too expensive to upgrade at the drop of a hat.

It’s not just me – a friend who bought a Canon EOS 600D last year suddenly found that she needed to upgrade from Photoshop Elements 8 to Elements 9 in order to work with her raw images (she could also use Apple iPhoto… but it’s seriously limited for anything more than the most basic of edits).

With the coming of Lightroom 4/Photoshop CS 6, I guess we’ll see Adobe Camera Raw 7 and, if past history is any judge of what’s coming, I’ll expect that will not work with Lightroom 3 or CS 5. In effect Adobe is forcing us to upgrade their software, in order to use the raw capabilities of a new camera.

Obviously, Adobe would like us to all use its digital negative (.DNG) format for raw images (indeed, Adobe offers a free DNG converter) but, given that neither Canon nor Nikon – the two largest camera manufactirers – are showing any sign of moving away from their proprietary formats, that doesn’t help a lot.

There may be other tools to convert from the P7100’s raw images to .DNG or .TIF for working on, but I can’t help feeling Adobe’s decision to tie Camera Raw to certain releases of its software is a retrograde step, and it won’t encourage me to upgrade my software again until I am forced to (probably by a new camera purchase…).

Camera raw support for my Canon Digital Ixus: enabling DNG support with CHDK

Late last year, Garry Martin alerted me to an alternative firmware for certain Canon cameras – the Canon Hackers Development Kit (CHDK) – and I’ve been running it on my Canon Digital Ixus 70 ever since.

Importantly, CHDK makes no modifications to my camera’s firmware. The appropriate version for the camera model is downloaded and placed in a folder on the SD card that the camera uses and, when the card is write protected, this firmware is loaded at startup. If CHDK is missing or the card is not write protected, then the normal camera firmware loads. In addition to the extended functionality afforded by CHDK (which is significant – Lifehacker wrote a review of CHDK last year), I have full access to the standard camera features but the main advantages I find from CHDK include camera raw support, a live histogram and better battery information (I may also take a look at using it for time lapse photography at some point).

One of the frustrations I’ve had with CHDK is that the raw format it produces is not recognised by any of the major image editing applications (for me, that means using Adobe Camera Raw to interface with Photoshop CS4 and Lightroom 2 on a Mac). I tried installing an application that should convert these files to Adobe Digital Negative (.DNG) format (DNG4PS2) but the pre-built Mac version is known to be unstable on Mac OS X 10.5 (Leopard) and I was unable to make it compile using the latest source. There seems little point in running it on another operating system when I do all of my digital media work on the Mac, so I went back to getting the camera to store raw files in .DNG format at capture.

Unfortunately, each time I tried this, I was greeted with a message which said something like Cannot load/CHDK/badpixel.bin. Thanks to a comment from James W Manning on Flickr, I was able to work my way through creating this file: downloading a program to analyse an existing raw image taken with the same camera and identify any bad bits to create the required badpixel.bin file (I did need to run this on Windows though – the command was show_bad_b.exe rawfile.crw), which I could then copy to the CHDK folder on my SD card and enable DNG support.

Now my £130 compact camera takes raw images in .DNG format (as well as some JPEGs for reference) and I can work with the raw files just as I do with the ones produced by my DSLR. Of course, the tiny sensor means that the 7 million pixels on my Ixus 70 are inferior to the six million pixels on my old D70 (and way behind my full-frame D700 – more on that in another post!) but the additional flexibility is useful – as is the knowledge that I have the actual data that was recorded by the camera sensor, rather than with any post-processing in camera (e.g. boosted saturation that’s typically used with the compressed JPEG images).

Camera raw support in Windows Vista and later

Most of my digital photography workflow takes place on a Mac, where I use Adobe Camera Raw and Bridge/Photoshop CS3 to handle camera raw images.  With my recent purchase of a netbook (which is small enough and light enough to take out with me on a shoot – and less expensive than a dedicated storage device like an Epson P-7000), it would be useful to view the images in Windows but the Microsoft Raw Image Thumbnailer and Viewer for Windows XP has not been updated since 2005 and is not compatible with Windows Vista or later.

I did wonder if the technology had been absorbed into Windows Explorer and it seems it has… I found a forum post that suggests using Windows Photo Gallery and then installing some codecs (this post has more information on raw support in Windows Vista) but it turns out that the camera raw codecs are also available for direct download (i.e. with no need for Windows Photo Gallery) and after installation the raw file contents are available in thumbnails, previews and applications.

Unfortunately the major manufacturers (Canon and Nikon) do not produce codecs for 64-bit Windows (i.e. for people running high-end workstations with lots of memory for editing large images…) but the 32-bit codecs are fine for my little netbook with 2.5GB of RAM and there is 64-bit support for Adobe digital negatives (.DNG).

During installation, the Canon codecs complained that the screen resolution was not high enough on the netbook (1024×576) and refused to install but that was easily overcome by connecting to an external monitor with a higher resolution (no such issue with the Nikon codecs).

Incidentally, whilst I was researching this blog post I found that Microsoft also has an interesting program called Pro Photo Tools, which includes the ability to geotag photos, edit metadata, convert between raw formats, TIFF, JPEG and HD Photo; and work with Sidecar (.XMP) files (for interoperability with Adobe products – i.e. Bridge).  It too relies on the installation of the relevant raw codecs but should fit in quite nicely for some basic metadata tagging on the netbook whilst still in the field before transferring the images to the MacBook for any final tweaks when I get home.
Nikon raw image viewed in Microsoft Pro Photo Tools

Working with raw digital camera images

Recently, I went along to one of the meetings of my local camera club, which is something I’ve been meaning to do for a while but somehow never got around to. At the meeting, one of the members (Andy Gailer) gave a really interesting presentation on working with raw images. I’ve repeated most of the highlights here, adding a few notes of my own along the way.

Technical details

Raw images are exactly that – the raw pixel data that is captured by a digital sensor. So, in order to understand the use of camera raw, it helps to understand a little bit about the technology that creates the image.

Probably the most important part of a digital camera is the sensor that converts available light into electrical signals. Two types of sensor are commonly used: charge-coupled device (CCD); and complementary metal oxide semiconductor (CMOS). CCD is a more mature technology but CMOS is gaining popularity as it can be implemented using fewer components, requires less power and provides the data more quickly.

Regardless of the technology in use, digital camera sensors consist of an array of photodiodes (or “pixels”) collecting photons (minute amounts of energy which combine to make light) and each pixel is fitted with a microlens to focus light into the sensor site. The number of photons collected in each pixel is converted into an electrical charge and this charge is converted into a voltage, amplified, and converted to a digital value to be processed into a digital image, either in camera or, if a raw image is used, using a computer. It’s important to understand that, in the same way that a bucket can only hold so much water, a pixel can only hold a certain amount of light.

Sensors also come in a variety of sizes. A “full frame” 35mm sensor is the same size as a frame of 35mm film (24x36mm) but a compact digital compact camera will have a much smaller sensor. My Canon Digital Ixus 70 has a 5.75×4.31mm sensor but my Nikon D70 DSLR has a 23.7×15.5 mm (Nikon DX) sensor. The Canon squeezes 7077888 pixels onto that tiny sensor whereas the Nikon only has 6016000 pixels, but each one of the Ixus 70’s pixels is significantly smaller than the D70’s and this will affect the image quality – that’s why not all megapixels are equal.

Bayer sensor patternThe quality of the image will also be affected by it’s contents – each pixel can only capture one of three colours as it has a red, green, or blue filter over the top, usually arranged in a pattern known as the Bayer Mask, with twice as many green filters as red or blue (the pattern is designed to mimic the way that we see colour). On top of the Bayer filter is an infrared filter, then an antialiasing filter (to reduce moiré) and each of these various filters steadily reduces the overall quality of the image.

An alternative filter (the Foveon X3) employs an arrangement that is similar to the one used to make up the coloured emulsion layers in photographic film, where the red, green, and blue pixels are placed on top of one another (different colours of light will penetrate further into the sensor), meaning that all pixels capture all colours, but this sensor type is relatively uncommon and also suffers from low light sensitivity.

Bayer sensor filteringThe resulting data from the sensor consists of three channels of photographic data – red, green and blue but, with the exception of the Foveon sensor, each channel is incomplete because the mask means that only certain pixels will be activated for a given colour. During raw conversion (either in-camera, or on the computer), a process known as demosaicing is used in an attempt to fill in the missing pixel data, based on the comparative brightness of the surrounding pixels, and then sharpening is applied to counteract the effect of so many filters on the sensor.

I mentioned earlier that each sensor site (or pixel) can only capture a finite amount of light, expressed as a number of levels.

The number of bits used in the analogue to digital conversion process will determine the light sensitivity, with 8 bits representing 255 levels, 12 bits for 4095 levels, 14 bits for 16383 levels and 16 bits for 65535 levels. It’s important to understand that a sensor records light in a linear fashion, so reducing the amount of light falling on the sensor by one stop (EV) will halve the number of levels of light that can be recorded. Equally, if the light is doubled, eventually the pixel will be full and the resulting effect is blown highlights.

Similarly, as the light levels drop, an effect known as posterisation (or colour banding) becomes visible, particularly in areas such as shadow detail, or the sky.

Even a few stops can make a huge difference to the number of light levels that the sensor can determine and so it is generally recommended to expose as far to the right of the histogram as possible without clipping (I’ll describe the histogram in a follow-up post). Because human vision is not linear, during raw conversion a tonal curve (including a gamma correction) is applied to the image to make it more pleasing on the eye.

The table below shows the difference between an image recorded as an 8-bit (gamma encoded) JPEG and others recorded as a 12-bit or 14-bit (linear encoded) raw file:

Stop 8-bit JPEG (gamma-encoded) 12-bit raw (linear) 14-bit raw (linear)
1st stop (brightest tones) 69 levels 2048 levels 8192 levels
2nd stop (bright tones) 50 levels 1024 levels 4096 levels
3rd stop (mid-tones) 37 levels 512 levels 2048 levels
4th stop (dark tones) 27 levels 256 levels 1024 levels
5th stop (darkest tones) 20 levels 128 levels 512 levels

Even though the logarithmic scale used for the gamma-encoded image does not fall off as sharply as the linear scale for the raw image, the overall number of discernible light levels is reduced in the JPEG (partly due to the 8-bit nature of the file format), whereas the raw files retain more detail, allowing for some exposure compensation to be applied post-capture. In addition, due to the lossy compression that is inherent with a JPEG, further image quality is sacrificed each time the image is saved.

Colour spacesColour spaces are another consideration, with each space defining the number of visible colours (or gamut) that may be represented in an image. Which colour space is “best” is often a personal consideration but it’s important to note that we can neither see, nor print all of the available colours; however, by storing the maximum possible amount of information, there is more scope for making changes later without degrading image quality. For print work, Adobe RGB may be a good colour space but for on-screen work (where the display device has a smaller gamut), sRGB may be more appropriate. I have now switched the default setting in my Nikon D70 to Adobe RGB 1998 but in reality it makes very little difference as the colour space can be altered later.

JPEG or raw?

For a JPEG image, the following process is applied to every image by the camera:

  1. RGB information from sensor is converted to colour data.
  2. Tone curve applied to convert linear-encoded data to gamma encoding.
  3. White balance set.
  4. Contrast adjusted.
  5. Colour saturation increased.
  6. Sharpening applied.
  7. 12/14-bit native file compressed to lossy 8-bit JPEG.
  8. Image is recorded to memory card.

By shooting raw, no data is lost from the sensor and a better tonal quality is retained. Images can be reprocessed years later for better (or alternative) results; however some raw processing software will be required.

Adobe Camera Raw is a free download and allows all of the adjustments that a camera would normally make to be applied to an image (and more), under the control of the photographer. It integrates with other Adobe applications (e.g. Bridge and Photoshop) for image organisation and editing. At first, the interface can be daunting – but the controls are organised in order of significance (left to right and top to bottom) and many may be ignored. Adobe’s white paper on understanding Adobe Photoshop Camera Raw 4 is also worth a read.

Adobe Camera Raw 4.0

There is one significant drawback with raw image capture though – even though the sensor data is captured in the same way, most camera manufacturers (particularly Canon and Nikon) record the data using a proprietary format. This is why software such as Adobe Camera Raw is constantly updated for new cameras; however it’s also a risk that one day those raw images will become obsolete. There is a potential solution, using Adobe’s Digital Negative (.DNG) format but adoption by manufacturers has been slow and, for many photographers, conversion from a proprietary raw format to DNG is an extra step in the workflow.

Working with raw images

Andy gave some good advice for working with raw images and I’ve added a few tips of my own to Andy’s advice:

  • At the capture stage:
    • Just because you can edit later, don’t rely on it – take your best shots with proper settings – particularly focus and exposure.
    • Get the brightest possible shot without clipping – use the camera’s histogram function and expose to the right.
    • Check the shot in-camera by zooming in on parts of the image on the LCD.
  • Back on the computer, organise the files:
    • Download images to the PC.
    • Sort, organise, tag, rank and caption as desired.
    • Add metadata.
    • Back up the images to a separate storage location.
    • Automate repetitive tasks (e.g. renaming and captioning).
  • Process the raw images:
    • Process for maximum quality.
    • Adjust colour balance.
    • Crop, straighten and sharpen (if required – and only if no more editing is to be performed).
    • Save converted files at 16-bit and back up to offline storage.
  • Edit (if required):
    • Apply any image enhancements, clean up flaws, etc.
    • Perform any creative enhancements.
    • Apply batch actions
    • Prepare for output (printing or web) – if sharpening is required, this should be the last action on the image before saving.
  • Archive:
    • Establish and implement an archival plan.
    • Save files on external devices and media for easy access and retrieval – consider off site storage.

Further reading

Further information may be found in the following articles:

Credits

Based on a presentation by Andy Gailer. The Bayer filter images used in this post are licensed under the GNU free documentation license and the colour space diagram is ©Jeff Schewe, used with permission (images from Wikipedia).

Adobe Photoshop CS3 from a photographer’s perspective

Photography has been one of my hobbies for almost 30 years now and for the last four years I’ve been exclusively shooting in digital format but I’m still struggling to work out a decent workflow. Adobe Photoshop CS3 box shotSo, last Friday I took the day off work to attend a short course introducing key features of Adobe Photoshop and Lightroom (provided by my local Adult Education service and presented by David Tunnicliffe) and, at £36 for 6 hours of tuition, it was a bargain. I learnt a lot – and some people might find what follows to be a little obvious – but I’m hopething that for others it might be as useful as it was for me.

Adobe Photoshop CS3 is just one product from the Adobe Creative Suite, providing many more features than most photographers will need as it is designed for the graphics art industry in general. Even so, Photoshop CS3 includes some additional components that may be very useful for a photographer:

  • Introduced with CS2, Adobe Bridge literally bridges the gap between a file browser and the various applications in the Creative Suite, providing digital asset management functionality for organising, previewing and editing images.
  • Frequently updated for new proprietary raw formats, Adobe Camera Raw (ACR) provides the tools to open and edit raw image data from a digital camera. In addition to this, it can also work with JPEG and TIFF files to provide basic editing functionality without Photoshop.

In addition to the CS3 components above, there are two more products that may be of interest:

  • DNG is Adobe’s non-proprietary and royalty-free Digital Negative format which is intended to provide a solution for archival of digital images and has now been embraced by many camera manufacturers – the notable exceptions being Canon and Nikon. I don’t fancy my chances of being able to open .NEF images from my Nikon D70 in 30 years time but I’ll stand a better chance if I convert them to .DNG and Adobe provides a free DNG converter for Windows and Macintosh users.
  • Adobe Photoshop Lightroom 2 125x125Adobe Photoshop Lightroom is a product designed by photographers, for photographers using separate library, develop, slideshow, print and web views and including the ability to import (including creating folders), catalogue (with keywords) and backup in one action.

For me, Bridge was a revelation – I use my Mac for photography (so Windows utilities were no good to me) but I’ve always found Apple iPhoto a little too simplistic. It turns out that Bridge (together with ACR) is exactly what I needed to organise my images, open them in ACR (and optionally Photoshop) to perform non-destructive edits, with the changes (and associated metadata) stored in Sidecar (.XMP) files alongside the original image (avoiding the need to maintain multiple copies of images). In addition, now that I’m using Bridge I can drop a couple of utilities that I had previously relied on:

Previously, I’d struggled to get to grips with ACR (in fact, episode 40 of This Week in Photography featured a question from me asking for some guidance) but, armed with the knowledge I gained on the course and Adobe’s understanding Adobe Photoshop Camera Raw 4 white paper, I now understand that ACR is more than just a converter and it’s often all that’s required to make many adjustments to images (the exposure control in ACR let me recover an image that had been three stops underexposed) – and that it can handle JPEG and TIFF files too.

Adobe Camera Raw 4.0

As for Lightroom, David Tunnicliffe was very keen on the product (not surprising as he took part in its development) but, whilst I can see that its potentially useful for a professional photographer and that the ability to import, catalogue and backup images in one go would make a huge improvement to my workflow, I’m still not totally convinced by the interface. Maybe I’ll change my mind at version 3!

Adobe Photoshop Elements 7 125x125For those who don’t want to spend the money on Photoshop CS3 (it is very expensive if you’re not going to use it to it’s full potential) and who can manage without Bridge (which is only sold as part of Photoshop), Photoshop Elements (for Windows or Macintosh) includes enough functionality for many photographers, although some elements are hidden from the interface (find out more about the hidden elements at Richard Lynch’s site). Also, expect to see Photoshop Express become more and more useful over time. Meanwhile, ACR and the DNG converter are free downloads so they are available to Elements users too.

For me, I’m pretty sure that my new digital photography workflow will be built around Bridge and ACR and I expect to be writing some more photography-related posts as that workflow starts to come together.