At some point, it is beneficial to settle on a single Photoshop masking plugin and learn its intricacies. I tried Vertus Fluid Mask, but it didn't even have a user manual! OnOne has a masking plugin that is great, but for some reason I gravitated to ReMask by Topaz. I will present some tips that I have learned after using this plugin for several years now.
After adjusting the image in Lightroom, I retouch it in Photoshop, then make a "stamped" or "composite" layer with no mask (Option/Alt Merge Visible from the flyout menu of the Layers panel) and invoke ReMask 5 (Filter -> Topaz Labs -> Topaz ReMask 5...).
However, and this is important: if there is not enough contrast between the model's hair and the background, move Lightroom's Exposure slider to the right until you seen nice contrast. Don't worry that the model's skin looks blown out and terrible; we are only using this overexposed version to get a mask. Once Topaz ReMask has rendered your mask, save it to your Desktop under the name "temp." Then go back to Lightroom, hit Reset, and adjust the sliders so the model looks good, and bring this version into Photoshop as well. Once the model has been retouched, Shift-drag the temporary version with the mask onto the good version, and drag the mask down onto a retouched layer of the model, then delete the overexposed layer. Note that the good version has the original file name because the overexposed version has been renamed "temp."
Once in ReMask, you are presented with a green version of your image. We will use this image of the gorgeous Marcela Zuniga as an example:
This is the beginning of what is called the tri-map. Whatever is green is kept, whatever is red is masked out, and blue is the area in between the two where ReMask does its computations. I always start by enlarging the image four times (Command/Control +). You are presented with the blue brush, which is also called the compute brush. The brush pointer will be a circle with a + sign inside. I decrease the brush size until the pointer becomes just a + sign with no circle. By using a small compute brush, if ReMask misinterprets the edge, it will do so in a narrow area. The idea is to paint over all the edges of the model, then add red to the background areas using the red bucket tool. Here is what a completed tri-map looks like - ready for us to hit the Compute Mask button:
Here are a few tips on creating a good tri-map. Check this area with the ties of her swimsuit:
Notice the red triangle on top? It is too small to outline in blue, so I simply drew red over the unwanted background. ReMask has a problem with sharp, pointed areas as seen in the other three triangular areas. To remedy that, use a tiny red brush to draw some red into those areas, as I did here.
The hair is where most of your time will be spent. Notice that I have used a tiny red brush to put a little red in between the hair strands. This helps ReMask immensely. You do not need to put red in every area where you can see background, because ReMask uses proximity matching to look at adjacent areas and it will mask those out too. But, the more red you paint into those areas, the better the mask will be.
Also note there are several "islands" of background bounded thick stands of hair or her shoulder. It is critical to paint red into those areas, or else ReMask will ignore them entirely.
One might ask, why go to all this trouble? There are tools in ReMask to fix any problems... Indeed, this is true, and sometimes I will go the quick route on an easy-to-mask image. But it is my feeling is that ReMask's first wave of computation, when you hit the Compute Mask button, is more accurate and powerful than any of the tools provided to fix the mask later.
Notice the small patch of background showing through the hair in the above image. I elected not to mask this area, as I thought it unsightly. Once back in Photoshop, I used the Spot Healing Brush Tool (set to Content Aware), to replace this area of background with hair.
Yes, this is a lot of work, but there is nothing more gratifying than a good mask! The above image shows how I painted red as close to the hair as possible. This makes ReMask's proximity matching more accurate during the computation.
Once the mask has been computed, I am only concerned with the part of the mask concerning the hair. To better visualize any problems, I hit the split view button, and hit the Keep button, and add a red background. If I see any unwanted original background, the transparency brush works great. Simply sample the hair color with the green eyedropper that is presented to you, then sample the background color with the red eyedropper, and paint over the problem area. Once you are happy with the mask in the area of the hair, hit the OK button to bring the mask back into Photoshop.
Dealing with the hard edges is easy, and best done in Photoshop. First Option/Alt click on the new mask in order to see it. Simply invoke the Brush tool with the foreground color set to white and set the Brush to Overlay in the Options Bar. Then, on the mask, paint over the hard edges of skin and clothing, staying away from the hair. Next, change to a black brush and repeat. The edges will become clear and distinct.
If there are still problems, they should be manually fixed with the Brush tool set to Normal, using white or black. Once I am happy with the mask, I always make a layer of the model with no mask. Do this by clicking on the layer thumbnail to select the pixels, then Command/Control click on the mask to get the "marching ants." Then hit Command/Control-J to create a new layer from the selection. The reason I want a layer with no mask is so I can run a Layer-> Matting -> Defringe (if a grey background was used) or the powerful Layer -> Matting -> Remove White Matte if a white background was used. I will run these once I have the new background in place so I can better judge how it looks.
I hope these tips prove useful. Here is what I did with the sample image of Marcela: