Rim lighting is always a finicky process that requires lots of finesse and trial and error to get to look the way you want in Mental Ray. In Maya's software render you can simply connect the samplerInfo.facingRatio into a rampShader.VCoord then plug that directly into a light's color but, unfortunately that does not work in Mental Ray (I'm not sure why) . I've come up with a work around for this by using the camera space normal direction output mode of the mi ambient occlusion shader.
The shading network is fairly straightforward. The occlusion node has its output mode set to 3 which is camera space normals. It Blue value (Z) is then hooked up into an RGB to HSV node so that it can be remapped in a ramp. The outputX of the RGB to HSV node is then plugged into the VCoord of a ramp which is then used to drive the thickness of the rim light. The light is then given a rim thickness attribute which can be used to control how thick the rim light is. If you're interested in trying this out or taking a closer look at it, here's a script that will create this network for you.
Make sure you're using mental ray or this will not work.
Also, there is a known issue that this does not yet work with raytraced shadows so you must use depth maps
MRRimLightShader.mel (Right click and save as)