Just adding to this in case it saves someone some time. I had the registry settings as listed by max2479 but still getting the error Failed to create snapshot for <vmname>, error -3960 (cannot quiesce virtual machine)when running a backup job in VDR.
My case was to simply enable the virtual disk service on the 2008 R2 server and it then all worked.