With the current network configuration the 3 vmnics are shared by storage, management as well as VM traffic which is not the best practice. Anyway, what's the backup target, i.e. does the backup use the vmnics too in order to transfer the data to the backup target?
I'd start troubleshooting by running esxtop from the ESXi host's console, to determine the load on the vmnics during backup.
André