Hi Guys, I wonder if any one can help me out.
Our V.D.P has been struggling for a few weeks now but its getting progressively worse. Over the last week or so the backups for the Windows VMs have been so slow that they don't actually finish. They Just fail and then try and restart.
Because of this V.D.P is constantly running which will be an issue should we need to migrate a VM for any reason. I have been doing some digging and trying to see what I can find. However, I am pretty new to V.D.P in general. After obtaining the logs from the server I can see the following errors keep repeating I wonder if anyone has any insight as to what they mean and how to fix them?
1:
2014-07-30 21:17:22,307 INFO [Timer_HFSValidator]-vi.VIAccess: Local Client found with name vdp-b
2014-07-30 21:17:26,437 INFO [VmwareFlrCLI-update]-services.BaseCLI$4: VmwareFlrCLI-update: State update complete. New state: running
2014-07-30 21:17:26,444 INFO [Timer_ServiceHealthValidator]-services.BaseCLI: MaintCLI: Updating service state.
2014-07-30 21:17:28,791 INFO [Timer_cache]-service.ServiceInstance: ServiceInstanceMoref desc=Service Id: urn:uuid:2D771F702AB73E8D211406751870806 name=urn:uuid:2D771F702AB73E8D211406751870806 value=SERVICE
2014-07-30 21:17:36,217 INFO [Timer_EvalLicenseValidator]-license.HostVmwareLicenseChecker: Checking if license edition: [esxEnterprise.vram] is valid...
2014-07-30 21:17:38,781 INFO [Timer_EvalLicenseValidator]-schedule.EvalLicenseValidator$EvalLicenseValidatorTimerTask: ESX Host License Is Valid For VDP
2014-07-30 21:17:40,593 INFO [Timer_EvalLicenseValidator]-vi.VIAccess: Attempting to obtain AVE VM with IP Address: [10.0.2.195]
2014-07-30 21:17:40,923 INFO [MaintCLI-update]-services.BaseCLI$4: MaintCLI-update: State update complete. New state: running
2014-07-30 21:17:42,433 INFO [Timer_ServiceHealthValidator]-services.BaseCLI: SchedCLI: Updating service state.
2014-07-30 21:17:42,627 INFO [Timer_EvalLicenseValidator]-vi.VIAccess: Local Client found with name vdp-b
2014-07-30 21:17:42,786 ERROR [Thread-122781]-util.ProcessWrapper: Caught Exception attempting to run command: [mccli mcs scheduler-status]
java.io.IOException: Cannot run program "mccli": error=2, No such file or directory
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at com.emc.avamar.adapter.cli.util.Command.executeCommand(Command.java:72)
at com.emc.avamar.adapter.cli.util.ProcessWrapper.start(ProcessWrapper.java:78)
at com.emc.avamar.adapter.cli.CommandExecute.exec(CommandExecute.java:170)
at com.emc.avamar.adapter.util.CliServiceCallable.run(CliServiceCallable.java:44)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 8 more
2:
2014-07-30 20:42:52,066 ERROR [Thread-121956]-util.ProcessWrapper: Caught Exception attempting to run command: [mccli mcs scheduler-status]
java.io.IOException: Cannot run program "mccli": error=2, No such file or directory
at java.lang.ProcessBuilder.start(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at java.lang.Runtime.exec(Unknown Source)
at com.emc.avamar.adapter.cli.util.Command.executeCommand(Command.java:72)
at com.emc.avamar.adapter.cli.util.ProcessWrapper.start(ProcessWrapper.java:78)
at com.emc.avamar.adapter.cli.CommandExecute.exec(CommandExecute.java:170)
at com.emc.avamar.adapter.util.CliServiceCallable.run(CliServiceCallable.java:44)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 8 more
3:
2014-07-30 21:21:51,466 FATAL [Timer_general]-schedule.TaskMonitor: Unable to establish connection. Attempt to reconnect.
I am also seeing alarms in vCentre to tell me that the V.D.P Core services are not running, The back up schedular is not running and the most recent check point is out of date.
I ran the following to see what was happening
:~/#: dpnctl status
Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid)
dpnctl: INFO: gsan status: up
dpnctl: INFO: MCS status: up.
dpnctl: INFO: Backup scheduler status: up.
dpnctl: INFO: axionfs status: up.
dpnctl: INFO: Maintenance windows scheduler status: enabled.
dpnctl: INFO: Unattended startup status: enabled.
Which looks like the vCentre alarms are fulse. but the other ones Im not sure about.
Any help is more than welcome.
Thanks