VMware 6.0 Update 1 and Veeam

VMware came out with some updates last week.  Like an idiot, I put them on fairly soon after I saw them.  Then my Veeam backup ran.  Or rather, it didn't run.  In Veeam, it gave the error:

9/14/2015 2:46:30 AM :: Processing Test Error: NFC storage connection is unavailable. Storage: [stg:datastore-23,nfchost:host-2706,conn:10.0.0.0]. Storage display name: [VM-Datastore1].
Failed to create NFC download stream. NFC path: [nfc://conn:10.0.0.0,nfchost:host-2706,stg:datastore-23@Test/Test.vmx].

The VMware knowledge base article said on your Veeam server, look in c:\Program Data\Veeam\Backup\"Name of your backup" and open the file Agent."backupname".Source."VM name".  It opens with Notepad.  Search for "NFC".  Scroll down from there.  You should see something similar to this:

Authd version: [1.10]
[12.09.2015 02:31:08] <  2000> nfc|             SSL connection is required to perform authentication.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing the SSL subsystem...
[12.09.2015 02:31:08] <  2000> nfc|             The SSL subsystem was successfully initialized.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing new SSL connection...
[12.09.2015 02:31:08] <  2000> nfc|               Establishing connection with the SSL server... Failed.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing new SSL connection... Failed.

And a little farther down:

[12.09.2015 02:31:08] <  2000>      ERR |SSL error, code: [336151568].error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
[12.09.2015 02:31:08] <  2000>      >>  |SSL_connect() function call has failed.
[12.09.2015 02:31:08] <  2000>      >>  |Failed to establish connection with the SSL server.
[12.09.2015 02:31:08] <  2000>      >>  |Cannot initialize new SSL connection.
[12.09.2015 02:31:08] <  2000>      >>  |Authd handshake has failed.

The important thing to note is the references to Authd.

The problem is that update 1 turns off SSLv3.  Unfortunately, Veeam is still using SSLv3 to communicate with your hosts.  SSLv3 has to be turned back on.  Per the VMware knowledge base article 2121021:

Enable support for SSLv3 on Authd service 902 in ESXi

1. Create a backup copy of the /etc/vmware/config file 
2. Edit the /etc/vmware/config file to append the following line at the end of the file:
   
   vmauthd.ssl.noSSLv3 = false
   
   Note: If you have the line vmauthd.ssl.noSSLv3 = true in the file, change it to vmauthd.ssl.noSSLv3 = false
   Example:
   
   [root@w1-fiqabj-003:~] cat /etc/vmware/config
   libdir = "/usr/lib/VMware"
   authd.proxy.nfc = "vmware-hostd:ha-nfc"
   authd.proxy.nfcssl = "vmware-hostd:ha-nfcssl"
   authd.proxy.vpxa-nfcssl = "vmware-vpxa:vpxa-nfcssl"
   authd.proxy.vpxa-nfc = "vmware-vpxa:vpxa-nfc"
   authd.fullpath = "/sbin/authd"
   vmauthd.ssl.noSSLv3 = false
   
3. Restart the rhttpproxy service with the command:
   
   /etc/init.d/rhttpproxy restart

This needs to be done on each of your hosts.  It is simple enough that it only took me maybe 2 minutes per host.  Since I only have 3 hosts, it wasn't a big deal.  I ran a test backup and it worked fine after making this work-around.

Right after I got this resolved, with the help of Veeam support, the support guy emailed me and said there was now a KB article on it.

Vcenter Server Appliance 6.0 - Running out of log space

So I was looking through my VCenter Server and found an entry that shows I was running out of log space.  Hmmm, this may be why I was getting the syslog alerts that VMware support was absolutely no help with.  I started searching around for ways to increase the space for the logs, but with 11 .vmdk's, which one was for the logs??  Fortunately I found the blog Virtually Ghetto by William Lam.  I have stumbled across that blog before but forgot about it, like I do most things in my advancing years! He tells what each vmdk is here.

Copied from VirtuallyGhetto.com

Next problem, although he shows what each vmdk is for, I don't know the command line very well for vcsa.  Fortunately, he assists with the actual mechanics of increasing the drive size with a link in the article here.  It is easy enough that even I can do it!  Using Putty to SSH in to vcsa, get to the BASH shell.  At the command prompt "shell.set --enabled True" and then "shell", Then, run df -h to see the current size of the log file vmdk.  Go in to the vSphere web client and increase the size of Hard Disk 5.  I bumped it up to 15GB.  Then back in Putty, "vpxd_servicecfg storage lvm autogrow". Finally, "df -h" again to show that the vmdk has increased.

This is a copy of my successful attempt at increasing the log size on my vcsa:

Migrating from Exchange 2007 to Exchange 2013

While evaluating MS Office 2016, an updated applied enforced the MS rule of only going back 2 versions of Exchange with Outlook.  Once day Outlook 2016 running off Exchange 2007 worked fine, the next day it didn't.  Well damn, I guess we'll be upgrading to Exchange 2013 now!

So far all I've done is set up a new VM (Server 2012 R2), installed all the system requirements and prerequisites, downloaded/installed Exchange 2013 and put on CU9.  Well, I guess that's pretty much everything to at least get it up and running on the latest version.

I have never done a new Exchange installation before and certainly never set one up with an existing, older version running.  So I'm getting help from a consultant I use from time to time when I want to make sure I don't end up making things worse.  He can't get in for a couple of weeks so I'll update this once he has gotten everything set up the way it needs to be for this to work and so I can start migrating mailboxes.

To be continued.....

Deduplication on Server 2012R2

I've been using deduplication on my file server for some time now.  Every once in a while I will get corrupt files.  I can restore them from backup, but then sometimes the same files get corrupt again, other times it is different files.  I don't have a lot of time to play with this so I want to turn off deduplication.  After researching this, I found that just turning off deduplication doesn't "un-dedup" the volume.  I found a few different sites that tell how to un-dedup the files but this is the process that worked for me.

1.  Do NOT disable deduplication.  However, to ensure your volume doesn't get optimized while you are trying to Un-Dedup it, exclude it from the optimization process.

     A.  Open Server Manager and go to Volumes and select your volume

     B.  Right click your volume and choose Data Deduplication.

     C.  Exclude the whole volume.  Click add, select the server name on the left and the volume name on the right.  Click Select Folder.

     D.  Click Set Deduplication Schedule and uncheck everything.  Again, we don't want an optimization to run while you are un-deduping.

2.  Open Powershell as an administrator

3.  Run Garbage Collection on the volume:  Start-DedupJob -Volume "X:" -Type GarbageCollection

4.  When Garbage Collection is done, Un-Dedup the volume:  Start-DedupJob -Volume "X:" -Type Unoptimization  This will, most likely, run for a LONG time - as in many, many hours.  Run Get-DedupJob in Powershell to monitor the status.

5.  When it is finally done, repeat on all volumes you want to un-dedup.  It will also no longer show any info in the deduplication status in the Server Manager.  G: is the volume I "un-deduped".  You may need to close Server Manager and get back in to see the change.  A refresh didn't show it for me.

6.  When you are finally done (when Server Manager shows no deduplication infor for any of your volumes, you can uninstall the deduplication role from the server.