Monday, September 14, 2015

VMware 6.0 Update 1 and Veeam

VMware came out with some updates last week.  Like an idiot, I put them on fairly soon after I saw them.  Then my Veeam backup ran.  Or rather, it didn't run.  In Veeam, it gave the error:

9/14/2015 2:46:30 AM :: Processing Test Error: NFC storage connection is unavailable. Storage: [stg:datastore-23,nfchost:host-2706,conn:10.0.0.0]. Storage display name: [VM-Datastore1].
Failed to create NFC download stream. NFC path: [nfc://conn:10.0.0.0,nfchost:host-2706,stg:datastore-23@Test/Test.vmx].

The VMware knowledge base article said on your Veeam server, look in c:\Program Data\Veeam\Backup\"Name of your backup" and open the file Agent."backupname".Source."VM name".  It opens with Notepad.  Search for "NFC".  Scroll down from there.  You should see something similar to this:

Authd version: [1.10]
[12.09.2015 02:31:08] <  2000> nfc|             SSL connection is required to perform authentication.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing the SSL subsystem...
[12.09.2015 02:31:08] <  2000> nfc|             The SSL subsystem was successfully initialized.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing new SSL connection...
[12.09.2015 02:31:08] <  2000> nfc|               Establishing connection with the SSL server... Failed.
[12.09.2015 02:31:08] <  2000> nfc|             Initializing new SSL connection... Failed.

And a little farther down:

[12.09.2015 02:31:08] <  2000>      ERR |SSL error, code: [336151568].error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
[12.09.2015 02:31:08] <  2000>      >>  |SSL_connect() function call has failed.
[12.09.2015 02:31:08] <  2000>      >>  |Failed to establish connection with the SSL server.
[12.09.2015 02:31:08] <  2000>      >>  |Cannot initialize new SSL connection.
[12.09.2015 02:31:08] <  2000>      >>  |Authd handshake has failed.

The important thing to note is the references to Authd.

The problem is that update 1 turns off SSLv3.  Unfortunately, Veeam is still using SSLv3 to communicate with your hosts.  SSLv3 has to be turned back on.  Per the VMware knowledge base article 2121021:

Enable support for SSLv3 on Authd service 902 in ESXi

  1. Create a backup copy of the /etc/vmware/config file 
  2. Edit the /etc/vmware/config file to append the following line at the end of the file:

    vmauthd.ssl.noSSLv3 = false

    Note: If you have the line vmauthd.ssl.noSSLv3 = true in the file, change it to vmauthd.ssl.noSSLv3 = false
    Example:

    [root@w1-fiqabj-003:~] cat /etc/vmware/config
    libdir = "/usr/lib/VMware"
    authd.proxy.nfc = "vmware-hostd:ha-nfc"
    authd.proxy.nfcssl = "vmware-hostd:ha-nfcssl"
    authd.proxy.vpxa-nfcssl = "vmware-vpxa:vpxa-nfcssl"
    authd.proxy.vpxa-nfc = "vmware-vpxa:vpxa-nfc"
    authd.fullpath = "/sbin/authd"
    vmauthd.ssl.noSSLv3 = false
  3. Restart the rhttpproxy service with the command:

    /etc/init.d/rhttpproxy restart
This needs to be done on each of your hosts.  It is simple enough that it only took me maybe 2 minutes per host.  Since I only have 3 hosts, it wasn't a big deal.  I ran a test backup and it worked fine after making this work-around.

Right after I got this resolved, with the help of Veeam support, the support guy emailed me and said there was now a KB article on it.

10 comments:

  1. Any ideas on how to get this to work on AutoDeploy stateless caching hosts?

    ReplyDelete
  2. Sorry John. No idea. Autodeploy is beyond what I'm licensed for so I'm not familiar with that. I did hear that Update 1 even broke stuff within VMware. Seems to me that it (Update 1) wasn't very well tested.

    ReplyDelete
  3. Thanks for the write up. Ran into this problem tonight after an update from 6.0 to 6 Update 1.

    ReplyDelete
  4. Thanks a lot.
    Veeam 8.0 Update 3 is now available

    ReplyDelete
  5. Yes there is and I had a problem pop up with it too. Forgot to write about it here. There was a problem where Veeam update 3 would not delete the logs for WID SQL db. Call Veeam support for a couple of updated files if you see this problem.

    ReplyDelete
  6. Yes there is and I had a problem pop up with it too. Forgot to write about it here. There was a problem where Veeam update 3 would not delete the logs for WID SQL db. Call Veeam support for a couple of updated files if you see this problem.

    ReplyDelete
  7. hi sir:
    i have installed ESXi 6.0u2 3620759 but still problem exit during backup

    ReplyDelete
  8. Yeah too bad 6.0 u2 keeps failing.. time to downgrade :D

    ReplyDelete