Automating VCF Upgrade - Workload Domains

calendar Oct 18, 2023 · 10 min read · VCF  ·
Share on: linkedin copy

This is the final part of my blog series in which I talk about automating the VMware Cloud Foundation upgrade process. At this point you should have completed the following tasks:

  • Downloaded All Bundles
  • Initial Health Check
  • SDDC Manager Upgrade
  • SDDC Manager Configuration Drift Update

We are now ready to begin the process of upgrading the individual components within a Workload Domain, for this we must first start with the Management Domain before we can carry out any component upgrades of a VI Workload Domain. The actual process is performed in the following way:

  • Step 1 - Management Domain
    • Step 1a - NSX
    • Step 1b - vCenter Server
    • Step 1c - ESXi
  • Step 2 - VI Workload Domain
    • Step 2a - NSX
    • Step 2b - vCenter Server
    • Step 2c - ESXi

VMware Cloud Foundation APIs Used

  • GET /v1/upgradables/domains/
  • GET /v1/domains/
  • GET /v1/bundles
  • GET /v1/upgrades/

PowerVCF Cmdlets

  • Get-VCFWorkloadDomain
  • Get-VCFBundle
  • Get-VCFUpgrade

Management Domain

Upgrade NSX

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the Management Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.type -eq "MANAGEMENT"}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status

In this step we use the Get-VCFWorkloadDomain cmdlet to get the Management Domain Id and then using Start-VCFSystemPrecheck we trigger an upgrade pre-check.

  1. Perform NSX upgrade of the Management Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "NSX_T_MANAGER") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 60; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Here we us the Management Domain ID with /v1/upgradables/domains/ API to get the details of next available bundle for upgrade (This API is not currently in PowerVCF). We perform a few checks before starting the upgrade which include:

  • Check that the upgrade bundle has been physically downloaded to SDDC Manager
  • Check that the upgrade bundle has a type of NSX_T_MANAGER

If both of these conditions are true we proceed using the Start-VCFUpgrade cmdlet to trigger the upgrade, capturing the API response in $task which we then use with the Get-VCFUpgrade cmdlet to perform a Do { } While { } loop on the upgrade task Id until the status is no longer in a Scheduled, Pending or In Progress state.

Once complete that's it we have successfully upgraded all NSX components within the Management Domain programmatically.

Upgrade vCenter Server

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the Management Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.type -eq "MANAGEMENT"}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status
  1. Perform vCenter Server upgrade of the Management Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "VCENTER") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 660; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Once complete that's it we have successfully upgraded vCenter Server for the Management Domain.

Upgrade ESXi Hosts

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the Management Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.type -eq "MANAGEMENT"}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status
  1. Perform ESXi Host upgrade of the Management Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "HOST") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 60; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Once complete that's it we have successfully upgraded the ESXi Hosts for the default cluster of the Management Domain.

VI Workload Domain

To automate the upgrade of the VI Workload Domain we simply follow the same process.

Upgrade NSX

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$sddcDomainName = "lax-w01"
5$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the VI Workload Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.name -eq $sddcDomainName}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status

In this step we use the Get-VCFWorkloadDomain cmdlet to get the Management Domain Id and then using Start-VCFSystemPrecheck we trigger an upgrade pre-check.

  1. Perform NSX upgrade of the VI Workload Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "NSX_T_MANAGER") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 60; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Upgrade vCenter Server

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$sddcDomainName = "lax-w01"
5$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the VI Workload Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.name -eq $sddcDomainName}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status
  1. Perform vCenter Server upgrade of the VI Workload Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "VCENTER") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 60; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Upgrade ESXi Hosts

  1. Start Windows PowerShell.

  2. Replace the values in the sample code with values for the instance of VMware Cloud Foundation and run the commands in the PowerShell console.

1$sddcManagerFqdn = "lax-vcf01.lax.rainpole.io"
2$sddcManagerUser = "[email protected]"
3$sddcManagerPass = "VMw@re1!"
4$sddcDomainName = "lax-w01"
5$vcfRelease = "4.5.2.0"
  1. Authenticate to the SDDC Manager appliance by running the following command:
1Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass
  1. Perform a Health Check of the VI Workload Domain
1$workloadDomain = (Get-VCFWorkloadDomain | Where-Object {$_.name -eq $sddcDomainName}).id
2$jsonSpec = '{ "resources" : [ { "resourceId" : "'+ $workloadDomain +'", "type" : "DOMAIN" } ] }'
3$task = Start-VCFSystemPrecheck -json $jsonSpec
4Do { $status = Get-VCFSystemPrecheckTask -id $task.id } While ($status.status -eq "IN_PROGRESS")
5$status.subtasks | Select-Object name, status
  1. Perform ESXi Host upgrade of the VI Workload Domain by running the following command:
 1$uri = "https://$sddcManager/v1/upgradables/domains/$workloadDomain/?targetVersion=$vcfRelease"
 2$bundle = ((Invoke-RestMethod -Method GET -URI $uri -ContentType application/json -headers $headers).elements) | Where-Object {$_.status -eq "AVAILABLE" }
 3if ((Get-VCFBundle -id $bundle.bundleId).downloadStatus -eq "SUCCESSFUL" -and (Get-VCFBundle -id $bundle.bundleId).components.type -eq "HOST") {
 4    $jsonSpec = '{
 5        "bundleId": "'+ $bundle.bundleId +'",
 6        "resourceType": "DOMAIN",
 7        "resourceUpgradeSpecs": [ {
 8            "resourceId": "'+ $workloadDomain +'",
 9            "upgradeNow": true
10        } ]}'
11    $task = Start-VCFUpgrade -json $jsonSpec
12    Write-Output " Waiting for Upgrade Task ($($task.name)) with Id ($($task.id)) to Complete"    
13    Do { Start-Sleep 60; Request-VCFToken -fqdn $sddcManagerFqdn -username $sddcManagerUser -password $sddcManagerPass | Out-Null; $status = Get-VCFUpgrade -id $task.id } While ($status.status -in "PENDING","SCHEDULED","INPROGRESS")
14    Write-Output " Upgrade Task with Id ($($task.id)) completed with status ($($status.status))"
15} else {
16    Write-Warning " Upgrade Not Possible as Required Bundle ($bundleId) Not Downloaded to SDDC Manager"
17}

Conclusion

Using the above steps I've shown how its possible to automation the upgrade process of the underlying components that make up a Workload Domain.

I hope this blog series has been useful to help illustrate the potential in automating the end-to-end process of a VMware Cloud Foundation environment, if you have comments or feedback let me know.

Posts in this Series