Streamlining VMware Update Management with DSvClient: A vExpert’s tool

In the ever-evolving landscape of VMware infrastructure management, keeping components updated while ensuring consistency across environments has become increasingly complex. As a VMware administrator, you’re likely familiar with the challenges of managing patches, updates, and ensuring file integrity across multiple environments. Today, I’d like to introduce you to a powerful utility that I have created.

DSvClient C:\VMware\UpdateRepo

DSvClient C:\VMware\UpdateRepo --verify

# Download once 
DSvClient /mnt/shared/VMware/Updates
# Verify before deployment to each environment
DSvClient /mnt/shared/VMware/Updates --verify

# Configure DSvClient to download specific VCSA versions
vim sources.toml
# Download all necessary VCSA components
DSvClient /storage/vcsa-updates

# The sources.toml already includes vSAN HCL sources
DSvClient /path/to/vsan-repository

DSvClient /path/to/repository

DSvClient /path/to/repository --verify

[download]
max_concurrent_downloads = 10

[verification]
max_concurrent_verifications = 1000

[exclude]
patterns = [
      "addon-main/addon/NEC/vib20/qedentv/QLC_bootbank_qedentv_3.40.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qedf/QLC_bootbank_qedf_2.2.4.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qedi/QLC_bootbank_qedi_2.19.5.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qedrntv/QLC_bootbank_qedrntv_3.40.9.1-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/smx-provider/HPE_bootbank_smx-provider_700.03.16.00.12-14828939.vib",
      "addon-main/addon/NEC/vib20/i40en/INT_bootbank_i40en_1.10.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/lpfc/EMU_bootbank_lpfc_12.6.228.4-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qfle3f/QLC_bootbank_qfle3f_2.1.6.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qcnic/QLC_bootbank_qcnic_2.0.0.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qfle3/QLC_bootbank_qfle3_1.4.6.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qfle3i/QLC_bootbank_qfle3i_2.1.2.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/amscli/HPE_bootbank_amscli_11.5.0.22-1OEM.700.0.0.15525992.vib",
      "addon-main/addon/NEC/vib20/amsd/HPE_bootbank_amsd_700.11.5.0.28-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/ilo/HPE_bootbank_ilo_700.10.1.0.16-1OEM.700.0.0.14828939.vib",
      "addon-main/addon/NEC/vib20/ixgben/INT_bootbank_ixgben_1.8.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/igbn/INT_bootbank_igbn_1.4.11.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/NEC/vib20/qlnativefc/Marvell_bootbank_qlnativefc_4.1.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qedentv/QLC_bootbank_qedentv_3.40.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qedi/QLC_bootbank_qedi_2.19.5.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qedrntv/QLC_bootbank_qedrntv_3.40.9.1-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qedf/QLC_bootbank_qedf_2.2.4.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/sut/HPE_bootbank_sut_700.2.5.5-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/smx-provider/HPE_bootbank_smx-provider_700.03.16.00.12-14828939.vib",
      "addon-main/addon/HTI/vib20/hpessacli/HPE_bootbank_hpessacli_4.18.1.0-7.0.0.15525992.hpe.vib",
      "addon-main/addon/HTI/vib20/lpfc/EMU_bootbank_lpfc_12.6.228.4-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/ilo/HPE_bootbank_ilo_700.10.1.0.16-1OEM.700.0.0.14828939.vib",
      "addon-main/addon/HTI/vib20/amsd/HPE_bootbank_amsd_700.11.5.0.28-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/amscli/HPE_bootbank_amscli_11.5.0.22-1OEM.700.0.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qfle3i/QLC_bootbank_qfle3i_2.1.2.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qfle3f/QLC_bootbank_qfle3f_2.1.6.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qcnic/QLC_bootbank_qcnic_2.0.0.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qfle3/QLC_bootbank_qfle3_1.4.6.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/qlnativefc/Marvell_bootbank_qlnativefc_4.1.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/ixgben/INT_bootbank_ixgben_1.8.9.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/bootcfg/HPE_bootbank_bootcfg_700.10.5.0.23-7.0.0.15525992.vib",
      "addon-main/addon/HTI/vib20/hponcfg/HPE_bootbank_hponcfg_700.10.5.0.25-7.0.0.15525992.vib",
      "addon-main/addon/HTI/vib20/testevent/HPE_bootbank_testevent_700.10.5.0.24-7.0.0.15525992.vib",
      "addon-main/addon/HTI/vib20/conrep/HPE_bootbank_conrep_700.10.5.0.34-7.0.0.15525992.vib",
      "addon-main/addon/HTI/vib20/cru/HPE_bootbank_cru_700.10.16-1OEM.700.0.0.14828939.vib",
      "addon-main/addon/HTI/vib20/igbn/INT_bootbank_igbn_1.4.11.0-1OEM.700.1.0.15525992.vib",
      "addon-main/addon/HTI/vib20/fc-enablement/HPE_bootbank_fc-enablement_700.3.5.0.40-1OEM.700.0.0.15525992.vib",
]

[logging]
term_level = "Debug"

DSvClient Application Flow Diagram

To understand how DSvClient works, here’s a visual representation of the application’s flow from configuration to repository creation:

┌────────────────────────────────────┐
│ Configuration Files                │
│ ┌──────────────┐  ┌──────────────┐ │
│ │ config.toml  │  │ sources.toml │ │
│ │  - Download  │  │  - VMware    │ │
│ │    settings  │  │    updates   │ │
│ │  - Verify    │  │  - VCSA      │ │
│ │    settings  │  │    versions  │ │
│ │  - Logging   │  │  - vSAN HCL  │ │
│ │  - Exclusions│  │  - Vendor    │ │
│ └──────────────┘  │    add-ons   │ │
│                   └──────────────┘ │
└───────────────────┬────────────────┘
                    │
                    ▼
┌────────────────────────────────────┐
│ DSvClient Core Process             │
│ ┌──────────────────────────────┐   │
│ │ 1. Parse configuration       │   │
│ │ 2. Initialize components     │   │
│ │ 3. Set up logging            │   │
│ │ 4. Process command arguments │   │
│ └──────────────┬───────────────┘   │
│                │                   │
│                ▼                   │
│ ┌──────────────────────────────┐   │
│ │         Downloader           │   │
│ │ ┌────────────────────────┐   │   │
│ │ │ ESXi Updates Pipeline  │   │   │
│ │ └────────────────────────┘   │   │
│ │ ┌────────────────────────┐   │   │
│ │ │ VCSA Updates Pipeline  │   │   │
│ │ └────────────────────────┘   │   │
│ │ ┌────────────────────────┐   │   │
│ │ │ vSAN HCL Pipeline      │   │   │
│ │ └────────────────────────┘   │   │
│ │ ┌────────────────────────┐   │   │
│ │ │ Vendor Add-on Pipeline │   │   │
│ │ └────────────────────────┘   │   │
│ └──────────────┬───────────────┘   │
│                │                   │
│                ▼                   │
│ ┌──────────────────────────────┐   │
│ │          Verifier            │   │
│ │ - SHA-1/SHA-256 verification │   │
│ │ - Parallel processing        │   │
│ │ - Error tracking             │   │
│ └──────────────────────────────┘   │
└────────────────┬───────────────────┘
                 │
                 ▼
┌────────────────────────────────────┐
│ Local Repository Structure          │
│ /download_path/                    │
│ ├── depot/                         │
│ │   ├── index.xml                  │
│ │   ├── vmw-depot-index-*.xml      │
│ │   └── com.vmware.vsan.*          │
│ │                                  │
│ ├── addon-main/                    │
│ │   ├── index.xml                  │
│ │   └── addon/                     │
│ │       ├── HTI/ (HPE)             │
│ │       ├── DELL/                  │
│ │       └── Other vendors...       │
│ │                                  │
│ ├── hcl.json                       │
│ │                                  │
│ └── vcsa/                          │
│     └── version folders (e.g. 8.0.2)│
│        ├── manifest-latest.xml     │
│        └── vmware-vcsa-* files     │
└──────────────────┬─────────────────┘
                   │
                   ▼
┌────────────────────────────────────┐
│ Optional Web Server Deployment     │
│ ┌──────────────┐  ┌──────────────┐ │
│ │ Apache/NGINX │  │  Docker      │ │
│ └──────┬───────┘  └────┬─────────┘ │
│        │               │           │
│        └───────┬───────┘           │
│                │                   │
│                ▼                   │
│ ┌──────────────────────────────┐   │
│ │ Internal Web Server Endpoint │   │
│ │ http://updates.internal.com/ │   │
│ └──────────────────────────────┘   │
└────────────────┬───────────────────┘
                 │
                 ▼
┌────────────────────────────────────┐
│ Consumption Methods                │
│ ┌──────────────┐  ┌──────────────┐ │
│ │ vCenter      │  │ Hierarchical │ │
│ │ Update       │  │ Multi-Site   │ │
│ │ Manager      │  │ Distribution │ │
│ └──────────────┘  └──────────────┘ │
└────────────────────────────────────┘

This diagram illustrates the complete flow of DSvClient from configuration to consumption:

1. Configuration Files: DSvClient begins by reading two key configuration files:
   • config.toml – Controls performance parameters, logging, and exclusions
   • sources.toml – Defines what content to download and from where
2. Core Processing: The application initializes components, parses arguments, and prepares the environment
3. Download Pipelines: Multiple specialized pipelines handle different content types:
   • ESXi updates (patches, base images)
   • VCSA updates (appliance upgrades)
   • vSAN HCL (hardware compatibility lists)
   • Vendor-specific add-ons (drivers, utilities)
4. Verification: All downloaded content undergoes cryptographic verification:
   • SHA-1 and SHA-256 checksum validation
   • Parallel processing for performance
   • Error tracking for failed verifications
5. Repository Structure: Content is organized into a VMware-compatible folder hierarchy
6. Optional Web Deployment: The repository can be exposed via web servers for network distribution
7. Consumption: The repository serves as a source for Update Manager or multi-site distribution

The separation of concerns in DSvClient’s architecture makes it efficient and maintainable. Each pipeline handles specific content types while sharing common infrastructure for downloading, verification, and error handling. This modular approach also allows for easy extension to support new content types or sources as VMware’s ecosystem evolves.

DSvClient Repository Structure: A Visual Breakdown

When you run DSvClient with a download path, it creates an organized repository structure that mirrors VMware’s official update repositories. Here’s a visual breakdown of the typical folder structure you’ll see based on the default sources.toml configuration:

/download_path
│
│
├── addon-main                     # Hardware vendor add-ons
│   ├── index.xml                  # Add-on index file
│   └── addon
│       ├── HTI                    # HP components
│       │   └── vib20
│       │       ├── ilo
│       │       ├── amsd
│       │       └── ... (other HP VIBs)
│       │
│       ├── DELL                   # Dell components
│       │   └── vib20
│       │       ├── iDRAC
│       │       ├── BIOS
│       │       └── ... (other Dell VIBs)
│       │
│       └── NEC                    # NEC components
│           └── vib20
│               └── ... (NEC VIBs)
│
└── ...
│
└── ...
│
└── ...
│
└── vcsa                           # vCenter Server Appliance updates
    └── 8.0.2                      # Version-specific folders
        ├── manifest-latest.xml    # Update manifest
        ├── vmware-vcsa-8.0.2.0*   # VCSA ISO components
        └── ... (other VCSA files)

This hierarchy is automatically created by DSvClient based on the sources defined in your sources.toml file. The default sources.toml includes entries for:

• ESXi Base Repository – Contains core ESXi updates and patches
• Hardware Vendor Add-ons – Drivers and utilities from partners like HPE, Dell and Cisco
• vSAN HCL Download – The vSAN Hardware Compatibility List
• VCSA Update Repositories – vCenter Server Appliance updates for various versions
• Async Driver Updates – Out-of-band driver releases from VMware

When DSvClient runs, it traverses the URLs in sources.toml, intelligently downloading and organizing files to create this consistent structure. VIB files (VMware Installation Bundles) are stored in their appropriate vendor folders, while metadata files like XML indexes are placed at the proper hierarchy levels.

The organization automatically matches what vCenter Update Manager expects, allowing you to point your internal web server directly at this folder structure without any additional configuration. This careful mirroring of VMware’s repository structure is what makes DSvClient so effective for maintaining internal update repositories.

The exclude section in config.toml lets you omit specific paths (like those “NEC” VIBs shown in the example configuration) to optimize storage space and download time, while still maintaining the overall repository structure.

DSvClient Decision Flow Chart

Here is a simplified visual representation of the applications decision flow

┌─────────────────────────┐
│ DSvClient Start         │
└──────────────┬──────────┘
               │
               ▼
┌─────────────────────────┐
│ Parse Configuration     │
│ - Read config.toml      │
│ - Read sources.toml     │
└──────────────┬──────────┘
               │
               ▼
┌─────────────────────────┐
│ Process Command Args    │
└──────────────┬──────────┘
               │
           ┌───┴────┐
           ▼        ▼
┌────────────┐    ┌────────────┐
│ Path Only  │    │ Path with  │
│ (Download) │    │ --verify   │
└─────┬──────┘    └──────┬─────┘
      │                  │
      ▼                  │
┌────────────────────────┐
│ Download Mode          │◄─────────────────┐
├────────────────────────┤                  │
│ For each source in     │                  │
│ sources.toml:          │                  │
└──────────┬─────────────┘                  │
           │                                │
           ▼                                │
┌────────────────────────┐                  │
│ Source Type Decision   │                  │
└──────────┬─────────────┘                  │
           │                                │
     ┌─────┼────────┬─────────┬────────┐    │
     ▼     ▼        ▼         ▼        ▼    │
┌─────────┐ ┌───────┐ ┌───────┐ ┌──────────┐│
│ ESXi    │ │ VCSA  │ │ vSAN  │ │ Add-ons  ││
│ Updates │ │ Files │ │ HCL   │ │ & Drivers││
└────┬────┘ └───┬───┘ └───┬───┘ └────┬─────┘│
     │          │         │          │      │
     └──────────┴────┬────┴──────────┘      │
                     │                       │
                     ▼                       │
┌────────────────────────┐                   │
│ For each file:         │                   │
└──────────┬─────────────┘                   │
           │                                 │
           ▼                                 │
┌────────────────────────┐                   │
│ File Exists?           │                   │
└──────────┬─────────────┘                   │
           │                                 │
      ┌────┴─────┐                           │
      ▼          ▼                           │
┌──────────┐  ┌──────────┐                   │
│   No     │  │   Yes    │                   │
└────┬─────┘  └────┬─────┘                   │
     │             │                         │
     │             ▼                         │
     │      ┌────────────────────┐           │
     │      │ Verify Checksum    │           │
     │      └────────┬───────────┘           │
     │               │                       │
     │        ┌──────┴─────┐                 │
     │        ▼            ▼                 │
     │  ┌──────────┐  ┌──────────┐           │
     │  │ Valid    │  │ Invalid  │           │
     │  └──────────┘  └────┬─────┘           │
     │        │            │                 │
     └────────┐            │                 │
              │            │                 │
              │            ▼                 │
              │    ┌─────────────────────┐   │
              │    │ Delete Invalid File │   │
              │    └──────────┬──────────┘   │
              │               │              │
              └───────────────┘              │
                              │              │
                              ▼              │
                     ┌─────────────────────┐ │
                     │ Download File       │ │
                     └──────────┬──────────┘ │
                                │            │
                                ▼            │
                     ┌─────────────────────┐ │
                     │ Add to Processed    │ │
                     │ Files List          │ │
                     └──────────┬──────────┘ │
                                │            │
                                └────────────┘
                                │
                                ▼
           ┌─────────────────────────┐
           │ --verify Flag Present?  │
           └──────────┬──────────────┘
                      │
               ┌──────┴─────┐
               ▼            ▼
        ┌──────────┐  ┌──────────┐
        │   No     │  │   Yes    │
        └────┬─────┘  └────┬─────┘
             │             │
             │             ▼
             │     ┌────────────────────┐
             │     │ Verification Mode  │
             │     ├────────────────────┤
             │     │ For each file in   │
             │     │ repository:        │
             │     └────────┬───────────┘
             │              │
             │              ▼
             │     ┌────────────────────┐
             │     │ Verify Checksum    │
             │     └────────┬───────────┘
             │              │
             │          ┌───┴────┐
             │          ▼        ▼
             │    ┌──────────┐ ┌──────────┐
             │    │  Valid   │ │ Invalid  │
             │    └─────┬────┘ └────┬─────┘
             │          │           │
             │          │           ▼
             │          │    ┌─────────────────────┐
             │          │    │ Report Mismatch     │
             │          │    └─────────┬───────────┘
             │          │              │
             │          └──────────────┘
             │                         │
             └─────────────────────────┘
                                      │
                                      ▼
                          ┌──────────────────────┐
                          │ Exit with status     │
                          │ code based on errors │
                          └──────────────────────┘

Decision Flow Explanation

DSvClient follows a logical decision tree to determine what actions to take. Here’s how the application makes decisions:

1. Initial Configuration

The application starts by parsing two key configuration files:

• config.toml – Contains operational parameters like concurrent download limits, verification settings, and exclusion patterns
• sources.toml – Defines what content to download and from where (ESXi updates, VCSA files, vSAN HCL, vendor add-ons)

2. Command Argument Processing

DSvClient then processes the command-line arguments to determine the primary mode of operation:

• Path only (e.g., DSvClient /path/to/repo) – Default download mode
• Path with –verify flag (e.g., DSvClient /path/to/repo --verify) – Verification mode

3. Download Mode Logic

When in download mode, DSvClient:

1. Processes each source defined in sources.toml
2. For each source, determines the appropriate handler based on the content type:
   • ESXi updates processor
   • VCSA files processor
   • vSAN HCL processor
   • Add-ons and drivers processor
3. For each file within each source:
   • Checks if the file already exists locally
   • If it exists, verifies its checksum
   • If the checksum is valid, skips the file
   • If the file doesn’t exist or has an invalid checksum, downloads the file
   • Adds the file to the processed files list to avoid redundant operations

4. Verification Mode Logic

When the –verify flag is provided (either directly or after download):

1. The application scans all files in the repository
2. For each file, it verifies the checksum against expected values
3. Valid files are noted as passing verification
4. Invalid files are reported as mismatches
5. A summary report is generated showing verification results

This decision flow ensures that DSvClient operates efficiently, avoiding unnecessary downloads while maintaining repository integrity. The semaphore-based concurrency controls (visible in the code) ensure that the application can process multiple files simultaneously without overwhelming system resources.

Serving Internal Update Repositories with DSvClient

A powerful extension to DSvClient’s capabilities is setting up an internal web server to distribute your meticulously maintained repositories across your organization. This approach offers significant advantages for enterprises with multiple VMware environments or distributed teams.

Why Expose DSvClient Repositories via Web Server?

By serving your DSvClient repositories through a web server, you can:

1. Create a centralized update source for all your environments
2. Reduce external bandwidth consumption by downloading updates once for the entire organization
3. Enforce update standardization across teams and departments
4. Implement controlled update rollouts through repository staging
5. Provide secure access to updates in restricted network segments

Implementation Options

Setting up a web server to host your DSvClient repository is straightforward. Here are three effective approaches:

1. Simple HTTP Server for Internal Networks

For basic needs, a lightweight HTTP server works perfectly:

# On Linux with Apache 
sudo mkdir -p /var/www/html/vmware-updates 
sudo ln -s /path/to/dsvclient-repository /var/www/html/vmware-updates 
sudo systemctl restart apache2 

# On Windows with IIS 
New-WebVirtualDirectory -Site "Default Web Site" -Name "vmware-updates" -PhysicalPath "C:\VMware\UpdateRepo"

2. Authenticated Access with NGINX

For environments requiring user authentication:

server { listen 80; server_name updates.internal.example.com; location / { root /path/to/dsvclient-repository; autoindex on; auth_basic "VMware Update Repository"; auth_basic_user_file /etc/nginx/.htpasswd; } }

3. Docker-Based Solution for Quick Deployment

For portable, consistent deployments:

docker run -d \ --name vmware-update-repo \ --restart=always \ -p 8080:80 \ -v /path/to/dsvclient-repository:/usr/share/nginx/html:ro \ -e NGINX_ENTRYPOINT_QUIET_LOGS=1 \ nginx:alpine

Implementation: Multi-Site Architecture

Implementing a hierarchical update distribution system using DSvClient:

1. Headquarters: Maintains the master repository with DSvClient and serves it via HTTPS

# Weekly update 
job 0 2 * * 0 /usr/local/bin/dsvclient /opt/vmware/master-repo >> /var/log/dsvclient.log 2>&1

2. Regional Offices: Synchronize from headquarters using a combination of rsync and DSvClient verification

# Pull updates from HQ 
rsync -avz --delete https://hq-updates.example.com/vmware-updates/ /opt/vmware/regional-repo/ 
# Verify integrity 
/usr/local/bin/dsvclient /opt/vmware/regional-repo --verify

3. Local Sites: Configure Update Manager to use their regional repository URL

This architecture reduced their external bandwidth usage by over 95% while ensuring consistent patch levels across all sites.

Integrating with vCenter Update Manager

The web-served repositories integrate seamlessly with vCenter:

1. In the vSphere Client, navigate to Update Manager > Settings
2. Select Download Settings
3. Choose Use a shared repository
4. Enter your internal URL: http://updates.internal.example.com/vmware-updates/

Now your Update Manager will source all patches and updates from your carefully maintained internal repository.

Security Considerations

When serving update repositories internally:

1. Consider HTTPS for sensitive networks, using internal CA certificates
2. Implement access controls based on network segmentation or authentication
3. Validate checksums at each distribution tier with DSvClient’s --verify option
4. Maintain logs of repository access for compliance purposes
5. Regularly scan repository content with security tools

This approach provides a perfect balance of security and accessibility for your VMware updates.

By leveraging DSvClient with an internal web server, you create a robust, efficient distribution system for VMware updates that minimizes external dependencies while maximizing control over your update process. The result is a more resilient, bandwidth-efficient infrastructure that maintains consistent standards across your entire VMware estate.

Note on Broadcom download token

I have not have time to test DSvClient with the new download system Broadcom has put in place.

As it is compatible with Update Manager without a special release, I’m expecting that the data flow remains the same.

So only a url change may be needed.