Inside the Nuget bootstraping process

A few days ago the PowerShell Team announced on their blog that PowerShellGet has been open-sourced. Both PowerShellGet and PackageManagement modules are now available on the PowerShell Gallery to more easily consume/update these modules and on github to contribute to these projects:

• PowerShellGet module on github
• PackageManagement module on github
• PowerShellGet module on the PowerShell Gallery
• PackageManagement module on PowerShell Gallery

The article says

PowerShellGet has a dependency on PackageManagement.

That’s true.
I’d add that you cannot use the Find-Module cmdlet from the PowerShellGet module before the PackagementManagement module installed the Nuget provider.

Do I have Nuget listed as a provider?

# You can either do
Get-PackageProvider
# and see if Nuget is in the list
# or
Get-PackageProvider | Where Name -eq 'NuGet'

But, as soon as you do

Get-PackageProvider -Name NuGet -ErrorAction SilentlyContinue

…or if you use the Find-Module cmdlet, you’ll be asked to complete the Nuget provider installation.

Find-Module -Name Pester -Repository PSGallery

Did you notice that the messages don’t say exactly the same thing and that one proposes to install a minimum version 2.8.5.201 and the other 2.8.5.207?
The Get-PackageProvider from the PackageManagement module seems more accurate than the Find-Module cmdlet from the PowerShellGet module.
Why?
Well, because the PowerShellGet module hardcodes the version 2.8.5.201

…whereas the PackageManagement module built the version on the fly:

After opening the PSModule.psm1 file located in C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1, I can notice that many functions inside that module call either:

# the BootstrapNuGetExe switch 
# used by the Publish-Module function
Install-NuGetClientBinaries -CallerPSCmdlet $PSCmdlet `
-BootstrapNuGetExe

or

Install-NuGetClientBinaries -CallerPSCmdlet $PSCmdlet

The reason is explained on this page: What’s the difference between nuget-anycpu.exe, Microsoft.PackageManagement.NuGetProvider.dll and nuget.exe?

[…]
nuget.exe is used by PowerShellGet to publish packages
[…]
Microsoft.PackageManagement.NuGetProvider.dll is used by OneGet and PowerShellGet to discover and install packages.
[…]
Only the Publish-Module cmdlet of PowerShellGet will require nuget.exe.

So far, we’ve seen that there are two components that can be downloaded and installed.
According to the content of the PSModule.psm1 file, Nuget.exe is downloaded by the PowerShellGet module and uses the following URL to get it:

# go fwlink for 'https://nuget.org/nuget.exe'
$script:NuGetClientSourceURL = 
'http://go.microsoft.com/fwlink/?LinkID=690216&clcid=0x409'

Where does Microsoft.PackageManagement.NuGetProvider.dll come from?

Microsoft proposes in their recent blog post to do the following to install the Nuget provider:

Install-PackageProvider -Name Nuget –Force –Verbose

The same piece of information is also mentioned in the FAQ of the PackagementManagement module on github: How do I install a package provider such as NuGet provider if I do not have Internet connection on my box?

Again inside the the PSModule.psm1 file (the PowerShellGet module), its internal Install-NuGetClientBinaries function actually launches the following to install the Nuget provider:

PackageManagement\Install-PackageProvider -Name 'Nuget' `
-MinimumVersion ([Version]'2.8.5.201')  `
-Scope $scope -Force

Instead of the above command, I propose to do directly the following and add the Debug switch to uncover what happens behind the scene:

Get-PackageProvider -Name NuGet `
-ErrorAction SilentlyContinue -Verbose -Debug

The following hardcoded URL is immediately used: https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409

Let’s examine this URL and set the MaximumRedirection to 0 to avoid any redirection.

$HT = @{
 Uri = 'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409'
 MaximumRedirection = 0
 ErrorAction = 'SilentlyContinue'
}
$req = Invoke-WebRequest @HT
# the page has moved
$req
# to this location
$req.Headers.Location

We can see the page is redirected to another URL:

The file downloaded from https://az818661.vo.msecnd.net/providers/providers.masterList.feed.swidtag is a swidtag file (Software Identiy Tag) written in XML:

Now, to get the latest version of the Microsoft.PackageManagement.NuGetProvider.dll file, I need to read the content of the providers.masterList.feed.swidtag XML file:

# set a variable
$MasterSwidTagURI = 'https://az818661.vo.msecnd.net/providers/providers.masterList.feed.swidtag'

# Get the link of the Nuget provider
([xml]([System.Text.Encoding]::UTF8.GetString(
(Invoke-WebRequest -Uri $MasterSwidTagURI -MaximumRedirection 0 -ErrorAction SilentlyContinue).Content
))).SoftwareIdentity.Link | Where {
    $_.rel -eq 'package' -and
    $_.latest -eq 'true' -and
    $_.name -eq 'nuget'
} | Select -expand href

# or more simply using the Invoke-RestMethod cmdlet
(Invoke-RestMethod $MasterSwidTagURI).SoftwareIdentity.Link | 
Where {
    $_.rel -eq 'package' -and
    $_.latest -eq 'true' -and
    $_.name -eq 'nuget'
} | Select -expand href

To get the latest version of the dll, there’s a specific swidtag file to read from the following URL: https://oneget.org/nuget-2.8.5.207.package.swidtag

Let’s examine the content of this new XML file:

$dllURI = 'https://oneget.org/nuget-2.8.5.207.package.swidtag'
# Get the version
(Invoke-RestMethod -Uri $dllURI).SoftwareIdentity.version
# Get the source (dll download location)
(Invoke-RestMethod -Uri $dllURI).SoftwareIdentity.Link.href
# New info about the file to download
(Invoke-RestMethod -Uri $dllURI).SoftwareIdentity.Payload.File

Now, we know how and from where the file is being downloaded.

As of version 2.8.5.206, there’s a hash being used by the Install-PackageProvider cmdlet to check the integrity of the downloaded Microsoft.PackageManagement.NuGetProvider.dll file.
If the Install-PackageProvider is invoked with the Debug switch, I can see at the end of the debug stream the following line: DEBUG: 00:02:07.7019092 BoostrapRequest::ValidateFileHash

The above hash that appears inside the swidtag file doesn’t look standard.

Here’s how it can be verified:

# Download the dll
Invoke-WebRequest -Uri $((Invoke-RestMethod -Uri $dllURI).SoftwareIdentity.Link.href) -OutFile ~/downloads/$((Invoke-RestMethod -Uri $dllURI).SoftwareIdentity.Payload.File.Name) -Verbose

# Get its SHA512 hash
(Get-FileHash ~/downloads/Microsoft.PackageManagement.NuGetProvider.dll -Algorithm SHA512 | Select -Expand Hash).ToLower()

# Check it matches the one specified in the swidtag file
(
 [BitConverter]::ToString(
  [system.convert]::FromBase64String(
   (Invoke-RestMethod -Uri 'https://oneget.org/nuget-2.8.5.207.package.swidtag').SoftwareIdentity.Payload.File.hash
  )
 ) -replace '-',''
).toLower()