8.5 KiB
Internals of build process
The purpose of this document is to explain build process internals with subtle nuances.
This document is not by any means complete.
The ultimate source of truth is the code in .\build.psm1
that's getting executed on the corresponding CI system.
This document assumes that you can successfully build PowerShell from sources for your platform.
Top directory
We are calling dotnet
tool build for $Top
directory
src\powershell-win-core
for CoreCLR on Windows.src\powershell-unix
for CoreCLR on Linux and macOS.
Dummy dependencies
We use dummy dependencies between projects to leverage dotnet
build functionality.
For example, src\powershell-win-core\powershell-win-core.csproj
has dependency on Microsoft.PowerShell.PSReadLine
,
but in reality, there is no build dependency.
Dummy dependencies allows us to build just $Top
folder, instead of building several folders.
Dummy dependencies rules
If assembly is part of CoreCLR build,
it should be listed as a dependency for $Top
folder (src\powershell-unix
or src\powershell-win-core
)
Preliminary steps
ResGen
Until the .NET CLI dotnet-resgen
tool supports the generation of strongly typed classes,
we run our own tool C# ResGen tool.
While the Start-PSBuild
command runs this automatically via the Start-ResGen
function,
it does not require PowerShell.
The same command can be run manually:
cd src/ResGen
dotnet restore
dotnet run
Running the program does everything else:
- For each project, given a
resources
folder, create agen
folder. - For each
*.resx
file from theresources
folder, fill in a strongly typed C# class, and write it out to the corresponding*.cs
file in thegen
folder.
These files are not automatically updated on each build, as the project lacks the ability to detect changes. Thus, running it for every build would break incremental recompilation.
If you pull new commits and get an error about missing strings,
you likely need to delete the gen
folders and re-run the tool.
Type Catalog
A pre-generated catalog of C# types is used in PowerShell to help type resolution.
Generating this catalog is a pre-build step that is run via Start-TypeGen
,
which Start-PSBuild
calls.
Again, however, PowerShell is not required.
The necessary steps can be run manually:
cd ../TypeCatalogGen
dotnet restore
dotnet run ../System.Management.Automation/CoreCLR/CorePsTypeCatalog.cs powershell.inc
The file powershell.inc
is generated by running a custom MSBuild target,
which can be set-up by navigating to the src
directory and running the following commands:
targetFile="Microsoft.PowerShell.SDK/obj/Microsoft.PowerShell.SDK.csproj.TypeCatalog.targets"
cat > $targetFile <<-"EOF"
<Project>
<Target Name="_GetDependencies"
DependsOnTargets="ResolveAssemblyReferencesDesignTime">
<ItemGroup>
<_RefAssemblyPath Include="%(_ReferencesFromRAR.ResolvedPath)%3B" Condition=" '%(_ReferencesFromRAR.Type)' == 'assembly' And '%(_ReferencesFromRAR.PackageName)' != 'Microsoft.Management.Infrastructure' " />
</ItemGroup>
<WriteLinesToFile File="$(_DependencyFile)" Lines="@(_RefAssemblyPath)" Overwrite="true" />
</Target>
</Project>
EOF
dotnet msbuild Microsoft.PowerShell.SDK/Microsoft.PowerShell.SDK.csproj /t:_GetDependencies "/property:DesignTimeBuild=true;_DependencyFile=$(pwd)/src/TypeCatalogGen/powershell.inc" /nologo
powershell.inc
contains the resolved paths to the DLLs of each dependency of PowerShell,
and is taken as input to the TypeCatalogGen
tool,
which generates a source file CorePsTypeCatalog.cs
for the Microsoft.PowerShell.CoreCLR.AssemblyLoadContext
project.
The error The name 'InitializeTypeCatalog' does not exist in the current context
indicates that the CorePsTypeCatalog.cs
source file does not exist,
so follow the steps to generate it.
Native Components
On Windows, PowerShell Core depends on the WinRM plugin pwrshplugin.dll
to enable remoting over WinRM.
On Linux/macOS, PowerShell Core depends on the binary libpsl-native.so/libpsl-native.dylib
to provide some necessary supports.
Building those native components requires setting up additional dependencies,
which could be a burden to those who don't seek to make changes to the native components.
At the meantime, the native component code seldom changes,
so it doesn't make sense to always build them with Start-PSBuild
.
Therefore, we decided to wrap the native components into NuGet packages,
so that we only need to build them once when changes are made,
and then reuse the produced binaries for many builds subsequently.
The NuGet package for pwrshplugin.dll
is psrp.windows
,
and the NuGet package for libpsl-native
is libpsl
.
psrp.windows
To build pwrshplugin.dll
, you need to install Visual Studio 2015 and run Start-PSBootstrap -BuildWindowsNative
to install the prerequisites.
Then run Start-BuildNativeWindowsBinaries
to build the binary.
For example, the following builds the release flavor of the binary targeting x64 architecture.
Start-BuildNativeWindowsBinaries -Configuration Release -Arch x64
After that, the binary pwrshplugin.dll
and its PDB file will be placed under 'src/powershell-win-core'.
To create a new NuGet package for pwrshplugin.dll
, first you need to get the psrp.windows.nuspec
from an existing psrp.windows
package.
You can find it at ~/.nuget/packages/psrp.windows
on your windows machine if you have recently built PowerShell on it.
Or you can download the existing package from powershell-core feed.
Once you get psrp.windows.nuspec
, copy it to an empty folder.
Then you need to build pwrshplugin.dll
targeting both win-x64
and win-x86
on Windows 10.
After building successfully, copy the produced files to the same folder,
and create the same layout of files as in the existing package.
The layout of files should look like this:
\---runtimes
+---win-x64
| \---native
| pwrshplugin.dll
| pwrshplugin.pdb
|
\---win-x86
\---native
pwrshplugin.dll
pwrshplugin.pdb
Lastly, run nuget pack .
from within the folder. Note that you may need the latest nuget.exe
.
libpsl
For linux-arm
, you need to run Start-PSBootstrap -BuildLinuxArm
to install additional prerequisites to build libpsl-native
.
Note that currently you can build linux-arm
only on a Ubuntu machine.
For linux-x64
and macOS, the initial run of Start-PSBootstrap
would be enough -- no additional prerequisite required.
After making sure the prerequisites are met, run Start-BuildNativeUnixBinaries
to build the binary:
## Build targeting linux-x64 or macOS
Start-BuildNativeUnixBinaries
## Build targeting linux-arm
Start-BuildNativeUnixBinaries -BuildLinuxArm
After the build succeeds, the binary libpsl-native.so
(libpsl-native.dylib
on macOS) will be placed under src/powershell-unix
.
To create a new NuGet package for libpsl-native
, first you need to get the libpsl.nuspec
from an existing libpsl
package.
You can find it at ~/.nuget/packages/libpsl
on your Linux or macOS machine if you have recently built PowerShell on it.
Or you can download the existing package from powershell-core feed.
Once you get psrp.windows.nuspec
, copy it to an empty folder on your Windows machine.
Then you need to build three binaries of libpsl-native
targeting linux-x64
, linux-arm
and osx
respectively.
Please note that, in order for the linux-x64
binary libpsl-native.so
to be portable to all other Linux distributions,
the linux-x64
binary needs to be built on CentOS 7
(.NET Core Linux native binaries are also built on CentOS 7 to ensure that they don't depend on newer glibc
).
After building successfully, copy those three binaries to the same folder, and create the same layout of files as in the existing package. The layout of files should look like this:
└── runtimes
├── linux-arm
│ └── native
│ └── libpsl-native.so
├── linux-x64
│ └── native
│ └── libpsl-native.so
└── osx
└── native
└── libpsl-native.dylib
Lastly, run nuget pack .
from within the folder. Note that you may need the latest nuget.exe
.