235 lines
10 KiB
Markdown
235 lines
10 KiB
Markdown
# Internals of build process
|
|
|
|
The purpose of this document is to explain build process **internals** with subtle nuances.
|
|
This document is not by any means complete.
|
|
The ultimate source of truth is the code in `.\build.psm1` that's getting executed on the corresponding CI system.
|
|
|
|
This document assumes that you can successfully build PowerShell from sources for your platform.
|
|
|
|
## Top directory
|
|
|
|
We are calling `dotnet` tool build for `$Top` directory
|
|
|
|
- `src\powershell-win-core` for CoreCLR on Windows.
|
|
- `src\powershell-unix` for CoreCLR on Linux and macOS.
|
|
|
|
### Dummy dependencies
|
|
|
|
We use dummy dependencies between projects to leverage `dotnet` build functionality.
|
|
For example, `src\powershell-win-core\powershell-win-core.csproj` has dependency on `Microsoft.PowerShell.Commands.Diagnostics.csproj`,
|
|
but in reality, there is no build dependency.
|
|
|
|
Dummy dependencies allows us to build just `$Top` folder, instead of building several folders.
|
|
|
|
### Dummy dependencies rules
|
|
|
|
If assembly is part of CoreCLR build,
|
|
it should be listed as a dependency for `$Top` folder (`src\powershell-unix` or `src\powershell-win-core`)
|
|
|
|
## Preliminary steps
|
|
|
|
### ResGen
|
|
|
|
Until the .NET CLI `dotnet-resgen` tool supports the generation of strongly typed resource access classes
|
|
(tracked by [Microsoft/msbuild #2272](https://github.com/Microsoft/msbuild/issues/2272)),
|
|
we run our own C# [ResGen tool](../../src/ResGen).
|
|
While the `Start-PSBuild` command runs this automatically via the `Start-ResGen` function,
|
|
it does *not* require PowerShell.
|
|
The same command can be run manually:
|
|
|
|
```sh
|
|
cd src/ResGen
|
|
dotnet restore
|
|
dotnet run
|
|
```
|
|
|
|
Running the program does the following work:
|
|
|
|
- For each project, given a `resources` folder, create a `gen` folder.
|
|
- For each `*.resx` file from the `resources` folder,
|
|
create a strongly typed C# resource access class,
|
|
and write it to the corresponding `*.cs` file in the `gen` folder.
|
|
|
|
These files are *not* automatically updated on each build,
|
|
as the project lacks the ability to detect changes.
|
|
Thus, running it for every build would break incremental recompilation.
|
|
|
|
If you pull new commits and get an error about missing strings,
|
|
you likely need to delete the `gen` folders and re-run the tool.
|
|
|
|
### Type Catalog
|
|
|
|
A pre-generated catalog of C# types is used in PowerShell to help type resolution.
|
|
Generating this catalog is a pre-build step that is run via `Start-TypeGen`,
|
|
which `Start-PSBuild` calls.
|
|
Again, however, PowerShell is not required.
|
|
The necessary steps can be run manually:
|
|
|
|
```sh
|
|
cd ../TypeCatalogGen
|
|
dotnet restore
|
|
dotnet run ../System.Management.Automation/CoreCLR/CorePsTypeCatalog.cs powershell.inc
|
|
```
|
|
|
|
The file `powershell.inc` is generated by running a custom MSBuild target.
|
|
`Start-TypeGen` handles generating this file,
|
|
but you can also do it manually by navigating to the `src` directory and running the following commands:
|
|
|
|
```sh
|
|
targetFile="Microsoft.PowerShell.SDK/obj/Microsoft.PowerShell.SDK.csproj.TypeCatalog.targets"
|
|
cat > $targetFile <<-"EOF"
|
|
<Project>
|
|
<Target Name="_GetDependencies"
|
|
DependsOnTargets="ResolveAssemblyReferencesDesignTime">
|
|
<ItemGroup>
|
|
<_RefAssemblyPath Include="%(_ReferencesFromRAR.HintPath)%3B" Condition=" '%(_ReferencesFromRAR.NuGetPackageId)' != 'Microsoft.Management.Infrastructure' "/>
|
|
</ItemGroup>
|
|
<WriteLinesToFile File="$(_DependencyFile)" Lines="@(_RefAssemblyPath)" Overwrite="true" />
|
|
</Target>
|
|
</Project>
|
|
EOF
|
|
dotnet msbuild Microsoft.PowerShell.SDK/Microsoft.PowerShell.SDK.csproj /t:_GetDependencies "/property:DesignTimeBuild=true;_DependencyFile=$(pwd)/TypeCatalogGen/powershell.inc" /nologo
|
|
```
|
|
|
|
`powershell.inc` contains the resolved paths to the DLLs of each dependency of PowerShell,
|
|
and is taken as input to the [`TypeCatalogGen`](../../src/TypeCatalogGen) tool,
|
|
which generates the source file `CorePsTypeCatalog.cs` for the `System.Management.Automation` project.
|
|
|
|
The error `The name 'InitializeTypeCatalog' does not exist in the current context`
|
|
indicates that the `CorePsTypeCatalog.cs` source file does not exist,
|
|
so follow the steps to generate it.
|
|
|
|
## Native Components
|
|
|
|
On Windows, PowerShell Core depends on the WinRM plugin `pwrshplugin.dll` to enable remoting over WinRM.
|
|
On Linux/macOS, PowerShell Core depends on the binary `libpsl-native.so/libpsl-native.dylib` to provide some necessary supports.
|
|
|
|
Building those native components requires setting up additional dependencies,
|
|
which could be a burden to those who don't seek to make changes to the native components.
|
|
At the meantime, the native component code seldom changes,
|
|
so it doesn't make sense to always build them with `Start-PSBuild`.
|
|
Therefore, we decided to wrap the native components into NuGet packages,
|
|
so that we only need to build them once when changes are made,
|
|
and then reuse the produced binaries for many builds subsequently.
|
|
|
|
The NuGet package for `pwrshplugin.dll` is `psrp.windows`,
|
|
and the NuGet package for `libpsl-native` is `libpsl`.
|
|
|
|
### Windows packages: PSRP.Windows and PowerShell.Core.Instrumentation
|
|
|
|
To build `pwrshplugin.dll` and `PowerShell.Core.Instrumentation.dll`, you need to install Visual Studio 2017 and run `Start-PSBootstrap -BuildWindowsNative` to install the prerequisites.
|
|
|
|
Ensure the following individual components are selected:
|
|
|
|
- [ ] VC++ 2017 v141 toolset (x86, x64)
|
|
- [ ] Visual C++ compilers and libraries for ARM
|
|
- [ ] Visual C++ compilers and libraries for ARM64
|
|
- [ ] Visual C++ tools for CMake
|
|
- [ ] Visual C++ ATL Support
|
|
- [ ] Windows 10 SDK (10.0.16299.0) for Desktop C++ (ARM and ARM64)
|
|
- [ ] Windows 10 SDK (10.0.16299.0) for Desktop C++ (x86 and x64)
|
|
|
|
Ensure [CMake](https://cmake.org/download/) 3.10.0 or newer is installed which supports VS2017 and ARM64 generator.
|
|
|
|
Then run `Start-BuildNativeWindowsBinaries` to build the binary.
|
|
For example, the following builds the release flavor of the binary targeting arm64 architecture.
|
|
|
|
```powershell
|
|
Start-BuildNativeWindowsBinaries -Configuration Release -Arch x64_arm64
|
|
```
|
|
|
|
Be sure to build and test for all supported architectures: `x86`, `x64`, `x64_arm`, and `x64_arm64`.
|
|
|
|
The `x64_arm` and `x64_arm64` architectures mean that the host system needs to be x64 to cross-compile to ARM.
|
|
When building for multiple architectures, be sure to use the `-clean` switch as cmake will cache the previous run and the wrong compiler will be used to generate the subsequent architectures.
|
|
|
|
After that, the binary `pwrshplugin.dll`, its PDB file, and `powershell.core.instrumentation.dll` will be placed under `src\powershell-win-core`.
|
|
|
|
To create a new NuGet package for `pwrshplugin.dll`, first you need to get the `psrp.windows.nuspec` from an existing `psrp.windows` package.
|
|
You can find it at `~/.nuget/packages/psrp.windows` on your windows machine if you have recently built PowerShell on it.
|
|
Or you can download the existing package from [powershell-core feed](https://powershell.myget.org/feed/powershell-core/package/nuget/psrp.windows).
|
|
Once you get `psrp.windows.nuspec`, copy it to an empty folder and update the `<version>` element.
|
|
|
|
After building successfully, copy the produced files to the same folder,
|
|
and create the same layout of files as in the existing package.
|
|
The layout of files should look like this:
|
|
|
|
```none
|
|
\---runtimes
|
|
+---win-x64
|
|
| \---native
|
|
| pwrshplugin.dll
|
|
| pwrshplugin.pdb
|
|
|
|
|
+---win-x86
|
|
| \---native
|
|
| pwrshplugin.dll
|
|
| pwrshplugin.pdb
|
|
+---win-arm
|
|
| \---native
|
|
| pwrshplugin.dll
|
|
| pwrshplugin.pdb
|
|
\---win-arm64
|
|
\---native
|
|
pwrshplugin.dll
|
|
pwrshplugin.pdb
|
|
```
|
|
|
|
Have the DLLs signed with `authenticode dual` certificate and run `nuget pack` from the parent of the `runtimes` folder where `psrp.windows.nuspec` resides.
|
|
Be sure to use the latest recommended version of [nuget.exe](https://www.nuget.org/downloads).
|
|
|
|
Publish latest nupkg to https://powershell.myget.org/feed/powershell-core/package/nuget/psrp.windows.
|
|
|
|
`PowerShell.Core.Instrumentation.dll` NuGet package is created the same way, but in a separate directory following the same layout above.
|
|
To create a new NuGet package for `PowerShell.Core.Instrumentation.dll`, you will need the `PowerShell.Core.Instrumentation.nuspec` found in the repo under `src\PowerShell.Core.Instrumentation`.
|
|
|
|
Publish latest nupkg to https://powershell.myget.org/feed/powershell-core/package/nuget/PowerShell.Core.Instrumentation.
|
|
|
|
### libpsl
|
|
|
|
For `linux-arm`, you need to run `Start-PSBootstrap -BuildLinuxArm` to install additional prerequisites to build `libpsl-native`.
|
|
Note that currently you can build `linux-arm` only on a Ubuntu machine.
|
|
|
|
For `linux-x64` and macOS, the initial run of `Start-PSBootstrap` would be enough -- no additional prerequisite required.
|
|
|
|
After making sure the prerequisites are met, run `Start-BuildNativeUnixBinaries` to build the binary:
|
|
|
|
```powershell
|
|
## Build targeting linux-x64 or macOS
|
|
Start-BuildNativeUnixBinaries
|
|
|
|
## Build targeting linux-arm
|
|
Start-BuildNativeUnixBinaries -BuildLinuxArm
|
|
```
|
|
|
|
After the build succeeds, the binary `libpsl-native.so` (`libpsl-native.dylib` on macOS) will be placed under `src/powershell-unix`.
|
|
|
|
To create a new NuGet package for `libpsl-native`, first you need to get the `libpsl.nuspec` from an existing `libpsl` package.
|
|
You can find it at `~/.nuget/packages/libpsl` on your Linux or macOS machine if you have recently built PowerShell on it.
|
|
Or you can download the existing package from [powershell-core feed](https://powershell.myget.org/feed/powershell-core/package/nuget/libpsl).
|
|
Once you get `psrp.windows.nuspec`, copy it to an empty folder on your Windows machine.
|
|
|
|
Then you need to build three binaries of `libpsl-native` targeting `linux-x64`, `linux-arm` and `osx` respectively.
|
|
**Please note that, in order for the `linux-x64` binary `libpsl-native.so` to be portable to all other Linux distributions,
|
|
the `linux-x64` binary needs to be built on CentOS 7**
|
|
(.NET Core Linux native binaries are also built on CentOS 7 to ensure that they don't depend on newer `glibc`).
|
|
|
|
After building successfully, copy those three binaries to the same folder,
|
|
and create the same layout of files as in the existing package.
|
|
The layout of files should look like this:
|
|
|
|
```none
|
|
└── runtimes
|
|
├── linux-arm
|
|
│ └── native
|
|
│ └── libpsl-native.so
|
|
├── linux-x64
|
|
│ └── native
|
|
│ └── libpsl-native.so
|
|
└── osx
|
|
└── native
|
|
└── libpsl-native.dylib
|
|
```
|
|
|
|
Lastly, run `nuget pack .` from within the folder. Note that you may need the latest `nuget.exe`.
|