Refactor macOS CI so that tests run in parallel
- Also:
- Fix bootstrap issue where curl was failing for a month (so obviously not needed)
- Change Windows test stages to use PowerShell Core (where we can) to avoid compat issues
## PR Context
This allows running all tests in about the same about of clock time.
* Add shim dll to redirect to appropriate runtime
* Build updates for creating global tool package
* Add building global tool to nuget.yml
* Add azure upload of nuget packages
Split Windows CI into the following Jobs
1. Combine Build and xUnit, which published build artifact
1. Elevated CI tests
1. Unelevated CI test
1. Elevated everything else tests
1. Unelevated everything else tests
1. Package build and test
1. Verify xUnit result
## PR Context
Before this test:
- A CI run would take ~14 minutes
- A Feature run would take ~30 minutes
After this change, it always runs all tests and takes ~20 minutes.
Job `Unelevated everything else tests` is the long poll, taking ~4 minutes longer than the next shortest job. We should be able to get this back down to ~15 if we split that test job somehow.
## PR Summary
Related: #8699
## PR Context
Because `PowerShellGet` does not support publishing/saving module on a per platform basis (see [this](https://github.com/PowerShell/PowerShellGet/issues/273) issue), PowerShell currently also ships the fullclr binaries of `PackageManagement`, which it does not need. Therefore removing it to minimise the package size, this saves 1.19 MB.
Fix broken URL
- Also, move other static analysis tests into that CI.
- Also, make the link analysis a pester test (partly to make sure the step fails in case of an error)
## Motivation
I have a PR where there are many new xUnit tests.
It would also be useful to create new xUnit tests for public APIs.
The number of xUnit tests will increase and their ordering is required.
## PR Summary
- Move C# xUnit tests in new folder. This allows to put new xUnit tests in directory structure in accordance with directory structure where cs files are.
- Use an xUnit TestCaseOrderer attribute to sequentially process tests for `powershell.config.json`.
- Update README.md
- A race condition was fixed which allowed to run all XUnit tests in single batch job.
Major changes are as follows:
- Avoid `SecuritySupport.IsProductBinary` and unnecessary AMSI/suspicious code scan at startup time
- Update `CompiledScriptBlockData.IsProductCode` to avoid unnecessary calls to `IsProductBinary`, which attempts to retrieve catalog signature of the target file.
- Update `PerformSecurityChecks` to skip AMSI and suspicious code scan for the `.psd1` file that contains a safe `HashtableAst` only.
- Use customized `ReadOnlyBag` instead of `ImmutableHashSet` so that we can avoid loading the `System.Collections.Immutable.dll` completely.
- Replace `SHA1` with `CRC32` when generating module analysis cache file name
- This remove the loading of `System.Security.Cryptography.Algorithms.dll` at startup
- Move `ConvertFrom-SddlString` to C# to remove the `Utility.psm1` file.
- Crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation
- Even pwsh with crossgen assemblies spends a lot time in jitting at the startup, about `191.6ms` comparing with `24.7ms` for Windows PowerShell.
- Jitting `Microsoft.ApplicationInsights.dll` takes about `51.6ms`.
- By crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation, the jitting time drops to about `98.9ms`.
Add a function to create a NuGet.config file to consume packages from the specified feed URL using the specified credentials.
- Also, exclude the `nuget.config` file under `src/Modules` from being clean by `Start-PSBuild -clean`
As of #7892, the PowerShell repository no longer uses Git submodules.
This is fantastic from a workflow standpoint, and so all the notes about
how to deal with submodules (and all the build steps explicitly
initializing and updating submodules) can be safely removed.
Native code for PowerShell has been moved to repository PowerShell/PowerShell-Native.
Hence, removing all the native code and related build components.
Add the functionality to build a framework dependent (shared framework) package for PowerShell.
The changes create two packages, one for Windows and other for Linux, due to #if code.
Add daily build non-windows platforms
- Also, make the [Feature] tag work in VSTS for non-windows
- Also, add a way to force feature tests to run
- Also, fix an issue where `-workingdirectory` didn't work when running async
On Debian 9, libmi cannot resolve libssl and libcrypto. This change adds symbolic links to packaging and build to the $PSHOME directory to resolve the issue.
The fix was verified interactively connecting to office 365, importing the session, and ensuring Get-MailBox and Get-User succeed.
Fix#7598
- Add VSTS CI for Windows
- Disable `Access-denied test for Get-Item C:\windows\appcompat\Programs\Install -ErrorAction Stop`, because the path does not always exist
- https://github.com/PowerShell/PowerShell/issues/7553
- Disable `Should give .sys file if the fullpath is specified with hidden and force parameter`, because pagefile.sys doesn't always exist and other files don't meet test's requirement.
- https://github.com/PowerShell/PowerShell/issues/7554
- Disable some `Test-Connection` tests for same reasons they failed on VSTS Linux
- https://github.com/PowerShell/PowerShell/issues/7555
- Disable `Test-FileCatalog should pass when catalog is in the same folder as files being tested`, because the CmdLet does not work in that scenario
- Also, give details needed to investigate when the test fails
- https://github.com/PowerShell/PowerShell/issues/7556
- Update `appveyor.psm1` to work with VSTS
- Update `HelpersRemoting.psm1` `New-RemoteSession` to work for CimSession (discovered an issue during the investigation)
- Update `Test wildcard with drive relative directory path` to work when there are multiple drives
- Disable on non-windows machines since the test is assuming drive letters
- Update `New-CimSession` Tests to requireAdmin
- Also, make sure session name is a string
- Add functions to save and restore psoptions
- update `.gitatttributes` so files clone like they do on appveyor
Update Start-PSPester and Start-PSxUnit to upload test results when running in VSTS
Disable tests which require non-root using when running as root
remove verifications in Test-Connection tests which verify buggy behavior
Skip tests which don't behave correctly in VSTS (due to running in a container?)
- Test-Connection - #7528
- Set-Date - #7529
When trying to build on RHEL, the environment is not recognized as RedHatFamily.
Added detection logic so that appropriate symbolic links are generated.
Remove code from build.psm1 that save the modules to a versioned folder. For servicing reasons after release it is preferred and easier to not have the versioned folder so that we can directly replace modules that need to be serviced (MSI specifically makes it difficult to service a module if the folder path changes).
Fix man page conflict between stable and preview Linux package
- rename preview man page pwsh-preview
Fix /etc/shell entries not being created correctly for preview packages
The main purpose of this was to enable full symbols for windows release build.
Also makes explicit where we are optimizing and where we are not optimizing due to https://github.com/dotnet/corefx/issues/29700
Ubuntu 17.04 is EOL, so we need to update our packaging for Ubuntu 17.10. This updates the package script and build.psm1 to use 17.10 rather than 17.04.
* Build Update
- Change `TargetFramework` to `netcoreapp2.1` and removed unnecessary `RuntimeFrameworkVersion` from `PowerShell.Common.props`
- Update dotnet SDK to 2.1.300-rc1-008662
- Update `TypeGen` target in `Build.psm1` to work with 2.1
- Rename macOS runtime to `osx-x64` as the old build logic expects 10.12 and breaks running on 10.13 system.
- Remove `PackageReference` to `System.Memory` as it's part of dotnetcore 2.1
- Update search for `crossgen` executable to find the matching version
* Test Update
- Update test tools `WebListener` to latest `asp.net core`
- Marked `AuthHeader Redirect` tests as `Pending` due to change in CoreFX
Since a PR added support to opt out of telemetry via an environment variable, we can remove the,
always intended to be a temporary, solution of deleting a file to opt out of telemetry since the
environment variable can be defined at the system level and exist before even installing PowerShell Core.
Because the variable is defined as opt out, a value of true, yes, or 1 means no telemetry is sent.
The release packaging build needs to have 'dotnet' in path, fixed that by adding Find-DotNet
Refactored restore logic so it can be used from both Start-PSBuild and Copy-PSGalleryModules
The release packaging build needs to have 'dotnet' in path, fixed that by adding Find-DotNet
Refactored restore logic so it can be used from both Start-PSBuild and Copy-PSGalleryModules
set Debian front end to non-interactive during apt-get commands, to avoid error with prompting
add libffi-dev, which is required for ruby/fpm
let fpm update to the latest
Fix#6332
This change renames log and logerror functions to a single Write-Log [$message] [-error] function to avoid conflicting with the log command on MacOS.
Attempt to fix the macOS build.
In #6263, I tried flushing the cache and it worked in the PR. However, after merging the PR, it turns out the master CI build failed again with the same Pester package restore error. So I'm reverting the change that got Pester from NuGet cache.
If `-sudo` is specified, make sure to include 'RequireSudoOnUnix' tag on Unix if the 'Tag' is not specified.
If `-sudo` is not specified, make sure to exclude `RequireSudoOnUnix` tag on Unix if the 'ExcludeTag' is not specified.
- add `wixpdb` output when creating `MSI` package
- capture `wixpdb` in official build
- clean up anything left behind from previous MSI builds before starting MSI build to prevent using dirty files.
- make sure MSI creation fails if there is an error
- ignore `.wixpdb` files in git
- Add functionality to `Start-NativeExecution` to
- only display output if there is an error
- log caller information
- WXS validation error fixes
- Remove unused `ExitDialog` to fix ICE82
- Add KeyPath to `SetPath` to fix ICE18
- Use `HKMU` which translates to `HKLM` to runtime to fix various validation errors about creating the shortcut
- Suppress Validation errors
- suppress ICE61, which is about same version upgrades being allowed
- suppress ICE57, caused by the shortcut not being installed per user
Based on standard practices, we need to have a copyright and license notice at the top of each source file. Removed existing copyrights and updated/added copyright notices for .h, .cpp, .cs, .ps1, and .psm1 files.
Updated module manifests for consistency to have Author = "PowerShell" and Company = "Microsoft Corporation". Removed multiple line breaks.
Separate PR coming to update contribution document for new source files: #6140
Manually reviewed each change.
Fix#6073
* Test fixes and changes needed to support Pester 4.0.8
* Replace 'Should Contain' with new 'Should FileContentMatch' assertion
Explicitly check for string creation with write-output
* Use the current version of pester and install it in modulesDir
* Simplify logic for relative path test.
Multiple '..' is not needed for a relative path, a single one will do. Also, on multi-drive systems using split-path -noqualifier will probably do the wrong thing with regard to constructing a correct path.
Remove extraneous Should Not Throw test, if this throws, the test will fail, we don't need to explicitly assert the not throw
* In some environments it is possible that computer name is 'localhost', so that should be allowed
* [feature] Add link for migrating tests from Pester v3 to v4
Fix up capitalization and white space issues
Change one test to check FullyQualifiedErrorId rather than just `Should Throw`
* [feature] update invoke-item test to handle the case where multiple notepad processes are running
* Fix spelling issue with Pester 4x, calling it Pester 4 should be sufficient
Instead of building PSReadLine from this repo, pull it from the gallery using nuget cache.
This pulls v2.0 of PSReadLine which does have documented breaking changes from v1.2, but the risk is small - the features that have changed are typically only used in a profile and aren't used all that often anyway.
Fix#996
Hardcodes version of modules pulled from PSGallery
This is the continuation of PR 5499 that had to be abandoned due to a fatal merge conflict and I did not want to risk accidentally reverting recent fixes.
Remove unnecessary/unused default for productGuid because it always gets a new Guid when being called from Start-PSPackage
Add defaults for required files but also add extra path validation attribute
Rename ProductGuid to ProductCode
Fix bug #5597: x86/x64 installer are uninstalling each other when installing either of them:
-Make x86 installer to be installed as an x86 component (-arch argument to candle.exe, which sets the `sys.BUILDARCH` variable)
-Make the UpgradeCode unique per platform
-Replace `var.ProductTargetArchitecture` variable with sys.BUILDARCH use to have only 1 variable for the architecture
-Additionally, the architecture was appended to the package name to be able to distinguish the installations.
Make PowerShell Core reads group policy settings from different registry keys (Windows only) and the configuration files (both Windows and Unix).
- On Windows, move to different GPO registry keys.
- On both Windows and Unix, read GPO related settings from the configuration file `powershell.config.json`.
- On Windows, the policy settings in registry take precedence over the configuration file.
- Enable policy controlled logging and transcription on Unix.
Update Restore-PSPester to include PR PowerShell/psl-pester#12 that fixesPowerShell/psl-pester#11 - unhandled exceptions in before/after bypasses Pester's enddescribe logic.
Update to the latest, while we are at it.
Before this PR, when a WiX compilation error occurs then an error is thrown, which appears in the log with details but the AppVeyor build itself is still marked as green.
This PR makes the console host also return an exit code of -1 when being run on AppVeyor so that it can then interpret it as a build failure and mark the build as red. It uses the fact that AppVeyor defines an environment variable named CI. Exiting is OK since the MSI build is the last step in CI and nothing happens after that.
The git history shows a test build that proves that this works if the installer was broken.
When changing the WiX installer locally and rebuilding the MSI one can get into the terrible situation whereby a code change does not get compiled into the new MSI.
This PR fixes the root cause of it, which is due to temporary WiX files that are created but not deleted after compilation and WiX then somehow decides to not recompile properly.
I also removed redundant parenthesis of surrounding code.
refactor code to restore pester into a separate function called Restore-PSPester
update message on what to do when pester is missing
Add ability for get-psoptions to default to new-psoptions
fix an issue with publish-pstesttools when a build has not been run since build.psm1 has been imported (try to use the default options)
make start-pspester use the last build, not just use the default options
fix an issue in restore caused some files not to be removed
* enable win-arm and win-arm64 builds
* fix using arm64 tools for build
fix finding vs2017 dynamically
* change install-powershellremoting.ps1 script to use reg provider rather than reg.exe so that error doesn't show on success
fix formatting issue in cmake.defs
* add check that path being read from config file is valid
address PR feedback
* fix copying of PowerShell.Core.Instrumentation as VS2017 puts it in $HOME/source
address PR feedback
* remove pester module
* restore Pester as a module only in CI build from the git repo
* mark appveyor builds as CI builds
* remove pester exclusions
* mark travis builds as ci
* exclude publish folder from spell check
* do not run spell check on publish folder
- Fix build in vscode
- Add v2.0.0 tasks.json thanks to Keith Hill
- Add '/property:GenerateFullPaths=true' to build to fix the $mscompile 'problem matcher' not working issue.
Replace the static `powershell.inc` filename with a name based on the current RID, so that the build doesn't confuse when running in the same local repository on WSL.
* Add two files that need to be signed
* make sure to set PSModuleRestore to true when expanding a signed build because we run PSModuleRestore at that point.
* suppress output of CmdLets which are noisy
- Include a serialized version of PSOptions in an includesymbols zip
- Add a function which will create a zip package from the expanded includesymbols zip and a folder of signed files
- Add a function to restore an includesymbols zip as a build and populated PSOptions with the options
- Add install-powershell.ps1 to install powershell core packages on windows.
- Update Start-PSBootStrap to check and install the latest PSCore package on Windows.
Fix#5260 - with this fix folks shouldn't hit the version mismatch check.
The approach is to see if the dotnet in the current PATH has a compatible SDK. Folks will have a globall installed dotnet if they've installed VS, VSCode C# ext or have installed the .NET Core SDK. This verion may not have the SDK required by PSCore. And the global cannot see user local dotnet SDK installs. So if the global dotnet doesn't have the right SDK, we prepend the path to the user's local dotnet dir.
Also, updateed $dotnetCLIRequiredVersion to read its value from global.json so there is one less source of the truth (for the SDK version).
- Build PowerShell.Core.Instrumentation.dll - Resource-only binary for the ETW resources.
- Create a registration script for registering/unregistering the ETW provider.
pwsh.exe today doesn't contain file version information and the icon is only associated with the shortcut file and not the exe
Fix is to use rcedit to embed:
icon
product version
file version
product name
copyright
Fix#2883Fix#5166Fix#5034
- Rename powershell.exe to pwsh.exe
- Fixe appveyor.psm1
- Update MSI to include 'pwsh' in path and app paths
- Revert change for hyper-v powershell direct
- Update names in packaging.psm1.
- Fix check for SxS
Currently, if a user does not clone with the `--recursive` flag or run `git submodule update --init`, `Start-PSPester` will fail to run due to the missing Pester module. While the error hints at the Pester module not being found, there is no suggested way of fixing the issue presented to the user.
This change makes a helpful warning to appear at the beginning of the execution of `Start-PSPester` if the Pester module cannot be found.
When uploading CodeCoverage artifacts, `Compress-TestContent` is called which calls `Publish-PSTestTools` to build test tools. The build output was captured by `$codeCoverageArtifacts` which causes `Push-AppveyorArtifact` to spit out a lot of errors. The issue is fixed by this change.
The code in `AssemblyLoadContext.dll` doesn't need to be in a separate DLL anymore.
S.M.A.dll depends on `AssemblyLoadContext.dll`, so keeping that code out of S.M.A.dll doesn't help make S.M.A smaller size or less dependent. So the code in `AssemblyLoadContext.dll` is moved to `S.M.A.dll` and then we remove `AssemblyLoadContext.dll`.
The changes are:
- Move `CorePsAssemblyLoadContext.cs` to `src\S.M.A\CoreCLR\`
- Update `CorePsAssemblyLoadContext.cs` to get the test took moved to `Utils.InternalTestHooks` and update tests
- Update `build.psm1` and `.csproj` accrodingly
- Update `pwrshcommon.cpp` to remove `AssemblyLoadContext.dll` from the TPA list.
- `S.M.A.AssemblyExtensions` is removed as `PackageManagement` has finished their move to .NET Core 2.0. (I will work with Bryan to get the latest version uploaded to powershell-core)
