Refactor the `ci.psm1` to publish the code coverage and test packages.
Allow `CodeCoverage` configuration on non-windows.
## PR Context
We plan to run code coverage on Windows, Linux and macOS. These changes are needed to unblock those runs.
Add support in packaging.psm1 to produce a .msix AppX package. Update the docker image to use the new msix package type. Update the associated yml files so AzDevOps performs the build.
## PR Context
Enable publishing PSCore6 to Microsoft Store
Fixes#8919
Preserve user shortcuts pinned to Taskbar during MSI upgrade by not removing shortcuts in this case (assuming the user has not changed the installation directory), see https://stackoverflow.com/a/33402698/1810304
This also requires the Guid to not always be re-generated, which PR #7701 originally added to ensure shortcuts get removed when RTM and preview are installed, the underlying problem was rather that RTM and preview shared the same GUIDs, therefore the GUIDs are hard-coded again but different for RTM and preview, therefore the shortcuts will still always get removed on uninstall. But this also means those GUIDs should change when the default installation directory changes, i.e. in PowerShell 7. Should we write the code to already take this into account that it does not get forgotten?
Tested by first reproducing the issue by building installers locally (and bumping the patch version. Then the fix was applied to verify the solution, it. For this to take effect the version from which an MSI is being upgraded must have this fix already, i.e. if this fix got shipped in `6.2.1`, then on upgrading to it, the issue would still occur but when upgrading `6.2.1` to `6.2.2` the shortcut would start being preserved. I am wondering if we could maybe improve this to show effect earlier by trying to extract the used (auto-generated) GUIDs in the `6.2.0` and `6.2.0-rc` packages out and use them...
Please not that we probably need to take this out for `7.0` because the base installation directory will change. This also assumes that the user has not specified a different installation directory on upgrade but this is a bit of an edge case where I think other things might break as well.
Moved check if able to write to $PSHome as way to skip test to `BeforeAll` which already contained a check if running on Windows.
## PR Context
As part https://github.com/PowerShell/PowerShell/pull/9279, tests were updated to be skipped if the test requires writing to `$PSHome` but is not able to. However, these tests already had a skip mechanism in place so the additional check caused the test to run when it should have skipped.
Co-authored-by: Travis Plunk <github@ez13.net>
This PR optimizes the download tasks to reduce build times. After the change we see a order of magnitude reduction in download times for Windows package signing and also the compliance tasks.
In addition to that the PR also makes changes to organize the build artifacts in folders.
Removes appveyor stuff
## PR Context
<!-- Provide a little reasoning as to why this Pull Request helps and why you have opened it. -->
#8686 , appveyor removed
The `f` in Dockerfile is not uppercase, this breaks syntax highlighting for vscode on my client
<!-- Summarize your PR between here and the checklist. -->
## PR Context
`DockerFile` breaks syntax highlighting as mentioned and described above.
<!-- Provide a little reasoning as to why this Pull Request helps and why you have opened it. -->
The NuGet build task should wait on uploads for DEB job to complete.
## PR Context
There is a race condition when the build for DEB is completed but the upload has not. If the NuGet build job starts it may not get all the packages needed for building nuget packages.
Refactor macOS CI so that tests run in parallel
- Also:
- Fix bootstrap issue where curl was failing for a month (so obviously not needed)
- Change Windows test stages to use PowerShell Core (where we can) to avoid compat issues
## PR Context
This allows running all tests in about the same about of clock time.
I noticed a couple of inconsistencies when reading through the install bash scripts.
- Make documentation for switches consistent over all files.
- Replace all `sed` implementations of `lowercase` with a more maintainable `tr` implementation.
- Set the `OS` variable in every install script, making it so previously unused checks are actually used.
- Exit with a non-zero exit code when the script reaches an illegal state.
## PR Context
A lot of people, including myself, read the install scripts before executing them. While doing so I noticed they contained inconsistencies. For example, certain flags you can pass to the install script were either undocumented or had an incorrect description. This PR fixes some of these inconsistencies, which should make them easier to maintain and easier to read.
Co-authored-by: Travis Plunk <github@ez13.net>
* Add shim dll to redirect to appropriate runtime
* Build updates for creating global tool package
* Add building global tool to nuget.yml
* Add azure upload of nuget packages
Make test file result names unique
- Also, remove dead code that I found during investigation
Fixes#8978
## PR Context
Making the filenames unique, ensures that we can download the results from the artifacts and do our own processing of the results.
This removes some of the last mentions of the AppVeyor name. Don't change the title, we don't want the word to appear in the CLs
Also problem: the last references are in the changelogs but I don't want to modify them without committee approval.
## PR Context
it was removed, #8686
Split Windows CI into the following Jobs
1. Combine Build and xUnit, which published build artifact
1. Elevated CI tests
1. Unelevated CI test
1. Elevated everything else tests
1. Unelevated everything else tests
1. Package build and test
1. Verify xUnit result
## PR Context
Before this test:
- A CI run would take ~14 minutes
- A Feature run would take ~30 minutes
After this change, it always runs all tests and takes ~20 minutes.
Job `Unelevated everything else tests` is the long poll, taking ~4 minutes longer than the next shortest job. We should be able to get this back down to ~15 if we split that test job somehow.
Remove references to uninitialized variable
regressed in #8822
## PR Context
Some artifacts never got published because the conditions would never be true
Refactors `travis.ps1`. Important for maintainers: I removed the webhook and badge stuff as neither exist anymore. I also merged the build and test phase with the bootstrap phase.
Added feature to classify experimental features in change log. Also fixed an issue with reporting all PRs with missing tags together.
## PR Context
Updates to the script which generates the change log.
Add binskim to coordinated build and increase timout
## PR Context
Total timeout for the old build was 220 minutes.
The portions before the compliance take ~30 minutes.
So, I went with 180.
I also found I missed binskim when doing this work
Refactor Unified Release Build
- Make injecting the nuget.config a template
- Make setting the version variable a template
- Add parameters to templates where needed
- Run defender scan before uploading linux packages (and rename step to plain english)
Add compliance to Coordinated build
- Also switch to mac internal pool for release build
- Also turn some duplicate tasks into templates
- Also fix issue with vscode configuration which causes yaml files not to be recogized as yaml
Update the universal build to also build the framework package needed for the dotnet sdk container image.
## PR Context
We build the package in individual builds. This change brings over the steps in the universal build.
Update `New-ReferenceAssembly` and `New-UnifiedNugetPackage` to generate reference assembly for `Microsoft.PowerShell.Commands.Utility.dll` and properly deploy it for `Microsoft.PowerShell.Commands.Utility` NuGet package and `Microsoft.PowerShell.SDK` NuGet package.
An incremental step to fix, eventually, #8121
Update `New-ReferenceAssembly` and `New-UnifiedNugetPackage` to generate reference assembly for `Microsoft.PowerShell.Commands.Utility.dll` and properly deploy it for `Microsoft.PowerShell.Commands.Utility` NuGet package and `Microsoft.PowerShell.SDK` NuGet package.
An incremental step to fix, eventually, #8121
This PR is a re-implementation of the -AddToPath switch in tools/install-powershell.ps1, without the side-effects the current implementation has. The changes only affects windows users.
Add script to generate macOS icon file. It does not attempt to integrate with the current build process as the macOS icon file is mostly static.
`libsvg` was chosen because of its low dependency count over other solutions that use backends like Chromium, PhontomJS, etc. ImageMagick's `convert` was tested but resulting PNGs where not satisfactory and would have required an overcomplicated script having to calculate the `density` argument.
* Has a dependency on librsvg (can be installed via Brew)
* Uses SVG file as input
* Can be integrated in later to build process if desired
Closes#7455
When installing (or upgrading) Preview builds the component ids for the files are the same as the installed files from the stable MSI. MSI sees this and assumes the file is already installed so skips installing files that haven't changed (been updated) which results in missing files in the preview install folder and pwsh fails to start. Fix is to dynamically generate new component ids (and compoentrefs) in `files.wxs` using `_Preview` suffix and use that when building a preview package.
Tested manually.
cc @bergmeister if you can help validate different scenarios.
Fix https://github.com/PowerShell/PowerShell/issues/8289
## Motivation
I have a PR where there are many new xUnit tests.
It would also be useful to create new xUnit tests for public APIs.
The number of xUnit tests will increase and their ordering is required.
## PR Summary
- Move C# xUnit tests in new folder. This allows to put new xUnit tests in directory structure in accordance with directory structure where cs files are.
- Use an xUnit TestCaseOrderer attribute to sequentially process tests for `powershell.config.json`.
- Update README.md
- A race condition was fixed which allowed to run all XUnit tests in single batch job.
Create unified release build for macOS and Linux packages
- Also, updated the definition with latest definitions from the definitions in use.
I will work on these in future PRs:
- Add compliance Job
- Add macOS Signing step
- Merging Windows into the YAML
Major changes are as follows:
- Avoid `SecuritySupport.IsProductBinary` and unnecessary AMSI/suspicious code scan at startup time
- Update `CompiledScriptBlockData.IsProductCode` to avoid unnecessary calls to `IsProductBinary`, which attempts to retrieve catalog signature of the target file.
- Update `PerformSecurityChecks` to skip AMSI and suspicious code scan for the `.psd1` file that contains a safe `HashtableAst` only.
- Use customized `ReadOnlyBag` instead of `ImmutableHashSet` so that we can avoid loading the `System.Collections.Immutable.dll` completely.
- Replace `SHA1` with `CRC32` when generating module analysis cache file name
- This remove the loading of `System.Security.Cryptography.Algorithms.dll` at startup
- Move `ConvertFrom-SddlString` to C# to remove the `Utility.psm1` file.
- Crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation
- Even pwsh with crossgen assemblies spends a lot time in jitting at the startup, about `191.6ms` comparing with `24.7ms` for Windows PowerShell.
- Jitting `Microsoft.ApplicationInsights.dll` takes about `51.6ms`.
- By crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation, the jitting time drops to about `98.9ms`.
Compare the PowerShell modules dependencies on PowerShell Gallery and sync them to AzDevOps artifacts feed if a newer is available. The release builds pick up modules from AzDevOps feed.
Remove tilde from Linux preview packages.
- Since we have a separate package for preview, we don't need a second indicator that these packages are preview.
- Also, many system we use to release don't like the `~` character, which leads to manual interventions and errors.
As of #7892, the PowerShell repository no longer uses Git submodules.
This is fantastic from a workflow standpoint, and so all the notes about
how to deal with submodules (and all the build steps explicitly
initializing and updating submodules) can be safely removed.
The variable was set to empty (meaning to delete the variable) in the non-preview case and the build fails.
The fix avoids setting the variable to empty
Add the functionality to build a framework dependent (shared framework) package for PowerShell.
The changes create two packages, one for Windows and other for Linux, due to #if code.
Add daily build non-windows platforms
- Also, make the [Feature] tag work in VSTS for non-windows
- Also, add a way to force feature tests to run
- Also, fix an issue where `-workingdirectory` didn't work when running async
On Debian 9, libmi cannot resolve libssl and libcrypto. This change adds symbolic links to packaging and build to the $PSHOME directory to resolve the issue.
The fix was verified interactively connecting to office 365, importing the session, and ensuring Get-MailBox and Get-User succeed.
Fix#7598
- Add VSTS CI for Windows
- Disable `Access-denied test for Get-Item C:\windows\appcompat\Programs\Install -ErrorAction Stop`, because the path does not always exist
- https://github.com/PowerShell/PowerShell/issues/7553
- Disable `Should give .sys file if the fullpath is specified with hidden and force parameter`, because pagefile.sys doesn't always exist and other files don't meet test's requirement.
- https://github.com/PowerShell/PowerShell/issues/7554
- Disable some `Test-Connection` tests for same reasons they failed on VSTS Linux
- https://github.com/PowerShell/PowerShell/issues/7555
- Disable `Test-FileCatalog should pass when catalog is in the same folder as files being tested`, because the CmdLet does not work in that scenario
- Also, give details needed to investigate when the test fails
- https://github.com/PowerShell/PowerShell/issues/7556
- Update `appveyor.psm1` to work with VSTS
- Update `HelpersRemoting.psm1` `New-RemoteSession` to work for CimSession (discovered an issue during the investigation)
- Update `Test wildcard with drive relative directory path` to work when there are multiple drives
- Disable on non-windows machines since the test is assuming drive letters
- Update `New-CimSession` Tests to requireAdmin
- Also, make sure session name is a string
- Add functions to save and restore psoptions
- update `.gitatttributes` so files clone like they do on appveyor
Update Start-PSPester and Start-PSxUnit to upload test results when running in VSTS
Disable tests which require non-root using when running as root
remove verifications in Test-Connection tests which verify buggy behavior
Skip tests which don't behave correctly in VSTS (due to running in a container?)
- Test-Connection - #7528
- Set-Date - #7529
Fixes#7074. This PR allows concurrent versions of the macOS launcher app to exist. The application name will be either PowerShell or PowerShell-preview. It is now dynamically built and removed post fpm to avoid post build conflicts, see #5262 .
* update to latest package references
* update runtime framework
* update sdk
* automatically read NuGet package dependency info from csproj, where version info is fully qualified
* update file.wxs
-preview switch:
- installs pwsh-preview packages when a package manager is being used.
- uses link name pwsh-preview when tar ball install is being done
- results in preview release being available for calling manually, but does not become systemwide version
Issue #7177
setdate.cpp was incorrectly assigning an int to a struct, fixed to initialize struct zero'd
update the typegen project in the bash script
added some help text at top of script for someone trying to build it
updated runtime to linux-musl-x64 instead of linux-x64 as Alpine distro uses the musl libc libraries instead of glibc which is used by most distros.
Fixed#6815 by adding -allowprereleases parameter
Fixed#6405 by adding -allowprereleases parameter
Added parameters to documentation
Ready for prerelease repositories if Microsoft starts providing them
Added -skip-sudo-check for all distros
Fixed -interactivetesting should do nothing if -includeide was not used
update the Windows release build script with the ability to capture all the project.assets.json files
add a Windows build to use the new feature in the release build script
update the Linux release build script with the ability to capture all the project.assets.json files
-no new build is needed because signing is not currently automated
Closes#6944
Following PR #6945, further, reduce the total build time (without any disadvantages) by around 5 minutes by making sure there is a more even split between the 2 build jobs (the 2nd build job used to be much shorter).
Therefore this PR moves also the xUnit and Pester-Admin tests into the 2nd build job. If it is a daily/feature test commit, then the feature tests will also happen (only) in the 2nd build job. Because both jobs now run tests, the failfast option was removed. The final question from my side is whether running tests in 2 build jobs is OK for the daily build, which uploads code coverage results?
The time to wait for the AppVeyor build results is now 15 +/- 2 minutes, which is a huge improvement to what used to be around 28 minutes before the build matrix was introduced.
Fix man page conflict between stable and preview Linux package
- rename preview man page pwsh-preview
Fix /etc/shell entries not being created correctly for preview packages
Update links that contain 'en-us' culture to remove 'en-us' culture (if possible) and in some cases update to newer re-directed link to docs.microsoft.com
Address mac portions of PowerShell/PowerShell-RFC#115 (comment)
Make path when stable /usr/local/microsoft/powershell/6
when not stable (preview) /usr/local/microsoft/powershell/6-preview and symlink name pwsh-preview
allow side-by-side install of stable and preview
Also:
fix an issue where the utime work around for mac requires sudo
refactor some code into common functions
Change the *nix packaging over with the following changes:
Package name (as used by e.g. apt):
Non-preview releases are namedpowershell
Preview releases are named powershell-preview
Installation path:
No longer looks like /opt/microsoft/powershell/6.1.0/ or /opt/microsoft/powershell/6.1.0-preview.1/
Non-previews go to a path like /opt/microsoft/powershell/6/
Previews go to a path like /opt/microsoft/powershell/6-preview/
Path to executable symlink:
Allows SxS with preview
Non-previews linked to /usr/bin/pwsh or /usr/local/bin/pwsh on macOS
Previews linked to /usr/bin/pwsh-preview or /usr/local/bin/pwsh-preview on macOS
Implements PowerShell/PowerShell-RFC#115 (comment)
Related: #6944
Reduce PR build time by 5 minutes by:
Having Packaging as a separate build job in a matrix -> runs in parallel in PR builds because the Microsoft account is a paid account that allows that (at no additional costs)
Not caching the dotnet folder anymore, which is too large and the overhead of zipping/unzipping/upload/download does not pay off (and fails in forked builds that are on a free AppVeyor account due to the size).
Setting the environment variable DOTNET_SKIP_FIRST_TIME_EXPERIENCE to 1 because the initialization of the dotnet CLI cache (1 minute) does not pay off for the whole build.
The total build time of builds on a fork that is on a free AppVeyor account and therefore does not have parallelism, remains the same due to the time saving of redundant caching.
This is just a simple example of what we can easily achieve, we could continue this pattern and split the test runs as per the referenced issue to bring PR builds down to 10 minutes (but this will incur an increase for fork builds on free AppVeyor accounts)
To support PowerShell modules built with .NET Windows Compatibility Pack, we decided that it was best to ship the WCP assemblies with PS Core. This also adds many new .NET APIs be default while only adding ~3.5 MB additional disk footprint (to a ~137 MB install currently).
Also update the build to adopt the official .NET Core 2.1.