Closes#6944
Following PR #6945, further, reduce the total build time (without any disadvantages) by around 5 minutes by making sure there is a more even split between the 2 build jobs (the 2nd build job used to be much shorter).
Therefore this PR moves also the xUnit and Pester-Admin tests into the 2nd build job. If it is a daily/feature test commit, then the feature tests will also happen (only) in the 2nd build job. Because both jobs now run tests, the failfast option was removed. The final question from my side is whether running tests in 2 build jobs is OK for the daily build, which uploads code coverage results?
The time to wait for the AppVeyor build results is now 15 +/- 2 minutes, which is a huge improvement to what used to be around 28 minutes before the build matrix was introduced.
Fix man page conflict between stable and preview Linux package
- rename preview man page pwsh-preview
Fix /etc/shell entries not being created correctly for preview packages
Update links that contain 'en-us' culture to remove 'en-us' culture (if possible) and in some cases update to newer re-directed link to docs.microsoft.com
Address mac portions of PowerShell/PowerShell-RFC#115 (comment)
Make path when stable /usr/local/microsoft/powershell/6
when not stable (preview) /usr/local/microsoft/powershell/6-preview and symlink name pwsh-preview
allow side-by-side install of stable and preview
Also:
fix an issue where the utime work around for mac requires sudo
refactor some code into common functions
Change the *nix packaging over with the following changes:
Package name (as used by e.g. apt):
Non-preview releases are namedpowershell
Preview releases are named powershell-preview
Installation path:
No longer looks like /opt/microsoft/powershell/6.1.0/ or /opt/microsoft/powershell/6.1.0-preview.1/
Non-previews go to a path like /opt/microsoft/powershell/6/
Previews go to a path like /opt/microsoft/powershell/6-preview/
Path to executable symlink:
Allows SxS with preview
Non-previews linked to /usr/bin/pwsh or /usr/local/bin/pwsh on macOS
Previews linked to /usr/bin/pwsh-preview or /usr/local/bin/pwsh-preview on macOS
Implements PowerShell/PowerShell-RFC#115 (comment)
Related: #6944
Reduce PR build time by 5 minutes by:
Having Packaging as a separate build job in a matrix -> runs in parallel in PR builds because the Microsoft account is a paid account that allows that (at no additional costs)
Not caching the dotnet folder anymore, which is too large and the overhead of zipping/unzipping/upload/download does not pay off (and fails in forked builds that are on a free AppVeyor account due to the size).
Setting the environment variable DOTNET_SKIP_FIRST_TIME_EXPERIENCE to 1 because the initialization of the dotnet CLI cache (1 minute) does not pay off for the whole build.
The total build time of builds on a fork that is on a free AppVeyor account and therefore does not have parallelism, remains the same due to the time saving of redundant caching.
This is just a simple example of what we can easily achieve, we could continue this pattern and split the test runs as per the referenced issue to bring PR builds down to 10 minutes (but this will incur an increase for fork builds on free AppVeyor accounts)
To support PowerShell modules built with .NET Windows Compatibility Pack, we decided that it was best to ship the WCP assemblies with PS Core. This also adds many new .NET APIs be default while only adding ~3.5 MB additional disk footprint (to a ~137 MB install currently).
Also update the build to adopt the official .NET Core 2.1.
The main purpose of this was to enable full symbols for windows release build.
Also makes explicit where we are optimizing and where we are not optimizing due to https://github.com/dotnet/corefx/issues/29700
Ubuntu 17.04 is EOL, so we need to update our packaging for Ubuntu 17.10. This updates the package script and build.psm1 to use 17.10 rather than 17.04.
* Build Update
- Change `TargetFramework` to `netcoreapp2.1` and removed unnecessary `RuntimeFrameworkVersion` from `PowerShell.Common.props`
- Update dotnet SDK to 2.1.300-rc1-008662
- Update `TypeGen` target in `Build.psm1` to work with 2.1
- Rename macOS runtime to `osx-x64` as the old build logic expects 10.12 and breaks running on 10.13 system.
- Remove `PackageReference` to `System.Memory` as it's part of dotnetcore 2.1
- Update search for `crossgen` executable to find the matching version
* Test Update
- Update test tools `WebListener` to latest `asp.net core`
- Marked `AuthHeader Redirect` tests as `Pending` due to change in CoreFX
The comparison to test if we were using sudo was failing when running as root. We would then run -v, and get a bad error message. fixing the comparison.
Update 'Get-ChangeLog' to make it more accurate
Use "||" as the delimiter instead of "|", so it works for commit message that has the pipeline operator in it.
Handle merges that create a merge node in history.
Do not treat powershell team members as external contributors.
Add meta properties to mac VSTS YAML
- Add a property to clean the build machine
- Add a property to set the format of the build name
- add a property to set which queue to run the build in
Since a PR added support to opt out of telemetry via an environment variable, we can remove the,
always intended to be a temporary, solution of deleting a file to opt out of telemetry since the
environment variable can be defined at the system level and exist before even installing PowerShell Core.
Because the variable is defined as opt out, a value of true, yes, or 1 means no telemetry is sent.
- Implementation of PowerShell/PowerShell-RFC#115 (If anything changes in the RFC, we will treat it as a bug, and fix it later)
- Update registry and directory paths to use 6 for the version for stable and 6-preview for a preview release
- Add checkbox to set path
- default checkbox to off for preview builds and on for stable builds
create a PowerShellPackage...ps1 that works with these VMs (vs our custom VM)
create a script to create the \powershell folder with correct permissions
a YML VSTS build definition
Fix#6332
This change renames log and logerror functions to a single Write-Log [$message] [-error] function to avoid conflicting with the log command on MacOS.
Clean build during the daily build to ensure MSI package is generated correctly
- Do another clean build directly before packaging to clean up files that test has added
Fixes#6315
make Linux packages use correct version scheme for preview releases
- Now uses <Major>.<Minor>.<Patch>~<PreviewName> instead of <Major>.<Minor>.<Patch>-<PreviewName> as the - was interpretted as an iteration of the release not a preview.
The script installpsh-debian only works for Ubuntu distributions. This PR adds support for Debian as well.
Related issue: #5700
* Remove support for Ubuntu 17.04 in installpsh-debian.sh
MSI: Make sure that file components are patchable
- avoid changing names and guids of components between builds as this prevents patch generation
- This required submitting the file generated by heat
- add code to make sure the generated file is not out of date
- add `wixpdb` output when creating `MSI` package
- capture `wixpdb` in official build
- clean up anything left behind from previous MSI builds before starting MSI build to prevent using dirty files.
- make sure MSI creation fails if there is an error
- ignore `.wixpdb` files in git
- Add functionality to `Start-NativeExecution` to
- only display output if there is an error
- log caller information
- WXS validation error fixes
- Remove unused `ExitDialog` to fix ICE82
- Add KeyPath to `SetPath` to fix ICE18
- Use `HKMU` which translates to `HKLM` to runtime to fix various validation errors about creating the shortcut
- Suppress Validation errors
- suppress ICE61, which is about same version upgrades being allowed
- suppress ICE57, caused by the shortcut not being installed per user
Remove the functions which generated Nuget packages for Windows.
Add function New-UnifiedNugetPackage to generate nuget packages for each assembly with unix and windows runtimes.
Add function New-NuSpec and New-ReferenceAssembly for creating the required items forNew-UnifiedNugetPackage.
Add a sample for cross platform project with conditional compilation for Linux.
Add function Publish-NugetToMyGet to publish nuget packages to powershell.myget.or
Based on standard practices, we need to have a copyright and license notice at the top of each source file. Removed existing copyrights and updated/added copyright notices for .h, .cpp, .cs, .ps1, and .psm1 files.
Updated module manifests for consistency to have Author = "PowerShell" and Company = "Microsoft Corporation". Removed multiple line breaks.
Separate PR coming to update contribution document for new source files: #6140
Manually reviewed each change.
Fix#6073
PR 6043 broke the installer (issue #6095). To prevent this from happening in the future, add a smoke test that installs the msi in appveyor build and make build fail if installation failed.
It uses the exit code to determine the success. The reason why it does not fail in the current state is because as I pointed out here, the failing custom action is not returning its exit code.
This is the continuation of PR 5499 that had to be abandoned due to a fatal merge conflict and I did not want to risk accidentally reverting recent fixes.
Remove unnecessary/unused default for productGuid because it always gets a new Guid when being called from Start-PSPackage
Add defaults for required files but also add extra path validation attribute
Rename ProductGuid to ProductCode
Make PowerShell Core reads group policy settings from different registry keys (Windows only) and the configuration files (both Windows and Unix).
- On Windows, move to different GPO registry keys.
- On both Windows and Unix, read GPO related settings from the configuration file `powershell.config.json`.
- On Windows, the policy settings in registry take precedence over the configuration file.
- Enable policy controlled logging and transcription on Unix.
Make PowerShell Core reads group policy settings from different registry keys (Windows only) and the configuration files (both Windows and Unix).
- On Windows, move to different GPO registry keys.
- On both Windows and Unix, read GPO related settings from the configuration file `powershell.config.json`.
- On Windows, the policy settings in registry take precedence over the configuration file.
- Enable policy controlled logging and transcription on Unix.
Create the module 'releaseTools.psm1' under 'tools' folder, which expose the function 'Get-ChangeLog' to generate a release change log based on the local commit history.
Create the module 'releaseTools.psm1' under 'tools' folder, which expose the function 'Get-ChangeLog' to generate a release change log based on the local commit history.
Note: changes in tools/appveyor.psm1 are not moved to the 6.0.0 branch because:
- the change is to produce win-arm32/win-arm64 zip packages in daily builds.
Since we don't have daily build for the 6.0.0 branch, the change is not needed.
- there are conflicts when cherry-picking the commits
The Linux release build was broken because AppImage,tar,tar-arm in build.json is turned into a string of "AppImage,tar,tar-arm" by docker run. Now it's changed to use switch parameters.
Create linux-arm tarball package in our release build.
Also moved the Linux-x64 tarball creation to Ubutntu 14.04, because Start-PSBuild adds additional symbolic link files when running on Red Hat Family distros.
Note that linux-arm can only be built on Ubuntu, so the tar-arm package will also be generated from the Ubuntu 14.04.
The Linux release build was broken because AppImage,tar,tar-arm in build.json is turned into a string of "AppImage,tar,tar-arm" by docker run. Now it's changed to use switch parameters.
Create linux-arm tarball package in our release build.
Also moved the Linux-x64 tarball creation to Ubutntu 14.04, because Start-PSBuild adds additional symbolic link files when running on Red Hat Family distros.
Note that linux-arm can only be built on Ubuntu, so the tar-arm package will also be generated from the Ubuntu 14.04.
* remove pester module
* restore Pester as a module only in CI build from the git repo
* mark appveyor builds as CI builds
* remove pester exclusions
* mark travis builds as ci
* exclude publish folder from spell check
* do not run spell check on publish folder
* Disambiguate icon for daily builds on Windows.
Some code had to be borrowed from build.psm1 because this script has to be self contained in case it gets executed by only downloading this file via the published download link https://twitter.com/Steve_MSFT/status/930585082451992576
* Disambiguate icon for daily builds on Windows.
Some code had to be borrowed from build.psm1 because this script has to be self contained in case it gets executed by only downloading this file via the published download link https://twitter.com/Steve_MSFT/status/930585082451992576
* add template xml for package signing
* Add script to generate package signing XML
* remove uploading artifact, it always fails.
* Allow the XML to be updated to be specified
1. Remove `[validate]` attribute from parameter so this script can be invoked directly from the web. This doesn't have any negative impact as the $Destination parameter will have a default value if null or empty.
2. Move removal of destination folder later as installing the package requires package management module and archive module (on Windows)
3. On Windows, because files have open handles when run from existing powershell-daily install, I rename the existing files and copy over the new ones. User needs to exit and restart `pwsh` to take effect (similar to macOS/Linux where you have to exit and restart anyways)
* Add two files that need to be signed
* make sure to set PSModuleRestore to true when expanding a signed build because we run PSModuleRestore at that point.
* suppress output of CmdLets which are noisy
- Include a serialized version of PSOptions in an includesymbols zip
- Add a function which will create a zip package from the expanded includesymbols zip and a folder of signed files
- Add a function to restore an includesymbols zip as a build and populated PSOptions with the options
Enhance 'install-powershell.ps1' to work on Linux and macOS too. A user can install release version or daily powershell core on Linux and macOS.
When `-AddToPath` is specified:
- On Windows, add the absolute destination path to the 'User' scope environment variable 'Path';
- On Linux, make the symlink '/usr/bin/pwsh' points to "$Destination/pwsh";
- On MacOS, make the symlink '/usr/local/bin/pwsh' points to "$Destination/pwsh".
- Support '[package]' tag in PR CI to validate packaging changes in PR CI runs.
When using '[package]' tag, regular CI tests will run and packaging steps will also run.
- Fix nightly build on macOS.
- Add install-powershell.ps1 to install powershell core packages on windows.
- Update Start-PSBootStrap to check and install the latest PSCore package on Windows.
- Build PowerShell.Core.Instrumentation.dll - Resource-only binary for the ETW resources.
- Create a registration script for registering/unregistering the ETW provider.
This PR is divided into the following areas:
1. Add/Rename the PowerShell/Windows ETW manifest to the repo and change both the provider id (guid) and name. See #4939.
The manifest is at tools/resxgen/PowerShell-Core-Instrumentation.man
2. Generate a resx file containing the string resources needed for logging ETW events to syslog.
This is accomplished by tools\resxgen\resxgen.psm1 and resxgen.ps1. The tool generates two files
A resx file containing string resources for each message string from the manifest. This is generated in the System.Management.Automation\gen directory.
A C# class (EventResource) that provides the mapping between the integer event id and the associated string resource name. The file is generated in the System.Management.Automation\CoreCLR directory with a compile-time condition of UNIX
NOTE: The EventResource.cs class generated by resgen is explicitly ignored in the csproj file; it is not used.
3. SMA\utils\tracing\PSSysLogProvider.cs
Implements the abstract LogProvider class and is the syslog equivalent of PSEtwLogProvider. The class contains a number of logical methods for logging lifecycle, health, and normal events.
4. SMA\utils\tracing\SysLogProvider.cs
This is the Syslog equivalent of ETW's log provider class and implements a Log method versus ETW's EventWrite. It is also responsible for resolving event ids to resource names and performing the Syslog call. There is a large comment block in the class XML doc that describes the types of log output it produces.
5. SMA\utils\tracing\PSEtwLog.cs
PowerShell's current implementation is tightly coupled to this class; with code calling it directly for all events. To simplify integration of syslog, I updated the class to create an instance of PSSysLogProvider on Linux and removed the Linux-specific stub file.
6. SMA\engine\PropertyAccessor.cs
This class provides a wrapper around PowerShellProperties.json and has been extended to have Unix-specific assessors for configuring logging. Note that the file is expected to be in the $PSHOME directory to ensure SxS.
Currently, there are four configuration properties:
- LogIdentity - the string identifier for the source application. This defaults to 'powershell' and can be configured to enable distinguishing between side-by-side installations.
- LogLevel - configures the tracing level (log level). Informational is the defauilt.
- LogChannels - used to enable operational and analytic logging. Operational is the default.
- LogKeywords - used to configure enabling tracing by keyword. All keywords other than `UseAlwaysAnalytic` are enabled by default.
NOTE: This will likely change. PowerShell sometimes confuses the analytic channel with this keyword and sends logging to the wrong channel. Once this is cleared up, `UseAlwaysAnalytic` and `UseAlwaysOperational` keywords will likely be removed.
Additional Notes:
1. The current implementation writes directly to syslog and writing to a separate log file is still pending.
2. The generated class and resx are not part of the build; instead, it is expected that Resxgen should be run when events are added to the manifest. To fully automate the process, resxgen will need to be updated to generate the other dependent enums such as 'PSChannel', 'PSEventId', 'PSTask', 'PSOpcode', etc. You will see parsing logic in resxgen.psm1 to prepare for that but it is not enabled at this point.
4. Documentation is pending that documents the format of the syslog output as well as associated configuration.
5. As mentioned at the start, tests are pending
- Update `installpsh-suse.sh` to work with the .tar.gz binary archive.
- Remove `download.sh` as it's not used anywhere now. (checked with PowerShellGet and OneGet, they are using a local copy of download.sh).
small fixes found while doing current work item
-replace expects a regex when I wanted to do a literal replace.
Have Sync-PSTags fix any issues it find
if something goes wrong during the build, we might exit and not update the badge or fire the webhook.
Enable the after_success and after_failure stages of the travis-ci build so we can have a better chance of getting these done.
* Update packaging script to only package PowerShell binaries when creating symbols zip. disallow all other package types for symbols
* remove instrumentation used for debugging during development
* Add error action silently continue to cleanup for reliability.
* Add comment about why folder is being deleted.
- Rename powershell.exe to pwsh.exe
- Fixe appveyor.psm1
- Update MSI to include 'pwsh' in path and app paths
- Revert change for hyper-v powershell direct
- Update names in packaging.psm1.
- Fix check for SxS
Commented out check for sudo as it was failing on Travis-CI
Updated version of Ruby to 2.3.3
Reorder install so Ruby gets installed before installing PS as Homebrew relies on Ruby 2.3
Remove sudo check which breaks Travis-CI
* Add file secret suppression
* Add line secret suppression
* This will be consumed in the VSTS daily build.
* Renamed tests with ConvertTo-SecureString to avoid false positives
* Add a switch to force tests to fail for testing CI system to Start-PSPester
* [includeFailingTest] Add commit tag to force tests to fail for testing CI system
Extract information about the release tag, number of commits since the tag and the hash of the latest commit within a MSBuild target, and bake that information into version properties of the assemblies appropriately.
The updates to build.psm1 and packaging.psm1 create a single package (per bitness), which works on all the Windows OS versions, namely Windows 7 / Windows Server 2008 R2, Windows 8.1 / Windows Server 2012 R2, Windows 10 / Windows Server 2016.
* Add ability to verify a test run using the `-passthru` object from pester
* update travis-ci to use `-passthru` object to verify test results
* Address PR feedback
* call show-pspestererror with a named parameter
* Fix the hang issue in Travis CI build triggered by the [Feature] tag
* [Feature] Update badge only in daily build
* [Feature] Use the term 'daily build'
* Updated to support amazon linux (centos variant)
* Updated to support amazon linux (centos variant)
* Updated to support amazon linux (centos variant)
