Re-add wildcard when searching AST + Missed test case.
## PR Context
In #8109, we removed the line that added a wildcard to the end of the command that was used to match commands in the script AST. This readds that line closer to where it is used.
This PR does 4 things:
* Adds a new cmdlet `New-PSBreakpoint` which creates new `Breakpoint` objects and writes them to the pipeline
* Adds a `-Breakpoint` parameter to `Debug-Runspace` which will receive `Breakpoint` objects
* Makes the constructors for `*Breakpoint` public for use with the API
* Makes `Debugger.GetBreakpoint(string id)` and `Debugger.GetBreakpoints()` public since `SetBreakpoints` is public
Note: `New-PSBreakpoint` and `Set-PSBreakpoint` (which already exists) are similar... but `Set-PSBreakpoint` also sets the breakpoints in the _current_ runspace. This is not ideal if we want to set breakpoints in a _different runspace than the current one_.
## PR Context
The "Attach to process" debugging experience in the PowerShell extension for VSCode is _ok_ but it's not great.
The reason it's not great is due to the `BreakAll` feature of PowerShell debugging which, when you run `Debug-Runspace`, will break at the first piece of code that gets run. This is not ideal when you "Attach to process" _and then_ run your code in the other runspace.
Today, the experience drops you in `PSReadLine`'s psm1 if PSRL is available or in the vscode PowerShell helper psm1.
It's unexpected for the user and not ideal.
This PR will allow the extension to pass in the breakpoints that need to be set initially with `BreakAll` turned off for none of this silly behavior.
### Silly behavior example
If you want a repro, try this:
PowerShell instance 1:
```
Enter-PSHostProcess -Id $otherprocesspid
Debug-Runspace 1
```
PowerShell instance 2:
```
./runfoo.ps1
```
Note that you end up NOT `runfoo.ps1`
Moved check if able to write to $PSHome as way to skip test to `BeforeAll` which already contained a check if running on Windows.
## PR Context
As part https://github.com/PowerShell/PowerShell/pull/9279, tests were updated to be skipped if the test requires writing to `$PSHome` but is not able to. However, these tests already had a skip mechanism in place so the additional check caused the test to run when it should have skipped.
Co-authored-by: Travis Plunk <github@ez13.net>
Fixes#7557
* Adds support for binary parsing in format echoing hex: `0b11010110`
* Works with all existing type suffixes and multipliers.
* Supports arbitrary length parsing with `n` suffix using BigInteger; details below.
* Adds `NumberFormat` enum to specify hex/binary/base 10 for the tokenizer, replacing old `bool hex`.
* Adds `n` suffix for all numeric literals to support returning value as a `BigInteger` if requested. This bypasses the issue of large literals losing accuracy when they cast through `double`.
* Adds tests for all new behaviours.
---
### Binary / Hex Parsing Implementation
* Mimics old sign bit behaviour for int and long types. Sign bits accepted for 8 or 16-bit Hex parsing, and 8, 16, 32, 64 for binary.
* i.e., `0xFFFFFFFF -eq ([int]-1)` and `0xFFFFFFFFFFFFFFFF -eq ([long]-1)`, but suffixing `u` creates `int.MaxValue` and `long.MaxValue`, respectively, instead.
* Sign bits higher than this are accepted for bigint-suffixed numerals:
* Hex: Bigint-suffixed hex treats the high bit of any literal with a length multiple of 8 as the sign bit
* Binary: Bigint-suffixed binary accepts sign bits at 96 and 128 chars, and from there on every 8 characters.
* Prefixing the literal with a 0 will bypass this and be treated as unsigned, e.g. `0b011111111`
* Specifying an `u`nsigned suffix (or combination suffix that includes `u`) ignores sign bits, similar to how parsing a hex string using `[Convert]::ToUint32()` would do so.
* Supports negating literals using `-` prefix. This can result in positive numbers due to sign bits being permitted, just like hex literals.
---
### Refactored numeric tokenizer parsing
**New flow:**
1. Check for `real` (`.01`, `0.0`, or `0e0` syntaxes)
1. If the decimal suffix is present, TryParse directly into decimal. If the TryParse fails, TryGetNumberValue returns `false`.
2. TryParse as `Double`, and apply multiplier to value. If the TryParse fails, TryGetNumberValue returns `false`.
1. Check type suffixes and attempt to cast into appropriate type. This will return `false` if the value exceeds the specified type's bounds.
2. Default to parsing as `double` where no suffix has been applied.
2. Check number format.
* If binary, manually parse into BigInteger using optimized helper function to directly construct the BigInteger bytes from the string.
* If hex, TryParse into `BigInteger` using some special casing to retain original behaviours in int/long ranges.
* If neither binary nor hex, TryParse normally as a `BigInteger`.
3. Apply multiplier value before attempting any casts to ensure type bounds can be appropriately checked without overflows.
4. Check type suffixes.
* If a specific type suffix is used, check type bounds and attempt to parse into that type.
* If the value exceeds the type's available values, the parse fails. Otherwise, a straight cast is performed.
5. If no suffix is used, the following types are bounds-checked, in order, resulting in the first successful test determining the type of the number.
* `int`
* `long`
* `decimal` (base-10 literals only)
* `double` (base-10 literals only)
* ~~`BigInteger` for binary or hex literals.~~ If the value is outside `long` range (for hex and binary) or `double` range (for base 10), the parse will fail; higher values must be explicitly requested using the `n`/`N` BigInteger suffix.
---
*This is a breaking change* as binary literals are now read as numbers instead of generic tokens which could potentially have been used as function / cmdlet names or file names.
Notes:
* Binary literal support was approved by the committee in #7557
* ~~The same issue is still under further discussion for underscore support in numeric literals and whether BigInteger parsing ought to be exposed to the user at all.~~
* ~~Supporting underscore literals is a further breaking change causing some generic tokens like `1_000_000` to be read as numerals instead.~~ Per @SteveL-MSFT's [comment](https://github.com/PowerShell/PowerShell/pull/7993#issuecomment-442651543) this proposal was rejected.
* ~~Removing underscore support or preventing standard parsing from accepting BigInteger ranges is a relatively trivial matter. It is my personal opinion that there is no particular reason *not* to hand the user a BigInteger when they enter a sufficiently large literal, but I will defer to the PowerShell Committee's judgement on this.~~
Add two properties in `ProviderInfo` class: `ItemSeparator` and `AltItemSeparator`.
On windows, the default values for those two properties are `ItemSeparator = '\'` and `AltItemSeparator = '/'`.
On unix, the default values for those two properties are `ItemSeparator = '/'` and `AltItemSeparator = '\'`.
Registry provider is the only exception, both properties for it have the value `\`.
On Unix systems, fallback to plaintext manipulation instead of using the DPAPI which is not available.
## PR Context
Currently, existing scripts that use SecureString cmdlets fail with an error complaining about crypt32.dll not being available. This change allows these cmdlets to be used, but there is no encryption of the string.
.Net already [states](https://docs.microsoft.com/en-us/dotnet/api/system.security.securestring?view=netcore-2.1#remarks) that the contents of a SecureString are not encrypted on .Net Core.
Fix https://github.com/PowerShell/PowerShell/issues/1654
Co-authored-by: Travis Plunk <travis.plunk@microsoft.com>
By not doing excessive amounts of extra work, formatting can be sped up quite significantly (about 8x faster).
The main change comes from adding new, more efficient, primitive to query an object for the existence of an instance member.
The formatting system has been checking for if an object has properties other than some decorated properties added by PS remoting, and it doesn't this by retrieving all properties which results in heavy allocations and wasted cycles.
By adding `GetFirstOrDefault` to `PSObject` and similar primitives to the underlying Adapters, we are able to return early, without having to get all properties back.
Improve type inference for foreach statement variables by:
Inferring strongly typed arrays from explicit array and array literal expressions when elements are of the same inferred type
Fix detection of foreach variable declaration. The previous logic was to check if the variable expression's start offset was after the end offset of the foreach statement, which will never be true in the body
Improve inference of what type the "Condition" of a foreach statement will enumerate as
Update the task-based async APIs added to PowerShell to not use the `aysnc/await` keywords, but to return a `Task` object directly.
There is nothing to continue on after the `Task.Factory.FromAsync` call in those methods, so there is not need to use `aysnc` and `await` keywords, which turns the method into a state machine class unnecessarily.
Here are the major changes:
- Add logic to skip checking for possible resource directories for the first set of sub directories from the top level.
- There was an additional skip if the folder is hidden, rather than doing an explicit attribute check, change the `EnumerationOption` to skip hidden folders.
- Since the `IsPossibleModuleDirectory()` helper now only checks to see if the name matches a culture, renamed to `IsPossibleResourceDirectory()`
- When getting the default modules, we don't search recursively into individual module folders, so removed additional check for possible resource directory.
Adds an optional -Depth parameter to the cmdlet which lets the user to specify a maximum depth allowed for deserialization, which will overwrite the default maximum of 1024.
`TypeBuilder.GetInterfaces()` returns only the interfaces that was explicitly passed to its constructor, so we need to flatten the interface hierarchy in order to properly support inherited interfaces.
This removes some of the last mentions of the AppVeyor name. Don't change the title, we don't want the word to appear in the CLs
Also problem: the last references are in the changelogs but I don't want to modify them without committee approval.
## PR Context
it was removed, #8686
Fix for #6741 Allow .exe files to be used as binary modules. Basically anywhere a .dll could be used with modules, you can now use a .exe file. Also did a little clean up, replacing constant strings with the StringLiteral values instead.
Major changes are:
- Make all commands return 'ConfirmImpact.None' if `SupportsShouldProcess` is not set to `true`.
- Update some cmdlets to explicitly use `ConfirmImpact.Low`.
- Update `DefaultCommands.Tests.ps1` to test for 'ConfirmImpact' level.
Instantiating a new MediaTypeHeaderValue object fails when the -ContentType parameter includes a charset such as application/json; charset=utf-8. This makes it impossible to set the content encoding on web requests. Moving to Parse() ensures we actually get a proper MediaTypeHeaderValue when the charset is present, thus allowing users to set their request encoding via proper -ContentType values.
* Improve check for developer mode by checking minimum required build number
The test would fail if the developer mode is enabled but the machine has an older build than the minimum required build.
The change adds a check for the build version in the test.
* Update test/powershell/Modules/Microsoft.PowerShell.Management/New-Item.Tests.ps1
We have the public API `JsonObject.ConvertFromJson` to convert from JSON string in the PowerShell context. It would be good to have a public API for conversion to JSON. This PR refactors the `ConvertTo-Json` cmdlet to move the core implementation to `JsonObject.ConvertToJson`, and make `ConvertTo-Json` call that public method.
This would help the Azure Function PowerShell worker. Currently, we depends on [calling the cmdlet](729710d259/src/PowerShell/PowerShellManager.cs (L198-L205)) to convert object to JSON which is expensive. Once we have the public method `JsonObject.ConvertToJson` exposed, we can call the API directly to avoid a command invocation.
# Conflicts:
# test/Test.Common.props
We have the public API `JsonObject.ConvertFromJson` to convert from JSON string in the PowerShell context. It would be good to have a public API for conversion to JSON. This PR refactors the `ConvertTo-Json` cmdlet to move the core implementation to `JsonObject.ConvertToJson`, and make `ConvertTo-Json` call that public method.
This would help the Azure Function PowerShell worker. Currently, we depends on [calling the cmdlet](729710d259/src/PowerShell/PowerShellManager.cs (L198-L205)) to convert object to JSON which is expensive. Once we have the public method `JsonObject.ConvertToJson` exposed, we can call the API directly to avoid a command invocation.
Web Cmdlets will no longer forcibly remove Expect: 100-continue from web requests.
This was a legacy setting that needed to be there because of platform differences that have since been resolved in CoreFX.
Fix#8028
This change adds support for specifying the underlying type for an enum:
```powershell
enum MyEnum : long
{
A = 0x0FFFFFFFFFFFFFFF
B
}
# or
enum MyByte : byte
{
A = 0x01
B = 0x02
C = 0x03
D
}
```
Fix broken URL
- Also, move other static analysis tests into that CI.
- Also, make the link analysis a pester test (partly to make sure the step fails in case of an error)
* support fuzzy matching with get-command and on CommandNotFound exception, show suggestion
* fix CodeFactor issues
* change algorithm to use Damerau-Levenshtein Distance which is more useful in finding close matches
Ubuntu18.04 seems to default to C.UTF-8 for LANG (representing InvariantCulture) which results in a case-sensitive hashtable since CurrentCultureIgnoreCase doesn't work for that culture. Fix is to use OrdinalIgnoreCase instead.
WriteMemberInfoCollection() calls WriteEndElement() at most once, even if WriteStartElement() has been called more than once. This commit moves the WriteEndElement() call up immediately after the elements text value has been written.
When using a screen reader or just getting the output of a table with lots of rows, the header is no longer on the screen and the columns may no longer make sense without the context. This change adds a `-RepeatHeader` switch to `Format-Table` to enable re-outputting the header after every screen full (minus 1 row). Expectation is that the user is piping the output to a pager (e.g. less) which uses the bottom row for pager information. I followed the `AutoSize` parameter as the way to get the parameter from the cmdlet into the deep formatting object where it's needed.
## Motivation
I have a PR where there are many new xUnit tests.
It would also be useful to create new xUnit tests for public APIs.
The number of xUnit tests will increase and their ordering is required.
## PR Summary
- Move C# xUnit tests in new folder. This allows to put new xUnit tests in directory structure in accordance with directory structure where cs files are.
- Use an xUnit TestCaseOrderer attribute to sequentially process tests for `powershell.config.json`.
- Update README.md
- A race condition was fixed which allowed to run all XUnit tests in single batch job.
* Disable NewItemUnauthorizedAccessError tests.
* Added expected value for $env:HOMEPATH for user with temporary profile.
* Added a missing null check in CommandHelpProvider.cs
WebListener.exe (which was already being built) is sufficient to start web listener. Remove the dependency on `dotnet` being present to start the web listener.
Use `Start-Process` instead of `Start-Job` to launch the WebListener.
change ellipsis when truncating to single unicode character
reset console output if previous column contains ESC
update existing format-table tests
If content included a VT100 ESC sequence (like changing color), this affected all output after that cell in the table. Fix is to detect that a cell contained ESC and reset the console after it. Also, change the 3 character ellipsis `...` to use the single unicode character `…` so that more text is available.
Fix https://github.com/PowerShell/PowerShell/issues/7767
Major changes are:
- Rename `s_wasSystemPolicyDebugPolicy` to `s_allowDebugOverridePolicy` to make it less confusing. Also slightly refactor `HelperSecurity.psm1` and `ConstrainedLanguageDebugger.Tests.ps1` to remove unneeded code. There is no functional change in this commit.
- Remove the unneeded static property `IsInbox`, as PowerShell Core won't be shipped in-box with Windows in the foreseeable feature. Even if we do in future, we won't be needing it because Windows PowerShell will probably be gone by that time.
- Update 'BindRunspace' to avoid getting all commands and unneeded method calls.
- Avoid creating a `IsSafeValueVisitor` every time when `IsScriptBlockInFactASafeHashtable` runs.
[breaking change]
Major changes are as follows:
- Add `Enable-ExperimentalFeature` and `Disable-ExperimentalFeature` cmdlets.
- Remove `-ListAvailable` from `Get-ExperimentalFeature`.
- Add `ArgumentCompleter` for `Get-ExperimentalFeature` cmdlet.
- Refactor some existing Experimental Feature tests.
- Make `ConfigScope` public and renamed `SystemWide` to `AllUsers`. Also update experimental feature code to prefer the current user config over the all user config.
Fix an intermittent failure in macOS logging tests
- make tests wait for the correct number of log entries
- make tests not fail with an `Index was outside the bounds of the array.` error
Major changes are as follows:
- Avoid `SecuritySupport.IsProductBinary` and unnecessary AMSI/suspicious code scan at startup time
- Update `CompiledScriptBlockData.IsProductCode` to avoid unnecessary calls to `IsProductBinary`, which attempts to retrieve catalog signature of the target file.
- Update `PerformSecurityChecks` to skip AMSI and suspicious code scan for the `.psd1` file that contains a safe `HashtableAst` only.
- Use customized `ReadOnlyBag` instead of `ImmutableHashSet` so that we can avoid loading the `System.Collections.Immutable.dll` completely.
- Replace `SHA1` with `CRC32` when generating module analysis cache file name
- This remove the loading of `System.Security.Cryptography.Algorithms.dll` at startup
- Move `ConvertFrom-SddlString` to C# to remove the `Utility.psm1` file.
- Crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation
- Even pwsh with crossgen assemblies spends a lot time in jitting at the startup, about `191.6ms` comparing with `24.7ms` for Windows PowerShell.
- Jitting `Microsoft.ApplicationInsights.dll` takes about `51.6ms`.
- By crossgen `Microsoft.ApplicationInsights.dll` and enable tiered compilation, the jitting time drops to about `98.9ms`.
Revert "Remove workaround for fixed invalid json array deserializing bug (#8346)"
This reverts commit 60a4e2f346.
This change caused the test which verified the functionality this workaround enabled, to start failing
Export-ModuleMember cmdlet throws an exception if module functions are exported across language boundaries (Windows only). But a scriptblock LanguageMode property can be null if the scriptblock is created without a PowerShell context, and this can happen through the PowerShell API called from C#. In this case Export-ModuleMember throws erroneously when no language mode restrictions are in play.
Fix is to check if LanguageMode is null before comparing context and scriptblock language modes.
With this commit, users who invoke a command with -Debug will no longer be presented with a prompt asking them if they want to enter a nested prompt, continue execution, or halt execution entirely. Instead, any messages sent to the debug stream will simply be sent to the debug stream and the script will continue execution.
When implementing interfaces, PowerShell incorrectly produces non-virtual get/set methods for interface-defined properties.
This commit adds a lookup method for interface-defined properties and marks get/set methods for properties with matching signatures virtual.
There a some differences in support of named pipes for Windows and non-Windows. Named pipes on Unix have a 104 character path limit. On macOS, the `$env:TMPDIR` (on my system) is already 49 characters; corefx adds 12 more. Since AppDomainName isn't really used, changed it from `DefaultAppDomain` to `None` to shorten the name. Need to keep it since Windows PowerShell expects it. Changed `starttime` part of pipe name to 8 hex characters which is to provide uniqueness to the pipe name.
Consolidation of all Windows PowerShell work ported to PSCore6
* Added ps1 file import restriction. Refactored InvokeLanguageModeTestingSupportCmdlet to HelpersSecurity module
* JEA loop back fix. Debugger running commands in CL mode.
* Support for new AMSI codes. Changed to use AMSI buffer API. Unhandled exception fix.
* Fixes for module bugs while running in ConstrainedLanguage mode
* Untrusted input tracking work
* Configuration keyword bug fix, PSRP protocol version check for reconstruct reconnect, Sharing InitialSessionState in runspace pool.
* Restricted remote session in UMCI, Applocker detection collision, Help command exposing functions, Null reference exception fix.
* Added mitigation for debugger function exposure
[Breaking Change]
When a binary module has the module assembly in GAC, we load the assembly from GAC before trying to load it from module base path.
This change attempts to load it from module base path before looking up in GAC.
As of #7892, the PowerShell repository no longer uses Git submodules.
This is fantastic from a workflow standpoint, and so all the notes about
how to deal with submodules (and all the build steps explicitly
initializing and updating submodules) can be safely removed.
[Breaking Change]
There is specific code that sets the `OutputFormat` to xml if pwsh is run non-interactive, with redirected output, and the command was encoded. However, it ignored whether OutputFormat was specified. Fix is to track whether `-OutputFormat` was used and respect that value rather than defaulting to xml.
Fix https://github.com/PowerShell/PowerShell/issues/5912
Add new parameters in Get-Culture cmdlet:
-Name - to allow retrieving a specific culture
-NoUserOverrides - ignore user changes for current culture
-ListAvalable - to allow retrieving all cultures supported on the platform
- Add new Offset and Count parameters.
- Modify ByteCollection class to support offsets up to UInt64.MaxValue size.
- Hide/obsolete Raw parameter because the behavior is by default.
- Optimize conversion to bytes to reduce allocations for large input data.
The root cause is that `OutputBuffer` is not cleaned up (set to `null`) in `Stop` and `BeginStop/EndStop` when the powershell instance owns the `OutputBuffer` object. Since the pipeline has been intentionally stopped by the caller, the `OutputBuffer` should be set to `null` as well when it's owned by the PowerShell instance, just like how it's cleaned up in `EndInvoke`.
- Adds y suffix that is used to specify a numeric literal as the sbyte data type.
- Can be combined with the existing u suffix as uy to specify the byte data type.
The type System.IntPtr has an explicit cast operator defined that takes a pointer type System.void* parameter. Now we handle that type properly in GetPSMethodProjectedType.
Fixes#7230.
Adding the concept of a PSSyntheticTypeName, derived from PSTypeName,
that extends it with a list of synthetic members.
This allows us to express the inferred type of
```
[pscustomobject] @{
A = 1
B = "2"
}
```
as a "PSObject#A:B" and with information about the types of A and B.
This is also used to annotate the output of
```
Select-Object -Property
Select-Object -ExcludeProperty
Select-Object -ExpandProperty
```
Finally, it adds information about the types of the
`Group` and `Value` properties of the output of `Group-Object`
'cd +' adds the ability to revert 'cd -' by having another bounded stack with the same limit for 'redo' actions.
Behaviour is similar to back/forward navigation in explorer.exe
When the location is set to a path and the redo stack is non-empty then the redo stack gets flushed.
When cd + happens, then the redo stack is pushed to as well for a more intuitive way of navigating backwards and forwards.
All the work is done of course on the Set-location cmdlet, which is aliased to cd on all platforms.
Support calling methods with ByRef-like type parameters in PowerShell, as long as the argument specified for the parameter can be implicitly/explicitly converted to the ByRef-like type.
We cannot create an instance of a ByRef-like type in PowerShell, but there are types that can be implicitly or explicitly converted to ByRef-like types, such as `T[] -> Span<T>`, `T[] -> ReadOnlySpan<T>`, `String -> ReadOnlySpan<T>`. `ArraySegment<T> -> Span<T>`, `ArraySegment<T> -> ReadOnlySpan<T>`. With changes in this PR, we can call methods with ByRef-like parameter types by providing the arguments of the types that can be cast to the target ByRef-like type.
**What is enabled?**
1. Invoking methods that have ByRef-like parameters, but the return type is not ByRef-like.
2. Invoking constructors with ByRef-like parameters via `[target-type]::new` syntax, but the `target-type` is not ByRef-like.
3. Accessing indexers that have ByRef-like parameters, but the return type is not ByRef-like.
This change updates ModuleIntrinsics.GetModulePath to handle the case where the Windows PowerShell module path is already in the environment's PSModulePath or when launched from a different version of PowerShell.
Previously, GetModulePath would append $PSHOME\Modules to the PSModulePath after removing the path for the launching version without considering the Windows PowerShell module path. The result, was the Windows PowerShell modules were found first and loaded incompatible modules; such as the built-in modules.
The change detects the Windows PowerShell module path and inserts $PSHOME\Modules path before it. The new test simulates launching from a different version of pwsh that has already added the Windows PowerShell module path.
Fixes#7679
Add daily build non-windows platforms
- Also, make the [Feature] tag work in VSTS for non-windows
- Also, add a way to force feature tests to run
- Also, fix an issue where `-workingdirectory` didn't work when running async