* Make null member access tests as string to avoid parsing errors
* Add PSNullConditionalOperators feature to testmetadata.json
* Add Parsing tests to be skipped
* Add tabcompletion tests
.NET Core changes to return "shortest roundtrippable string" by default for the ToString() method of double and float types. This results in ToString() for double/float values sometimes return a string in 17-digit/9-digit precision format. This PR updated the double/float-to-string conversion in PowerShell to continue using the old precision specifier before the change in .NET Core 3.0.
Expands line continuance for pipelines to allow lines to continue automatically without backticks with the pipe symbol at the start of a line.
This adds to the existing functionality where pipes can be used to continue lines by placing them at the end of a line.
Fixes#7557
* Adds support for binary parsing in format echoing hex: `0b11010110`
* Works with all existing type suffixes and multipliers.
* Supports arbitrary length parsing with `n` suffix using BigInteger; details below.
* Adds `NumberFormat` enum to specify hex/binary/base 10 for the tokenizer, replacing old `bool hex`.
* Adds `n` suffix for all numeric literals to support returning value as a `BigInteger` if requested. This bypasses the issue of large literals losing accuracy when they cast through `double`.
* Adds tests for all new behaviours.
---
### Binary / Hex Parsing Implementation
* Mimics old sign bit behaviour for int and long types. Sign bits accepted for 8 or 16-bit Hex parsing, and 8, 16, 32, 64 for binary.
* i.e., `0xFFFFFFFF -eq ([int]-1)` and `0xFFFFFFFFFFFFFFFF -eq ([long]-1)`, but suffixing `u` creates `int.MaxValue` and `long.MaxValue`, respectively, instead.
* Sign bits higher than this are accepted for bigint-suffixed numerals:
* Hex: Bigint-suffixed hex treats the high bit of any literal with a length multiple of 8 as the sign bit
* Binary: Bigint-suffixed binary accepts sign bits at 96 and 128 chars, and from there on every 8 characters.
* Prefixing the literal with a 0 will bypass this and be treated as unsigned, e.g. `0b011111111`
* Specifying an `u`nsigned suffix (or combination suffix that includes `u`) ignores sign bits, similar to how parsing a hex string using `[Convert]::ToUint32()` would do so.
* Supports negating literals using `-` prefix. This can result in positive numbers due to sign bits being permitted, just like hex literals.
---
### Refactored numeric tokenizer parsing
**New flow:**
1. Check for `real` (`.01`, `0.0`, or `0e0` syntaxes)
1. If the decimal suffix is present, TryParse directly into decimal. If the TryParse fails, TryGetNumberValue returns `false`.
2. TryParse as `Double`, and apply multiplier to value. If the TryParse fails, TryGetNumberValue returns `false`.
1. Check type suffixes and attempt to cast into appropriate type. This will return `false` if the value exceeds the specified type's bounds.
2. Default to parsing as `double` where no suffix has been applied.
2. Check number format.
* If binary, manually parse into BigInteger using optimized helper function to directly construct the BigInteger bytes from the string.
* If hex, TryParse into `BigInteger` using some special casing to retain original behaviours in int/long ranges.
* If neither binary nor hex, TryParse normally as a `BigInteger`.
3. Apply multiplier value before attempting any casts to ensure type bounds can be appropriately checked without overflows.
4. Check type suffixes.
* If a specific type suffix is used, check type bounds and attempt to parse into that type.
* If the value exceeds the type's available values, the parse fails. Otherwise, a straight cast is performed.
5. If no suffix is used, the following types are bounds-checked, in order, resulting in the first successful test determining the type of the number.
* `int`
* `long`
* `decimal` (base-10 literals only)
* `double` (base-10 literals only)
* ~~`BigInteger` for binary or hex literals.~~ If the value is outside `long` range (for hex and binary) or `double` range (for base 10), the parse will fail; higher values must be explicitly requested using the `n`/`N` BigInteger suffix.
---
*This is a breaking change* as binary literals are now read as numbers instead of generic tokens which could potentially have been used as function / cmdlet names or file names.
Notes:
* Binary literal support was approved by the committee in #7557
* ~~The same issue is still under further discussion for underscore support in numeric literals and whether BigInteger parsing ought to be exposed to the user at all.~~
* ~~Supporting underscore literals is a further breaking change causing some generic tokens like `1_000_000` to be read as numerals instead.~~ Per @SteveL-MSFT's [comment](https://github.com/PowerShell/PowerShell/pull/7993#issuecomment-442651543) this proposal was rejected.
* ~~Removing underscore support or preventing standard parsing from accepting BigInteger ranges is a relatively trivial matter. It is my personal opinion that there is no particular reason *not* to hand the user a BigInteger when they enter a sufficiently large literal, but I will defer to the PowerShell Committee's judgement on this.~~
Ubuntu18.04 seems to default to C.UTF-8 for LANG (representing InvariantCulture) which results in a case-sensitive hashtable since CurrentCultureIgnoreCase doesn't work for that culture. Fix is to use OrdinalIgnoreCase instead.
Consolidation of all Windows PowerShell work ported to PSCore6
* Added ps1 file import restriction. Refactored InvokeLanguageModeTestingSupportCmdlet to HelpersSecurity module
* JEA loop back fix. Debugger running commands in CL mode.
* Support for new AMSI codes. Changed to use AMSI buffer API. Unhandled exception fix.
* Fixes for module bugs while running in ConstrainedLanguage mode
* Untrusted input tracking work
* Configuration keyword bug fix, PSRP protocol version check for reconstruct reconnect, Sharing InitialSessionState in runspace pool.
* Restricted remote session in UMCI, Applocker detection collision, Help command exposing functions, Null reference exception fix.
* Added mitigation for debugger function exposure
- Adds y suffix that is used to specify a numeric literal as the sbyte data type.
- Can be combined with the existing u suffix as uy to specify the byte data type.
Measure-Object should handle `ScriptBlock` properties. Fixed by renaming `MshExpression` to `PSPropertyExpression` and making it public. Then in `MeasureObjectCommand`, lifting it up to the parameter level. Previously the implementation exposed the Property as a string and
wrapped it internally as a `PSPropertyExpression`. Now the parameter type is `PSPropertyExpression` directly allowing for both wildcard strings and `ScriptBlock`.
`PSPropertyExpression` now lives in a public namespace where it can be used by cmdlet and script authors to easily add the same type of functionality to their commands. I also modified `PSPropertyExpression` to handle hashtables properly as objects so
@{prop = 3} | measure-object prop
and
@{prop = 3} | measure-object {$_.prop}
will work the same. (Previously the example using just the property name would fail.)
After parsing `if () { }`, new lines are skipped in order to see if either `elseif` or `else` is present. When neither is present, we should resync back to the pointer before skipping those possibly available new lines, so that the new line tokens can be utilized by the subsequent parsing.
Convert ShouldBeErrorId to Should -Throw -ErrorId in PowerShell tests.
Get rid of try { } catch { } formula to assert that errors were thrown.
Small fixes in tests to obey the new Pester -Parameter syntax.
Refactor code to make it easier to maintain and a little faster. Changes are as follows:
1. Support finding a matching signature with variance. But make PowerShell prefer exact match over a match with variance.
2. The metadata signatures in `PSMethod<..>` are generated based on the array of method overloads in `MethodCacheEntry.MethodInformationStructures`, in the exact same order. So in `LanguagePrimitive.ConvertViaParseMethod`, when we try to figure out if there is a match using the metadata signatures in `PSMethod<..>`, we can get the index of the matching signature, and the same index should locate the matching method in `MethodCacheEntry.MethodInformationStructures`. Therefore, we don't need to compare signatures again in the actual conversion method, and instead, we can just leverage the index we found when figuring out the conversion in `ConvertViaParseMethod`.
- This gets rid of the reflection call `GetMethod("Invoke")` and the subsequent signature comparisons in the final conversion method.
- Also, when comparing signatures using `PSMethod<..>` in `ConvertViaParseMethod`, we can just use the generic argument types of each `Func<..>` metadata type, instead of calling `GetMethod("Invoke")` and then `GetParameters()`. This makes the code for comparing signatures simpler (the type `SignatureComparator`).
- Move `MatchesPSMethodProjectedType` from `PSMemberInfo.cs` to the type `SignatureComparator` in `LanguagePrimitives.cs`, as it's closely related to the signature comparison. Also, renamed it to `ProjectedTypeMatchesTargetType`.
- These changes make PSMethod-to-Delegate conversion a little faster, but no big improvement, as the true bottleneck probably is in delegate creation(?). Actually, the performance of this conversion is not critical at all at this moment because this feature should rarely be used in any hot script path. So this exercise is mainly for fun.
3. Remove `PSEnum<T>`. We can directly use enum types when constructing the metadata type `Func<..>`.
4. Remove the code that generates metadata signatures for generic method definitions (call `MakeGenericMethod` with fake types like `GenericType0`, `GenericType1`). This is because:
- We don't support convert generic method to delegate today, so may be better not spending time on preparing the metadata signature types for those methods.
- When the day comes that we need to support it, it's better to use generic argument types directly to construct the `Func<..>` metadata types. I left comments in `GetMethodGroupType` method in `PSMemberInfo.cs` to explain why that approach is better.
