maxmustermann/PowerShell
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Merge pull request #1547 from PowerShell/jameswtruher/TestGuidelineUpdate

Browse source 
updates from test-guideline PR to incorporate feedback
This commit is contained in:
James Truher [MSFT] 2016-08-02 14:49:48 -07:00 committed by GitHub
parent b654b7d4b1 935185cf6b
commit cf3519b8eb
6 changed files with 197 additions and 154 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
docs/testing-guidelines/Images/AppVeyor-Badge-Green.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 2.3 KiB

BIN
docs/testing-guidelines/Images/AppVeyor-Github.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 14 KiB

41
docs/testing-guidelines/PesterDoAndDont.md Normal file → Executable file
View file

@ -1,23 +1,34 @@
## Do
1. Name your files <description>.tests.ps1
1. Name your files <descriptivetest>.tests.ps1
2. Keep tests simple
1. Test only what you need
2. Reduce dependencies
3. Be sure to tag your Describe blocks with "inner" and "outer"
4. Make sure that Describe/Context/It descriptions are useful
3. Be sure to tag your `Describe` blocks based on their purpose
1. Tag `CI` indicates that it will be run as part of the continuous integration process. These should be unit test like, and generally take less than a second.
2. Tag `Feature` indicates a higher level feature test (we will run these on a regular basis), for example, tests which go to remote resources, or test broader functionality
3. Tag `Scenario` indicates tests of integration with other features (these will be run on a less regular basis and test even broader functionality than feature tests.
4. Make sure that `Describe`/`Context`/`It` descriptions are useful
1. The error message should not be the place where you describe the test
5. Use "Context" to group tests
1. Multiple Contexts can help you group your test suite into logical sections
6. Use BeforeAll/AfterAll/BeforeEach/AfterEach instead of custom initiators
7. Use Try-Catch for expected errors and check $_.fullyQualifiedErrorId
8. Loop It blocks for checking multiple properties
5. Use `Context` to group tests
1. Multiple `Context` blocks can help you group your test suite into logical sections
6. Use `BeforeAll`/`AfterAll`/`BeforeEach`/`AfterEach` instead of custom initiators
7. Prefer Try-Catch for expected errors and check $_.fullyQualifiedErrorId (don't use `should throw`)
8. Use `-testcases` when iterating over multiple `It` blocks
9. Use code coverage functionality where appropriate
10. Use Mock functionality when you don't have your entire environment
11. Avoid free code in a Describe block
1. Use `[Before|After][Each|All]` _see Free Code in a Describe block_
10. Use `Mock` functionality when you don't have your entire environment
11. Avoid free code in a `Describe` block
1. Use `[Before|After][Each|All]` see [Free Code in a Describe block](WritingPesterTests.md#free-code-in-a-describe-block)
12. Avoid creating or using test files outside of TESTDRIVE:
1. TESTDRIVE: has automatic clean-up
13. Keep in mind that we are creating cross platform tests
1. Avoid using the registry
2. Avoid using COM
14. Avoid being too specific about the _count_ of a resource as these can change platform to platform
1. ex: checking for the count of loaded format files, check rather for format data for a specific type
## Don't
1. Have too many evaluations in a single It block
1. The first "Should" failure will stop that block
2. Don't use "Should" anywhere but within an "It" Block
1. Don't have too many evaluations in a single It block
1. The first `Should` failure will stop that block
2. Don't use `Should` outside of an `It` Block
3. Don't use the word "Error" or "Fail" to test a positive case
1. ex: "Get-Childitem TESTDRIVE: shouldn't fail", rather "Get-ChildItem should be able to retrieve file listing from TESTDRIVE"

106
docs/testing-guidelines/Testing.md
View file

@ -1,106 +0,0 @@
# DRAFT
_I have more questions than answers_
#### Current Test Infrastructure
We currently rely heavily on STEX environment for our testing, and we will continue to do so through the Server2016 release. We
need to use that current infrastructure to continue to test full PowerShell builds; it should be possible to build automation
which takes a full PowerShell build and lay it on an existing lab system, update the appropriate test files and execute
a test pass in the same way that we do with official builds.
The test artifacts which are applicable to full PowerShell are not universally applicable to Core/Nano/OtherPlatform, we will need
to create tooling which allows us to apply a set of test artifacts to a configuration, and then execute tests. Eventually, we need
to have our CI environment test all the flavors of PowerShell we create.
**Question**: Can AppVeyor/Travis service that need?
#### Organization
**Proposal**: Create 3 tiers of testing:
* Checkin
* These are run as part of the CI process, and should run quickly. How quickly is an open question. We need to determine
the right amount of coverage without spending too much time. It may be that we can improve our coverage here through parallelization
but we have not investigated enough to determine whether it's possible.
* Feature
* the tests which look at corner cases, and stand-alone modules (for example, the archive module tests could fall into this
category)
* Scenario
* these are tests which span features, and determine whether the whole product is working correctly. The current P3 tests fall
largely here
**Actions**: Decide what goes where. My initial thoughts are to migrate our current TTEST unittests into tier 1 (Checkin)
**Current Migration Activity**
We have teams working on migrating tests which are in non-portable frameworks (TTest, Lite1, Lite3, etc) to portable frameworks.
The first effort is to migrate our TTEST cmdlet unit tests to Pester, we should be taking those migrated tests and get them into
SD
##### Layout
We need to have a reasonable layout of our tests, not sure what that looks like yet. We need to make it
easy to find both feature and test code to reduce our maintainance burden.
##### Self Hosting
Self-Hosting remains problematic while are still so early in the development phase, but it is _imperative_
that we dog food as early as possible. This is especially true on the non-Windows platforms where we have made
assumptions about the working environment with regard to a number of issues:
* removal of well known aliases
* case sensitivity of some operations
* coverage
We should be using these non-windows platforms as much as possible to
=======
# DRAFT
_I have more questions than answers_
#### Current Test Infrastructure
We currently rely heavily on STEX environment for our testing, and we will continue to do so through the Server2016 release. We
need to use that current infrastructure to continue to test full PowerShell builds; it should be possible to build automation
which takes a full PowerShell build and lay it on an existing lab system, update the appropriate test files and execute
a test pass in the same way that we do with official builds.
The test artifacts which are applicable to full PowerShell are not universally applicable to Core/Nano/OtherPlatform, we will need
to create tooling which allows us to apply a set of test artifacts to a configuration, and then execute tests. Eventually, we need
to have our CI environment test all the flavors of PowerShell we create.
**Question**: Can AppVeyor/Travis service that need?
#### Organization
**Proposal**: Create 3 tiers of testing:
* Checkin
* These are run as part of the CI process, and should run quickly. How quickly is an open question. We need to determine
the right amount of coverage without spending too much time. It may be that we can improve our coverage here through parallelization
but we have not investigated enough to determine whether it's possible.
* Feature
* the tests which look at corner cases, and stand-alone modules (for example, the archive module tests could fall into this
category)
* Scenario
* these are tests which span features, and determine whether the whole product is working correctly. The current P3 tests fall
largely here
**Actions**: Decide what goes where. My initial thoughts are to migrate our current TTEST unittests into tier 1 (Checkin)
**Current Migration Activity**
We have teams working on migrating tests which are in non-portable frameworks (TTest, Lite1, Lite3, etc) to portable frameworks.
The first effort is to migrate our TTEST cmdlet unit tests to Pester, we should be taking those migrated tests and get them into
SD
##### Layout
We need to have a reasonable layout of our tests, not sure what that looks like yet. We need to make it
easy to find both feature and test code to reduce our maintainance burden.
##### Self Hosting
Self-Hosting remains problematic while are still so early in the development phase, but it is _imperative_
that we dog food as early as possible. This is especially true on the non-Windows platforms where we have made
assumptions about the working environment with regard to a number of issues:
* removal of well known aliases
* case sensitivity of some operations
* coverage
We should be using these non-windows platforms as much as possible to
>>>>>>> Create Testing.md

66
docs/testing-guidelines/WritingPesterTests.md Normal file → Executable file
View file

@ -1,9 +1,13 @@
Because we are planning to continue to extend Pester to support remote/parallel execution, it's important to keep in mind the following when create tests:
### Writing Pester Tests
Note that this document does not replace the documents found in the [Pester](https://github.com/pester/pester "Pester") project. This is just
some quick tips and suggestions for creating Pester tests for this project. The Pester community is vibrant and active, if you have questions
about Pester or creating tests, the [Pester Wiki](https://github.com/pester/pester/wiki) has a lot of great information.
When creating tests, keep the following in mind:
* Tests should not be overly complicated and test too many things
* boil down your tests to their essence, test only what you need
* Tests should be as simple as they can
* Tests should generally not rely on any other test
Examples:
Here's the simplest of tests
@ -27,14 +31,28 @@ Describe "One is really one" {
}
It "1 is really an int" {
$i = 1
$i.GetType().FullName | Should Be System.Int32
$i.GetType() | Should Be "int"
}
}
```
If you are checking for proper errors, do that in a try catch, and then check fully qualified error id
alternatively, you could do the following:
```
Describe "One is really one" {
It "Compare 1 to 1" {
$a = 1
$a | Should be 1
}
It "1 is really an int" {
$i = 1
$i.GetType() | Should Be ([System.Int32])
}
}
```
If you are checking for proper errors, do that in a `try/catch`, and then check `FullyQualifiedErrorId`. Checking against `FullyQualifiedErrorId` is recommended because it does not change based on culture as an error message might.
```
...
it "Error should be PathNotFound" {
it "Get-Item on a nonexisting file should have error PathNotFound" {
try
{
get-item "ThisFileCannotPossiblyExist" -ErrorAction Stop
@ -42,8 +60,7 @@ it "Error should be PathNotFound" {
}
catch
{
$_.FullyQualifiedErrorId |
should be "PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand"
$_.FullyQualifiedErrorId | should be "PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand"
}
}
```
@ -51,12 +68,10 @@ it "Error should be PathNotFound" {
Note that if get-item were to succeed, a different FullyQualifiedErrorId would be thrown and the test will fail because the FQErrorId is wrong. This is the suggested path because Pester wants to check the error message, which will likely not work here because of localized builds, but the FullyQualifiedErrorId is constant regardless of the locale.
### Describe/Context/It
From an organizational standpoint, a Describe block roughly compares to a LITE3 Test Suite and an It block roughly compares to a LITE3 testcase. Unlike LITE3, individual tests are not prioritized (remember that a test name in Pester is really a descriptive sentence, rather than a name), but a Describe block (TestSuite) may be tagged with a string.
For creation of PowerShell tests, the Describe block is the level of granularity suggested and one of two tags should be used: "Inner" or "Outer". If the tag is not provided, tests in that describe block will be run any time tests are executed.
For creation of PowerShell tests, the Describe block is the level of granularity suggested and one of three tags should be used: "CI", "Feature", or "Scenario". If the tag is not provided, tests in that describe block will be run any time tests are executed.
#### Describe
Creates a logical group of tests. All Mocks and TestDrive contents defined within a Describe block are scoped to that Describe ; they will no longer be present when the Describe block exits. A Describe block may contain any number of Context and It blocks.
Creates a logical group of tests. All Mocks and TestDrive contents defined within a Describe block are scoped to that Describe; they will no longer be present when the Describe block exits. A `Describe block may contain any number of Context and It blocks.
#### Context
Provides logical grouping of It blocks within a single Describe block. Any Mocks defined inside a Context are removed at the end of the Context scope, as are any files or folders added to the TestDrive during the Context block's execution. Any BeforeEach or AfterEach blocks defined inside a Context also only apply to tests within that Context .
@ -92,16 +107,7 @@ Describe "Add-Footer" {
When this test completes, the contents of the TestDrive PSDrive will be removed.
#### Parameter Generation
This is quite a bit different from LITE3, but still possible, see the following:
```
Describe "A test" {
function Test-Xor {
param ($a, $b, $ExpectedResult)
It ("Applies XOR to inputs {0} and {1}" -f $a, $b) {
$a -xor $b | Should be $ExpectedResult
}
}
$testCases = @(
@{ a = 0; b = 1; ExpectedResult = 1 }
@ -109,13 +115,16 @@ $testCases = @(
@{ a = 1; b = 1; ExpectedResult = 0 }
@{ a = 0; b = 0; ExpectedResult = 0 }
)
foreach ($testCase in $testCases) {
Test-Xor @testCase
}
Describe "A test" {
It "<a> -xor <b> should be <expectedresult>" -testcase $testcases {
param ($a, $b, $ExpectedResult)
$a -xor $b | Should be $ExpectedResult
}
}
```
You can construct the values to pass as parameters, including the expected value, and use that iteratively in a loop. Note the location of the `It` block
You can also construct loops and pass values as parameters, including the expected value, but Pester does this for you.
#### Mocking
Mocks the behavior of an existing command with an alternate implementation. This creates new behavior for any existing command within the scope of a Describe or Context block. The function allows you to specify a script block that will become the command's new behavior.
@ -128,7 +137,7 @@ Context "Get-Random is not random" {
}
}
```
More information may be found here: https://github.com/pester/Pester/wiki/Mock
More information may be found on the [wiki](https://github.com/pester/Pester/wiki/Mock)
### Free Code in a Describe block
Code execution in Pester can be very subtle and can cause issues when executing test code. The execution of code which lays outside of the usual code blocks may not happen as you expect. Consider the following:
```
@ -138,14 +147,14 @@ Describe it {
Write-Host -for DarkRed "Before BeforeAll"
BeforeAll { write-host -for Blue "In Context BeforeAll" }
Write-Host -for DarkRed "After BeforeAll"
 
Write-Host -for DarkRed "Before AfterAll"
AfterAll { Write-Host -for Blue "In Context AfterAll" }
Write-Host -for DarkRed "After AfterAll"
 
BeforeEach { Write-Host -for Blue "In BeforeEach" }
AfterEach { Write-Host -for Blue "In AfterEach" }
 
Write-Host -for DarkRed "Before It"
It "should not be a surprise" {
1 | should be 1
@ -189,3 +198,4 @@ The DESCRIBE BeforeAll block is executed before any other code even though it wa
Generally, you should have code reside in one of the code block elements of `[Before|After][All|Each]`, especially if those block rely on state set by free code elsewhere in the block.

138
docs/testing-guidelines/testing-guidelines.md Normal file → Executable file
View file

@ -1,7 +1,135 @@
author: Jim
# Testing Guidelines
Testing is a critical part of the PowerShell project.
The PowerShell team created nearly 100,000 tests over the last 12 years which we run as part of the release process for PowerShell in Windows.
Having all of those tests available for the initial release of OPS was not feasible, and we have targeted those tests which
we believe will provide us the ability to catch regressions in the areas which have had the largest changes for OPS. It is our
intent to continue to release more and more of our tests until we have the coverage we need.
For creating new tests, please review the
[documents](https://github.com/PowerShell/PowerShell/tree/master/docs/testing-guidelines) on how to
create tests for OpenPowerShell
## CI System
We use [AppVeyor](http://www.appveyor.com/) as a continuous integration (CI) system for Windows
and [Travis CI](http://www.travis-ci.com) for non-Windows platforms.
### AppVeyor
In the `README.md` at the top of the repo, you can see AppVeyor badge.
It indicates the last build status of `master` branch.
Hopefully, it's green:
![AppVeyor-Badge-Green.png](Images/AppVeyor-Badge-Green.png)
This badge is **clickable**; you can open corresponding build page with logs, artifacts and tests results.
From there you can easily navigate to the build history.
### Travis CI
Travis CI works similarly to AppVeyor. For Travis CI there will be multiple badges:
The badges indicate the last build status of `master` branch for different platforms.
Hopefully, each badge is green:
![Travis-CI-Badge-Green.png](Images/Travis-CI-Badge-Green.png)
This badge is **clickable**; you can open corresponding build page with logs, artifacts and tests results.
From there you can easily navigate to the build history.
### Getting CI Results
CI System builds (Appveyor and Travis CI) and runs tests on every pull request and provides quick feedback about it.
![AppVeyor-Github](Images/AppVeyor-Github.png)
These green check boxes and red crosses are **clickable** as well.
They will bring you to the corresponding page with details.
## Test Frameworks
### Pester
Our script based test framework is [Pester](https://github.com/Pester/Pester). This is the framework which we are using internally
at Microsoft for new script based tests, and a large number of the tests which are part of the OPS project have been migrated from that test base. Pester tests can be used to test most of PowerShell behavior (even some API operations can
easily be tested in Pester)
Substantial changes were required, to get Pester executing on Non-Windows systems. These changes are not yet in the
official Pester code base. Some features of Pester may not be available or may have incorrect behavior. Please make sure
to create issues in [PowerShell/PowerShell](https://github.com/PowerShell/PowerShell/issues) (not Pester) for anything that you find.
### xUnit
For those tests which are not easily run via Pester, we have decided to use [xUnit](https://xunit.github.io/) as the test framework.
Currently, we have a minuscule number of tests which are run via xUnit,
## Running Tests outside of CI
When working on new features or fixes, it is natural to want to run those tests locally before
making a PR. Two helper functions are part of the build.psm1 module to help with that:
* `Start-PSPester` will execute all Pester tests which are run by the CI system
* `Start-PSxUnit` will execute the available xUnit tests run by the CI system
Our CI system runs these as well; there should be no difference between running these on your dev system, versus in CI.
When running tests in this way, be sure that you have started PowerShell with `-noprofile` as some tests will fail if the
environment is not the default or has any customization.
for example, to run all the Pester tests for CI (assuming you are at the root the PowerShell repo):
```
import-module ./build.psm1
Start-PSPester
```
if you wish to run specific tests, that is possible as well:
```
Start-PSPester -Directory test/powershell/engine/Api
```
or a specific Pester test file:
```
Start-PSPester -Directory test/powershell/engine/Api -Test XmlAdapter.Tests.Api
```
### What happens after your PR?
When your PR has successfully passed the CI test gates, your changes will be used to create PowerShell binaries which can be run
in Microsoft's internal test frameworks. The tests that you created for your change and the library of historical tests will be
run to determine if any regressions are present. If these tests find regressions, you'll be notified that your PR is not ready, and provide
you enough information for you to investigate why the failure happened.
> Jason/Jim: Testing requirements (code coverage, required for incoming features/enhancements)
> local build/test execution instructions
> tools needed for build/test execution
> our CI explained
## Test Layout
We have taken a functional approach to the layout of our Pester tests. You should place new tests in their appropriate
location. If you are making a fix to a cmdlet in a module, the test belongs in the module directory.
If you are unsure; you can make it part of your PR, or create an issue. The current layout of tests is:
* test/powershell/engine
* test/powershell/engine/Api
* test/powershell/engine/Basic
* test/powershell/engine/ETS
* test/powershell/engine/Help
* test/powershell/engine/Logging
* test/powershell/engine/Module
* test/powershell/engine/ParameterBinding
* test/powershell/engine/Runspace
* test/powershell/engine/Logging/MessageAnalyzer
* test/powershell/Host
* test/powershell/Host/ConsoleHost
* test/powershell/Host/TabCompletion
* test/powershell/Language
* test/powershell/Modules
* test/powershell/Provider
* test/powershell/Scripting
* test/powershell/Scripting/Debugging
* test/powershell/Scripting/NativeExecution
* test/powershell/SDK
* test/powershell/Security
* test/powershell/Language/Classes
* test/powershell/Language/Interop
* test/powershell/Language/Operators
* test/powershell/Language/Parser
* test/powershell/Language/Interop/DotNet
* test/powershell/Modules/Microsoft.PowerShell.Archive
* test/powershell/Modules/Microsoft.PowerShell.Core
* test/powershell/Modules/Microsoft.PowerShell.Diagnostics
* test/powershell/Modules/Microsoft.PowerShell.Management
* test/powershell/Modules/Microsoft.PowerShell.Security
* test/powershell/Modules/Microsoft.PowerShell.Utility
* test/powershell/Modules/Microsoft.PowerShell.Security
* test/powershell/Modules/PSReadLine