maxmustermann/pulumi
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
pulumi/scripts/test_subsets.py

77 lines
3.2 KiB
Python
Raw Normal View History

Fanout build experiment (#7628) * Experiment with gotestsum and test timings * Fix to locating the helper script * Fix the code for installing gotestsum * Try alternative installation method * Use go to compute test stats; Python fails parsing time values * Try version without v * Try with fixed gorelaser config * Fix test time correlation * Try a stable test stat sort finally * Use more accurate test duration aggregation * Include python and auto-api tests in the Go timing counts * Bring back TESTPARALLELISM * Fix test compilation * Only top 100 slow tests * Try to fracture build matrix to fan out tests * Do not run Publish Test Results on unsuppored Mac * Auto-create test-results-dir * Fix new flaky test by polling for logs * Try to move native tests to their own config * Actually skip * Do not fail on empty test-results folder * Try again * Try once more * Integration test config is the crit path - make it smaller * Squash underutilized test configurations * Remove the test result summary box from PR - counts now incorrec * Remove debugging step
2021-07-27 16:07:15 +02:00
"""Defines test subsets.
When removing or introducing new test subsets, make sure the
Apply the same build tricks we run on PRs to master workflow (#7685)
2021-07-30 00:18:15 +02:00
`test-subset` build matrix in these files:
- `.github/workflows/run-build-and-acceptance-tests.yml`
- `.github/workflows/master.yml`
Fan out release workflow (#7687)
2021-07-30 02:38:56 +02:00
- `.github/workflows/release.yml`
Apply the same build tricks we run on PRs to master workflow (#7685)
2021-07-30 00:18:15 +02:00
An implied subset `etc` will catch tests not matched by any
Fanout build experiment (#7628) * Experiment with gotestsum and test timings * Fix to locating the helper script * Fix the code for installing gotestsum * Try alternative installation method * Use go to compute test stats; Python fails parsing time values * Try version without v * Try with fixed gorelaser config * Fix test time correlation * Try a stable test stat sort finally * Use more accurate test duration aggregation * Include python and auto-api tests in the Go timing counts * Bring back TESTPARALLELISM * Fix test compilation * Only top 100 slow tests * Try to fracture build matrix to fan out tests * Do not run Publish Test Results on unsuppored Mac * Auto-create test-results-dir * Fix new flaky test by polling for logs * Try to move native tests to their own config * Actually skip * Do not fail on empty test-results folder * Try again * Try once more * Integration test config is the crit path - make it smaller * Squash underutilized test configurations * Remove the test result summary box from PR - counts now incorrec * Remove debugging step
2021-07-27 16:07:15 +02:00
explicit subset listed here.
A note on the format of TEST_SUBSETS. The keys are test configuration
names, and the values are lists of either Go packages containing the
tests, or test suites names as passed to `run-testsuite.py`.
Faster CI PR verification on Mac OS: run fewer tests, use fewer workers (#8004) * Experiment with matrix exclude/include rules * Check if this applies to current PR * Try again on the right yml file * Why is GHA trying to find self-hosted workers * Revert all changes * No-op change * Try a system where PR verification uses only 1 worker * Fix include directive so that mac job has the right variables * Fix YAML syntax
2021-09-22 01:54:47 +02:00
The special `etc` test subset catches all unlisted tests. This subset
will always run on every PR. Other tests subsets may be skipped on PR
verification for platforms such as Mac OS where there is currently a
shortage of runners; these tests will still run on `master` and
`release` verifications.
Fanout build experiment (#7628) * Experiment with gotestsum and test timings * Fix to locating the helper script * Fix the code for installing gotestsum * Try alternative installation method * Use go to compute test stats; Python fails parsing time values * Try version without v * Try with fixed gorelaser config * Fix test time correlation * Try a stable test stat sort finally * Use more accurate test duration aggregation * Include python and auto-api tests in the Go timing counts * Bring back TESTPARALLELISM * Fix test compilation * Only top 100 slow tests * Try to fracture build matrix to fan out tests * Do not run Publish Test Results on unsuppored Mac * Auto-create test-results-dir * Fix new flaky test by polling for logs * Try to move native tests to their own config * Actually skip * Do not fail on empty test-results folder * Try again * Try once more * Integration test config is the crit path - make it smaller * Squash underutilized test configurations * Remove the test result summary box from PR - counts now incorrec * Remove debugging step
2021-07-27 16:07:15 +02:00
"""
TEST_SUBSETS = {
'integration': [
Faster CI PR verification on Mac OS: run fewer tests, use fewer workers (#8004) * Experiment with matrix exclude/include rules * Check if this applies to current PR * Try again on the right yml file * Why is GHA trying to find self-hosted workers * Revert all changes * No-op change * Try a system where PR verification uses only 1 worker * Fix include directive so that mac job has the right variables * Fix YAML syntax
2021-09-22 01:54:47 +02:00
'github.com/pulumi/pulumi/tests/integration',
],
'integration-and-codegen': [
'github.com/pulumi/pulumi/tests/integration/aliases',
'github.com/pulumi/pulumi/tests/integration/custom_timeouts',
'github.com/pulumi/pulumi/tests/integration/delete_before_create',
'github.com/pulumi/pulumi/tests/integration/dependency_steps',
'github.com/pulumi/pulumi/tests/integration/double_pending_delete',
'github.com/pulumi/pulumi/tests/integration/duplicate_urns',
'github.com/pulumi/pulumi/tests/integration/partial_state',
'github.com/pulumi/pulumi/tests/integration/policy',
'github.com/pulumi/pulumi/tests/integration/protect_resources',
'github.com/pulumi/pulumi/tests/integration/query',
'github.com/pulumi/pulumi/tests/integration/read/import_acquire',
'github.com/pulumi/pulumi/tests/integration/read/read_dbr',
'github.com/pulumi/pulumi/tests/integration/read/read_relinquish',
'github.com/pulumi/pulumi/tests/integration/read/read_replace',
'github.com/pulumi/pulumi/tests/integration/recreate_resource_check',
'github.com/pulumi/pulumi/tests/integration/steps',
'github.com/pulumi/pulumi/tests/integration/targets',
'github.com/pulumi/pulumi/tests/integration/transformations',
'github.com/pulumi/pulumi/tests/integration/types',
'github.com/pulumi/pulumi/pkg/v3/codegen',
'github.com/pulumi/pulumi/pkg/v3/codegen/docs',
'github.com/pulumi/pulumi/pkg/v3/codegen/dotnet',
'github.com/pulumi/pulumi/pkg/v3/codegen/go',
'github.com/pulumi/pulumi/pkg/v3/codegen/hcl2',
'github.com/pulumi/pulumi/pkg/v3/codegen/hcl2/model',
'github.com/pulumi/pulumi/pkg/v3/codegen/hcl2/syntax',
'github.com/pulumi/pulumi/pkg/v3/codegen/importer',
'github.com/pulumi/pulumi/pkg/v3/codegen/nodejs',
'github.com/pulumi/pulumi/pkg/v3/codegen/python',
'github.com/pulumi/pulumi/pkg/v3/codegen/schema',
Fanout build experiment (#7628) * Experiment with gotestsum and test timings * Fix to locating the helper script * Fix the code for installing gotestsum * Try alternative installation method * Use go to compute test stats; Python fails parsing time values * Try version without v * Try with fixed gorelaser config * Fix test time correlation * Try a stable test stat sort finally * Use more accurate test duration aggregation * Include python and auto-api tests in the Go timing counts * Bring back TESTPARALLELISM * Fix test compilation * Only top 100 slow tests * Try to fracture build matrix to fan out tests * Do not run Publish Test Results on unsuppored Mac * Auto-create test-results-dir * Fix new flaky test by polling for logs * Try to move native tests to their own config * Actually skip * Do not fail on empty test-results folder * Try again * Try once more * Integration test config is the crit path - make it smaller * Squash underutilized test configurations * Remove the test result summary box from PR - counts now incorrec * Remove debugging step
2021-07-27 16:07:15 +02:00
],
Faster CI PR verification on Mac OS: run fewer tests, use fewer workers (#8004) * Experiment with matrix exclude/include rules * Check if this applies to current PR * Try again on the right yml file * Why is GHA trying to find self-hosted workers * Revert all changes * No-op change * Try a system where PR verification uses only 1 worker * Fix include directive so that mac job has the right variables * Fix YAML syntax
2021-09-22 01:54:47 +02:00
'auto': [
# Primary Go-driven Auto API tests
'github.com/pulumi/pulumi/sdk/v3/go/auto',
# Auto API tests driven by dotnet
[automation-api] Exclude tests from test_fast. (#7986) The Automation API tests take long enough that thay don't really fit into `test_fast`. These changes move those tests to `test_all`.
2021-09-17 02:33:33 +02:00
'auto-dotnet',
Faster CI PR verification on Mac OS: run fewer tests, use fewer workers (#8004) * Experiment with matrix exclude/include rules * Check if this applies to current PR * Try again on the right yml file * Why is GHA trying to find self-hosted workers * Revert all changes * No-op change * Try a system where PR verification uses only 1 worker * Fix include directive so that mac job has the right variables * Fix YAML syntax
2021-09-22 01:54:47 +02:00
# Auto API tests driven by npm
[automation-api] Exclude tests from test_fast. (#7986) The Automation API tests take long enough that thay don't really fit into `test_fast`. These changes move those tests to `test_all`.
2021-09-17 02:33:33 +02:00
'auto-nodejs',
Faster CI PR verification on Mac OS: run fewer tests, use fewer workers (#8004) * Experiment with matrix exclude/include rules * Check if this applies to current PR * Try again on the right yml file * Why is GHA trying to find self-hosted workers * Revert all changes * No-op change * Try a system where PR verification uses only 1 worker * Fix include directive so that mac job has the right variables * Fix YAML syntax
2021-09-22 01:54:47 +02:00
# Auto API tests driven by pytest
[automation-api] Exclude tests from test_fast. (#7986) The Automation API tests take long enough that thay don't really fit into `test_fast`. These changes move those tests to `test_all`.
2021-09-17 02:33:33 +02:00
'auto-python',
Fanout build experiment (#7628) * Experiment with gotestsum and test timings * Fix to locating the helper script * Fix the code for installing gotestsum * Try alternative installation method * Use go to compute test stats; Python fails parsing time values * Try version without v * Try with fixed gorelaser config * Fix test time correlation * Try a stable test stat sort finally * Use more accurate test duration aggregation * Include python and auto-api tests in the Go timing counts * Bring back TESTPARALLELISM * Fix test compilation * Only top 100 slow tests * Try to fracture build matrix to fan out tests * Do not run Publish Test Results on unsuppored Mac * Auto-create test-results-dir * Fix new flaky test by polling for logs * Try to move native tests to their own config * Actually skip * Do not fail on empty test-results folder * Try again * Try once more * Integration test config is the crit path - make it smaller * Squash underutilized test configurations * Remove the test result summary box from PR - counts now incorrec * Remove debugging step
2021-07-27 16:07:15 +02:00
]
}
Reference in a new issue Copy permalink