Build the test on the host, then mount it and run it in the container.
This allows us to make use of the usual integration testing framework to
handle running `npm/yarn`, `pip`, etc.
The test files are currently included in the npm package, and we have packages that depend on the test files currently, so when installing the linkable `@pulumi/pulumi` package locally, don't delete the tests.
This fixes an issue we didn't forsee. If we do a patch release by
tagging a commit as `v1.1.1`, the next version we generate will be
`v1.2.1-alpha.....` which is not what we want. When bumping the minor
version, we should reset the patch version to zero, regardless of what
it is.
This is https://github.com/pulumi/scripts/pull/84 but for
pulumi/pulumi (which doesn't use the shared script because we have to
do special things for pulumi/pulumi)
Previously, we would compute a version for a build by taking the last
relaesed bversion and then increment the patch number. We did this
because during pre 1.0 milestones, we would use the minor version to
say that there had been a "breaking change" and use the patch version
to mean there had been fixes or new features.
For packages that are 1.0 or later, we are now going to bump the minor
version per release and will use the patch version just for cases
where we do one off hot fixes.
This change updates our version generation logic to be in line with
this new plan.
These changes wire up a new script that clones
https://github.com/pulumi/docs, regenerates the appropriate parts of the
docs site, and opens a pull request with the changes. This script
executes on any build in which we publish packages that _also_ has a tag
present.
Previously, we only published docker images for non-prerelease
builds. However, on our move to 1.0.0, we will be publishing a few
`-beta` (and perhaps one or more `-rc`) builds. These are high quality
builds we want folks to use. So, we will publish them to DockerHub and
update the `latest` tag, similar to what we do for NPM and PyPI.
I will manually publish 1.0.0-beta.2, but landing these changes means
that future releases will be published automatically
Fixes#3092
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
This change does two things:
- It ensures that as part of publishing the SDK, we also publish an
updated pulumi/pulumi docker image (tagged with both `latest` and
`vX.Y.Z`
- Makes this image published by this repository less perscriptive in a
workflow. The instead of having wrapper scripts that try to invoke
Pulumi based on conventions. It is now just a base image that has
the pulumi CLI installed, as well as the SDKs for the major cloud
providers. We'll use this base layer in our github actions image,
which will layer on a github actions centric workflow
Fixes#1991
We publish some artifacts from this repository, which are specific to
an OS. In the past, we only had a single Linux leg and so we did not
need any additional conditional logic.
However, we now have multiple Linux legs (across different node
versions) running in parallel, so if we try to publish the same
artifacts at the same time, we can run into issues because two `aws s3
cp` jobs try to write to the same file at the same time.
Follow similar logic to our skips we do around publishing NPM and PyPI
packages here, as well.
The logic to append the `+dirty` tag to a generated version on Windows
was incorrect in the case where we were producing a version for a
tagged build. In that case we would append `+dirty` when the work tree
was clean and not append it when it was dirty.
With this change, you can now use the `dev` tag of `@pulumi/pulumi` to
get the latest version of the SDK and install it from get.pulumi.com
This is helpful for some of our internal testing.
Under our old versioning system, when we started a new point release,
we'd tag the HEAD commit of master with a tag like `v0.16.6-dev` and
our scripts would use this to generate a new version number. This
required a great deal of gymnastics when producing a release and
caused us to litter these -dev tags everywhere.
To improve this, we change version number generation to the following
strategy:
1. If the commit we are building has a tag applied to it, use that tag
as the version (appending the dirty bit metadata to the version, if
needed).
2. If the commit we are bulding does not have a tag applied to it,
take the version from the next reachable tag, increment the patch
version and then append the `-dev` pre-release tag. As part of this,
we also make a slight tweek to our semver generation such that instead
of `-dev<TIMESTAMP>` we use `-dev.<TIMESTAMP>` which is more in line
with what semver recommends.
* Remove TODO for issue since fixed in PPCs.
* Update issue reference to source
* Update comment wording
* Remove --ppc arg of stack init
* Remove PPC references in int. testing fx
* Remove vestigial PPC API types
For CI situations, we'll support a simple stacks map file, e.g.
{
"refs/heads/master": "production",
"refs/heads/testing": "test'
}
and, when PULUMI_CI is set, we'll use it to select the stack.
This is purely for experimental purposes; we're not sure this is
exactly what we want right now, but it's better than the manual
munging we've been doing with various bash scripts, etc. right now.
This adds a package installation step just before calling the
`pulumi` CLI, making it easier to just volume mount a project
without needing to have `npm install`ed its contents yet.
Pulumi used to have a much more complicated install process, whereas
today, this is no longer the case. You simply unpack pulumi to a
folder of your choice, add it to the `$PATH` and then go.
The `install.sh` was writtten back when Pulumi had to be installed
into its own directory. It assumed that it "owned" this directory and
when the script hit an error would clean up its half processed state
before trying to exit. While this was fine out of the box (since we
default to installing to `/usr/local/pulumi`) if you overrode the
install location to just say `/usr/local` *and* we hit an error in the
script, the script would try to remove everything from `/usr/local` as
part of cleaning itself up.
Since we no longer need any of this extra install logic, we'll just
remove `install.sh` completely. The SDK tarball will now contain a
single top level directory (named `pulumi`) with all of our binaries
under it. Manual installs will now just mean unpacking the tarball
somewhere and putting that `pulumi` folder on your path, or as a
simplification, copying all the binaries from the `pulumi` folder into
an existing folder that is already on your path.
This also removes the need to ever ask the user to `sudo` during an
install. Users now have complete control over where they put our
binaries, which is exactly what you want from a manual install
process.
Stop cloning pulumi/home. This doesn't work in Travis because public
repositories can not have private SSH keys, which we'd need to clone
this repository. All the scripts we consume from there are now in
pulumi/scripts and so we'll just consume them from there.
This change includes the Python and Golang language hosts in the Windows
SDK. As part of this change, I had to adjust how we launched the second
stage of the language host, since we can't depend on the shebang, so now
we invoke `python` passing the executor and then the arguments.
Fixes#1509
Previously, we published builds to rel.pulumi.com and only put actual
released builds (eg. rc's and final builds) on get.pulumi.com. We
should just publish all of the SDK builds to get.pulumi.com.
This also makes a slight tweak to the filename of the package we
upload (we took the switch over to get.pulumi.com to make this change
and now that we are uploading automatically, we need to encode this
change instead of doing it by hand).
All scripts that are generally useful across all builds have been moved
into `pulumi/scripts`. These changes clone that repository and retarget
the various scripts to their new location.
We retained these modules to support using v0.11.X and earlier
versions of @pulumi/pulumi, which required a native module to do
closure serialization. 0.12.X does not need this, so lets stop
including it.
Our logic for converting npm style versions to PEP-440 style versions
was not correct in some cases. This change fixes this.
As part of this change we no longer produce a NPM version that would
be just X.Y.Z-dev, instead for development versions we always include
both the timestamp of the commit and the commit hash.
Instead of trying to use a bunch of sed logic to do our conversions,
we now have a small go program that uses a newly added library in
pkg/util. A side effect of this is that we can more easily write tests
to ensure the conversion works as expected.
Fixes#1243
Instead of using a shell script to jump from the language host into
node, just invoke node directly. This makes our start-up path a little
simpler to understand and indirectly fixespulumi/home#156, where we
would fail on Windows if the `-exec` script was in a folder that had
spaces in it (due to a subtle interaction between how go launches cmd
files and how cmd.exe parses arguments).
This change eliminates our dependencies on the SDK repo. Now that
SDKs are comprised solely of pulumi/pulumi artifacts, a separate repo
isn't required. This allows us to simplify some of the distribution.
The install.sh script is modified slightly, to permit overriding the
default install location using $PULUMI_INSTALL_PATH.
While we no longer use the native runtime module, older versions of
@pulumi/pulumi still require it. Let's continue to have the launcher
put the native module location on the `$PATH`. And we'll include them
in the SDK for a while longer.
Fixes#1177
After the move to stop including packages in the SDK, we no longer
published an SDK per build. This corrects this. Since the only things
in the SDK today are the language plugins and the CLI itself, we can
publish an SDK per build from pulumi/pulumi.
This change re-uses the existing infrastructure we have in
pulumi/sdk.
Fixes#1076
* Lift snapshot management out of the engine
This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.
As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.
Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.
Remove dead code, Steps no longer need a reference to the plan iterator that created them
Fixing various issues that arise when bringing up pulumi-aws
Line length broke the build
Code review: remove dead field, fix yaml name error
Rebase against master, provide implementation of StackPersister for cloud backend
Code review feedback: comments on MutationStatus, style in snapshot.go
Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager
Code review feedback: use a channel for synchronization
Add a comment and a new test
* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots
* Add a lot of tests - covers all of the non-error paths of BeginMutation and End
* Fix a test resource provider
* Add a few tests, fix a few issues
* Rebase against master, fixed merge
* Implement closure scope chain analysis in pure TypeScript
This change makes use of four V8 intrinsics to avoid having to use a
native module to inspect the scope chains of live Function objects. This
unfortunately leads to the limitation of not allowing captures of 'this'
in arrow functions, but that is something we are willing to live with
for now.
* Remove native module build and restore from the Makefile
* CR feedback: Be a little more efficient when scanning the scope chain
* Nuke everything related to custom Node versions and the native Node module
* CR feedback: rename native.ts -> v8.ts, document some interfaces in v8.ts
`npm publish`'s default was to tag the package we published with
`latest` tag. The NPM ecosystem has expected semantics around this
tag (it uses it by default if you don't pass a version).
From the NPM Docs:
> Typically, projects only use the latest tag for stable release
> versions, and use other tags for unstable versions such as
> prereleases.
We were not doing this, but now we will. We'll have a new tag `dev`
which is the latest build out of CI, and we'll tag builds without a
pre-release tag with "latest".
This change actually makes our Python version numbers conformant
to PEP440. Previously we were including the Git commit hash in the
alpha "version number" part, which is incorrect. This simply led to
warnings upon publication and installation, but that warning very
clearly states that support for invalid versions will stop at some
point. This change puts any "informative" parts, like the Git hash,
inside of a local version tag, where such things are permitted.
Also move away from the inline sed silliness so that we can more
easily share this logic across all of our repos.
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
This change uses virtualenv to insulate us from platform differences
in our building of the Python SDK, and to create an isolated Python 2
environment. This includes meaning we don't need to worry about the
specific location and behavior of Pylint. I *think* this will work
no matter whether it's Mac, Ubuntu, ArchLinux, Windows, and so on.
We do install to the --user directory in the install target using
`pip install -e`, however, which enables the machine-wide symlinking
that we need to support various workflows.
This fixespulumi/pulumi#1007.
While it's safe to publish the tgz that we use internally for other
repositories that are on "the link plan" after the build completes, we
shouldn't publish packages to NPM and PyPi at that point. There are
two reasons for doing this:
1. Publishing packages before they are tested, which means we could
end up publishing packages that don't work.
2. NPM prevents publishing the same package more than once, so if we
had to re-run the job (due to tests failing for transient issues), the
publish step will start failing, preventing us from running the tests
at all.
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
We now publish the Pulumi Python SDK package to our private PyPI
server at the same time we also publish the NPM package. For now,
we use the test Pulumi.com service, and will switch to staging as
soon as it becomes available.
This change refactors the way we do ${VERSION} substitution in both
the Node.js SDK's version.js and package.json, so that it can work on
Windows. This is required now that we are actually parsing semvers.
This brings back the Node.js language plugin's GetRequiredPlugins
function, reimplemented in Go now that the language host has been
rewritten from JavaScript. Fairly rote translation, along with
some random fixes required to get tests passing again.
New-Item produces a new object on success, which ultimately gets
output by the make_release.ps1 cmdlet and consumed by the release.ps1
script. This messes up the release script that is expecting exactly
one object to come out of the pipeline from make_release.ps1.
Fixes#356. Instead of downloading a node binary with our closure
serialization code linked-in, this PR instead publishes the
`nativeruntime.node` produced by the NodeJS SDK build as part of the SDK.
This has a number of advantages. First, it is vastly more easy to
develop closure.cc in this configuration. Second, we have the ability
to ship different `nativeruntime.node`s side-by-side, paving the way
for enabling future versions of Node. Third, we don't have to stay
in the business of shipping custom builds of Node, although we do still
need to ship a version of Node with minor modifications in order for
Windows to still work.
The windows build was still on the old plan from way back when where
we had binaries littered in the build tree and you had to add parts of
your build-tree to the `%PATH%` for the integration tests to work.
This cleans that up and moves all of our scripts that invoke
javascript to be on the same plan. They invoke our specially named
node with a relative path to the JS code we want to run.
In order to begin publishing our core SDK package to NPM, we will
need it to be underneath the @pulumi scope so that it may remain
private. Eventually, we can alias pulumi back to it.
This is part of pulumi/pulumi#915.
* experimental: separate language host from node
* Remove langhost details from the NodeJS SDK runtime
* Cleanup
* Work around an issue where Node sometimes loads the same module twice in two different contexts, resulting in two distinct module objects. Some additional cleanup.
* Add some tests
* Fix up the Windows script
* Fix up the install scripts and Windows build
* Code review feedback
* Code review feedback: error capitalization
Ideally we would just use `make build` and `make install` in favor of
having yet another way to build the product, but before we can do that
in general we need to come up with a better story for cross
building. For now, just ensure we pass the correct version string to
go build when building.
