Do not attempt to load checkpoint before saving a snapshot

For historical reasons, we used to need to require to load an existing
checkpoint to copy some data from it into the snapshot when saving a
new snapshot. The need for this was removed as part of the general
work in #2678, but we continued to load the checkpoint and then just
disregard the data that was returned (unless there was an error and
that error was not FileNotFound, in which case we would fail).

Our logic for checking if something was FileNotFound was correct when
we wrote it, but when we adopted go-cloud in order to have our
filestate backend also write to blob storage backends like S3, we
forgot that we had checks like `os.IsNotExists()` floating around
which were now incorrect. That meant if the file did not exist for
some reason, instead of going along as planned, we'd error out now
with an error saying something wasn't found.

When we write a checkpoint, we first "backed up" the initial version
by renaming it to include a `.bak` suffix, then we write the new file
in place. However, this can run afoul of eventual consistency models
like S3, since there will be a period of time in which a caller may
observe that the object is missing, even after a new version is
written (based on my understanding of [S3's consistency
model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)

Since we no longer need to actually copy any information from the
previous checkpoint, we can simply remove the call entirely to load
it.

As a follow up, we need to audit places inside the filebased backend
that assume `os.*` functions are going to do what we want them to do,
since in general they will not.

Fixes #2714

CHANGELOG.md

@@ -11,6 +11,8 @@ CHANGELOG
 - Do not crash when renaming a stack that has never been updated, when using the local backend. (fixes 
   [#2654](https://github.com/pulumi/pulumi/issues/2654))
 
+- Fix intermittet "NoSuchKey" issues when using the S3 based backend. (fixes [#2714](https://github.com/pulumi/pulumi/issues/2714)).
+
 ## 1.0.0-beta.2 (2019-08-13)
 
 - Fix the package version compatibility checks in the NodeJS language host.

pkg/backend/filestate/snapshot.go

@@ -15,8 +15,6 @@
 package filestate
 
 import (
-	"os"
-
 	"github.com/pulumi/pulumi/pkg/resource/deploy"
 	"github.com/pulumi/pulumi/pkg/secrets"
 	"github.com/pulumi/pulumi/pkg/tokens"
@@ -35,12 +33,7 @@ func (sp *localSnapshotPersister) SecretsManager() secrets.Manager {
 }
 
 func (sp *localSnapshotPersister) Save(snapshot *deploy.Snapshot) error {
-	_, _, err := sp.backend.getStack(sp.name)
-	if err != nil && !os.IsNotExist(err) {
-		return err
-	}
-
-	_, err = sp.backend.saveStack(sp.name, snapshot, sp.sm)
+	_, err := sp.backend.saveStack(sp.name, snapshot, sp.sm)
 	return err
 
 }