Commit graph

103 commits

Author SHA1 Message Date
Erick Bajao 59cc92897d Include upload_duration in finalized fields 2021-11-09 18:34:44 +00:00
Steve Abrams 0c11cd4024 Dependency Proxy uses workhorse for manifest pulls 2021-11-04 08:53:33 +00:00
Catalin Irimie 6b01a82a70 Exclude internal API and Geo git_ssh routes from secondary proxying
The api/internal routes need to not be proxied to allow detection if
the repository exists locally for a project, and redirection to the
primary if not.

The geo/proxy_git_ssh route is used internally by gitlab-shell to
then proxy the git+ssh operations to the primary, using the internal
URL of the primary (hence why these don't get proxied and is intended
that they hit the secondary).

Changelog: changed
EE: true
2021-10-29 09:50:14 +00:00
John T Skarbek 9a2d2e0c44
Merge remote-tracking branch 'security/master' 2021-10-28 12:06:57 -04:00
Quang-Minh Nguyen d16d9c5712 Fix gitaly configuration file for Workhorse integration tests 2021-10-28 11:20:28 +07:00
GitLab Release Tools Bot 9aaf54dc9c Merge branch 'security-id-limit-uploading-files' into 'master'
Workhorse: Allow uploading only a single file

See merge request gitlab-org/security/gitlab!1892
2021-10-25 11:17:35 +00:00
GitLab Release Tools Bot c449c20f3f Merge branch 'security-id-workhorse-tiff' into 'master'
Avoid decoding the whole tiff image on isTIFF check

See merge request gitlab-org/security/gitlab!1890
2021-10-25 09:40:07 +00:00
Alessio Caiazza 726cd4f974 Run workhorse-gitaly integration tests w/out docker 2021-10-25 14:18:00 +07:00
Quang-Minh Nguyen 82fc506e23 Upgrade gitaly image for workhorse integration test 2021-10-25 14:18:00 +07:00
Patrick Bajao 2ca18f44fe Merge branch 'id-dependency-proxy-set-content-length' into 'master'
Set Content-Length header on Dependency Proxy injection

See merge request gitlab-org/gitlab!72670
2021-10-21 06:37:47 +00:00
Igor Drozdov 02fef51f02 Set Content-Length header on Dependency Proxy injection
Without it the progress bar is not displayed
2021-10-20 09:58:16 +03:00
Igor Drozdov c8a45af37e Deduplicate http transport creation
We use a very tweaked transport to perform http calls

Let's extract it into a helper function to deduplicate code

Changelog: changed
2021-10-19 23:03:54 +03:00
Igor Drozdov 254e11b7a8 Workhorse: Allow uploading only a single file
Gitlab Rails doesn't have endpoints that require uploading
multiple files.

Let's limit it to prevent performance issues and proceed with
a proper solution out of Security Release

Changelog: security
2021-10-19 12:32:23 +03:00
Igor Drozdov 486ee8411c Avoid decoding the whole tiff image on isTIFF check
Changelog: security
2021-10-18 12:57:27 +03:00
Jacob Vosmaer 367573acc7 Merge branch 'id-dependency-proxy-workhorse-inject-2' into 'master'
Implement Dependency proxy via Workhorse injectors

See merge request gitlab-org/gitlab!71890
2021-10-15 13:48:14 +00:00
Igor Drozdov b3e6d6c37e Implement Dependency proxy via Workhorse injectors
Changelog: changed
2021-10-15 14:49:13 +03:00
Vasilii Iakliushin cd60a5bff7 Merge branch 'id-clean-up-gitaly-unavailable-err' into 'master'
Improve Git HTTP err message given when Gitaly unavailable

See merge request gitlab-org/gitlab!71120
2021-10-13 10:30:50 +00:00
Igor Drozdov 772df282e6 Improve Git HTTPS message given when Gitaly unavailable
Changelog: changed
2021-10-12 12:46:56 +03:00
Quang-Minh Nguyen c82a66e14d Implement PostUploadPackWithSidechannel client in Workhorse
Issue: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1220
Changelog: added
2021-10-11 09:29:23 +07:00
Quang-Minh Nguyen a0032b725c Bump Gitaly client version for workhorse 2021-10-11 09:29:22 +07:00
Jacob Vosmaer c9ec6e4e94 Merge branch '625-add-tests-for-serving-content-from-large-zip-files-with-lots-of-metadata' into 'master'
Update workhorse zipartifacts test to ensure minimum number of range requests

See merge request gitlab-org/gitlab!70443
2021-10-08 12:47:52 +00:00
Catalin Irimie f1812b88bd Proxy Geo secondary HTTP pushes + lfs through Workhorse
As the secondary is read-only, we want to ensure pushes are proxied
to the primary, while reads are served localy, same with LFS files.

Changelog: changed
EE: true
2021-10-07 12:23:39 +01:00
Arran Walker 1e4dba76e8 Update workhorse zipartifacts test to ensure minimum number of range requests
zipartifacts uses a http-client based ReaderAt implementation that fetches parts of a remote zip
file by using Range headers. We don't have direct control of the ranges that will be requested, so
this test ensures that changes to the underlying zip reader won't negatively affect performance by
making more range requests than expected.
2021-10-06 14:27:00 +01:00
Jacob Vosmaer 692ffcd97b Revert "Merge branch 'id-dependency-proxy-workhorse-inject' into 'master'"
This reverts merge request !68157
2021-10-05 14:42:48 +00:00
Nick Thomas 1ded4b1939 Merge branch 'jv-workhorse-cleanup-flag' into 'master'
Remove unused struct field

See merge request gitlab-org/gitlab!71738
2021-10-05 14:34:56 +00:00
Jacob Vosmaer 916a647334 Remove unused struct field
Changelog: other
2021-10-05 13:31:29 +00:00
Igor Drozdov c9c9a36a7d Implement Dependency proxy via Workhorse injectors
Changelog: changed
2021-10-05 14:57:39 +03:00
Catalin Irimie b3d88bdafd Prevent Workhorse panics when Geo proxy URL is unset
When the Geo Rails API intends to disable the proxying functionality,
like for example not being a secondary anymore, or the Rails feature
flag is disabled, an empty URL will be returned.

This prevents the empty, invalid URL from getting parsed and resulting
in panics because of it.

Changelog: changed
EE: true
2021-10-01 18:03:49 +01:00
Igor Drozdov e9d2f771e6 Scrub artifacts signed URL in SendEntry logs
Changelog: security
2021-09-24 14:29:55 +03:00
Ash McKenzie 94bcaacebc Merge branch '340162-measure-cached-image-scaler-latencies' into 'master'
Measure image scaler metric when serving cached images

See merge request gitlab-org/gitlab!70483
2021-09-17 02:32:31 +00:00
Ash McKenzie 7e74949113 Merge branch 'build/bump-jwt' into 'master'
build: replace jwt-go with maintained fork

See merge request gitlab-org/gitlab!69147
2021-09-17 02:22:50 +00:00
Manoj M J e6333fb216 Measure image scaler duration for cached images
With this change, we measure durations in Prometheus
(in `imageResizeDurations`) whenever the requested image has not been modified and hence the cached version can be used by the client.

In this case, the image resizer process is not invoked, but
the client gets to see the image anyway.

As apdex is a measure of user satisfaction, it should not matter “where does the image come from?”, rather it should only care about “how soon is the image served”, which is why we have now decided toinclude the measurement of “cached versions” delivery in this apdex.

Changelog: changed
2021-09-16 13:03:22 +05:30
Catalin Irimie ee925603ca Update Devise sign_in path for Geo secondaries
In order to support proxying from a secondary to a primary,
while the authentication can still happen on both sites,
we need to have two different paths so that the secondary-specific
path can be excluded from proxying.

Changelog: changed
EE: true
2021-09-14 17:46:58 +01:00
feistel 4829772ec7 build: Replace jwt-go with maintained fork 2021-09-02 10:22:06 +00:00
feistel 24a3e734d8 build: Bump go version to 1.16 2021-09-02 10:20:47 +00:00
Michael Kozono 6764676e08 Add Workhorse support for HTTPS backend 2021-09-01 18:51:13 +00:00
Jonston Chan 8675a15875 Replace "the the" with "the" 2021-08-26 23:32:25 +00:00
Nick Thomas 674d02a402 Merge branch 'repro-builds' into 'master'
Reproducible Builds: Use SOURCE_DATE_EPOCH for build time if present

See merge request gitlab-org/gitlab!68464
2021-08-20 08:53:09 +00:00
kpcyrd 4d10212841 Reproducible Builds: Use SOURCE_DATE_EPOCH for build time if present 2021-08-18 13:30:48 +02:00
Jacob Vosmaer d80521a01e Remove files that exist at the top level too
Workhorse does not need its own license or contributing
guidelines. Whatever applies to gitlab-org/gitlab applies
to gitlab-org/gitlab/workhorse too.
2021-08-17 16:09:32 +00:00
Jacob Vosmaer c292385208 Remove obsolete changelog 2021-08-17 16:08:23 +00:00
Jacob Vosmaer 5e5bb88d43 Remove code related to standalone development
Remove code and documentation related to the processes
we followed when Workhorse development happened in its
own project.
2021-08-17 16:06:47 +00:00
Mike Kozono a1481cb474
Implement async API calls 2021-08-04 09:31:59 -07:00
Stan Hu 5e865c2e68
Add support for propagation correlation IDs from trusted CIDRs
When Gitaly makes internal API calls back to Workhorse in Git hooks,
Workhorse previously would generate new correlation IDs, making it hard
to trace the entire call flow.

In https://gitlab.com/gitlab-org/labkit/-/merge_requests/123, we added
the ability to propagate correlation IDs from trusted CIDR blocks.

To use this feature, we add two configuraton parameters:

* `trusted_cidrs_for_x_forwarded_for`
* `trusted_cidrs_for_propagation`

If propagation of correlation ID is enabled,
`trusted_cidrs_for_x_forwarded_for` tells LabKit what remote IPs can be
trusted to use the `X-Forwarded-For` HTTP header to resolve the actual
client IP. Note that this parameter is not yet used in Workhorse's
remote IP resolution, but it should be.

`trusted_cidrs_for_propagation` allows Workhorse to restrict propagation
to certain IP ranges. We will want to add the Gitaly servers to this
list.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/324836

Changelog: added
2021-07-26 14:07:38 -07:00
Mike Kozono 51421f25bf
Implement Geo proxy routing
By default, a Geo proxy will proxy all requests to the URL returned by
the API endpoint `/api/v4/geo/proxy`.

Certain routes, defined in `GeoLocalRoutes`, can be served locally for
improved performance. Many more local routes will be added later.
2021-07-22 10:03:25 -07:00
Mike Kozono b5438e7454
Add Debug level logging to Workhorse 2021-07-22 10:03:22 -07:00
Philippe Lafoucrière 7155dc015c Rename Workhorse go module to gitlab/workhorse 2021-07-21 12:15:14 +00:00
Patrick Bajao 9d38f1cef5 Merge branch 'workhorse-fix-cable-backend-default-value' into 'master'
Default cableBackend to authBackend

See merge request gitlab-org/gitlab!65068
2021-07-01 05:24:06 +00:00
Heinrich Lee Yu ff9bd308eb
Default cableBackend to authBackend
Docs already mention this but we are actually defaulting this to
localhost:8080

Changelog: fixed
MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65068
2021-06-30 15:33:07 +08:00
Mike Kozono 376635a78b
Add function to call /api/v4/geo/proxy
When this instance is a Geo proxy, just log that fact for now.
2021-06-29 10:45:32 -07:00