MirrorHub/forgejo
0
0
Fork 0
mirror of https://codeberg.org/forgejo/forgejo.git synced 2024-11-09 03:11:51 +01:00
Code Issues Projects Releases Packages Wiki Activity
forgejo/modules/git
History
Peter Gardfjäll e28cc79c92
Improve sync performance for pull-mirrors (#19125) 
This addresses https://github.com/go-gitea/gitea/issues/18352

It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors.

For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold:
    
1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date.
    
2. the procedure for getting _each tag_ involves a series of git operations    
    ```bash
     git show-ref --tags -- v8.2.4477
     git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
     ```    

     of which the `git rev-list --count` can be particularly heavy.
    
This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go.
    
For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine:

- https://github.com/spring-projects/spring-boot (223 tags)
  - before: `0m28,673s`
  - after: `0m2,244s`
- https://github.com/kubernetes/kubernetes (890 tags)
  - before: `8m00s`
  - after: `0m8,520s`
- https://github.com/vim/vim (13954 tags)
  - before: `14m20,383s`
  - after: `0m35,467s`

 

I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used.  I'll add unit tests for those if the overall PR looks promising.
2022-03-31 14:30:40 +02:00
..
foreachref Improve sync performance for pull-mirrors (#19125) 2022-03-31 14:30:40 +02:00
pipeline Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
tests/repos rm .sample hooks which aren't used (#19101) 2022-03-16 10:33:07 +00:00
batch_reader.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
blame.go Propagate context and ensure git commands run in request context (#17868) 2022-01-19 23:26:57 +00:00
blame_test.go refactor: move from io/ioutil to io and os package (#17109) 2021-09-22 13:38:34 +08:00
blob.go Read expected buffer size (#17409) 2021-10-24 22:12:43 +01:00
blob_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
blob_nogogit.go Make Requests Processes and create process hierarchy. Associate OpenRepository with context. (#17125) 2021-11-30 20:06:32 +00:00
blob_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
command.go Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266) 2022-03-31 13:56:22 +02:00
command_race_test.go Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266) 2022-03-31 13:56:22 +02:00
command_test.go Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266) 2022-03-31 13:56:22 +02:00
commit.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
commit_convert_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
commit_info.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
commit_info_gogit.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
commit_info_nogogit.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
commit_info_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
commit_reader.go Tagger can be empty, as can Commit and Author - tolerate this (#15835) 2021-05-12 01:11:42 -04:00
commit_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
diff.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
diff_test.go Prevent off-by-one error on comments on newly appended lines (#18029) 2021-12-20 02:15:49 +00:00
error.go Fix 500 Error with branch and tag sharing the same name #15592 (#16040) 2021-06-23 17:08:26 -04:00
git.go Refactor git.Command.Run*, introduce RunWithContextString and RunWithContextBytes (#19266) 2022-03-31 13:56:22 +02:00
git_test.go Use gitea logging module for git module (#16243) 2021-06-25 17:54:08 +01:00
hook.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
last_commit_cache.go Defer Last Commit Info (#16467) 2021-10-08 15:08:22 +02:00
last_commit_cache_gogit.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
last_commit_cache_nogogit.go Defer Last Commit Info (#16467) 2021-10-08 15:08:22 +02:00
lfs.go Unexport git.GlobalCommandArgs (#18376) 2022-01-25 18:15:58 +00:00
log_name_status.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
notes.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
notes_gogit.go Stop logging an error when notes are not found (#18626) 2022-02-06 15:11:35 +08:00
notes_nogogit.go Stop logging an error when notes are not found (#18626) 2022-02-06 15:11:35 +08:00
notes_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
parse_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
parse_gogit_test.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
parse_nogogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
parse_nogogit_test.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
README.md move code.gitea.io/git to code.gitea.io/gitea/modules/git (#6364) 2019-03-27 17:33:00 +08:00
ref.go Refactor various strings (#17784) 2021-12-02 15:28:08 +08:00
remote.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo.go Use a more general (and faster) method to sanitize URLs with credentials (#19239) 2022-03-31 10:25:40 +08:00
repo_archive.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_attribute.go Don't show context cancelled errors in attribute reader (#19006) 2022-03-08 08:30:14 +00:00
repo_attribute_test.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
repo_base.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_base_gogit.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_base_nogogit.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_blame.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo_blob.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
repo_blob_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
repo_blob_nogogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
repo_blob_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_branch.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_branch_gogit.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_branch_nogogit.go Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) 2022-03-29 19:12:33 +02:00
repo_branch_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_commit.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_commit_gogit.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo_commit_nogogit.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo_commit_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_commitgraph.go Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) 2022-03-29 19:12:33 +02:00
repo_commitgraph_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
repo_compare.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_compare_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_gpg.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo_hook.go move code.gitea.io/git to code.gitea.io/gitea/modules/git (#6364) 2019-03-27 17:33:00 +08:00
repo_index.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_language_stats.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
repo_language_stats_gogit.go Add .gitattribute assisted language detection to blame, diff and render (#17590) 2021-11-17 20:37:00 +00:00
repo_language_stats_nogogit.go Prevent dangling GetAttribute calls (#18754) 2022-02-14 18:03:56 +01:00
repo_language_stats_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_object.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_ref.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
repo_ref_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
repo_ref_nogogit.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_ref_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_stats.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_stats_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_tag.go Improve sync performance for pull-mirrors (#19125) 2022-03-31 14:30:40 +02:00
repo_tag_gogit.go Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) 2022-03-29 19:12:33 +02:00
repo_tag_nogogit.go Use full output of git show-ref --tags to get tags for PushUpdateAddTag (#19235) 2022-03-29 19:12:33 +02:00
repo_tag_test.go Improve sync performance for pull-mirrors (#19125) 2022-03-31 14:30:40 +02:00
repo_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
repo_tree.go Change git.cmd to RunWithContext (#18693) 2022-02-11 13:47:22 +01:00
repo_tree_gogit.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
repo_tree_nogogit.go Fix NPE on try to get tag reference via API (#18245) 2022-01-12 20:37:46 +00:00
sha1.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
sha1_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
sha1_nogogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
signature.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
signature_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
signature_nogogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
submodule.go Fix some lints (#17337) 2021-10-17 20:47:12 +01:00
submodule_test.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
tag.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
tag_test.go Fix NPE on try to get tag reference via API (#18245) 2022-01-12 20:37:46 +00:00
tree.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
tree_blob.go Use native git variants by default with go-git variants as build tag (#13673) 2020-12-17 22:00:47 +08:00
tree_blob_gogit.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
tree_blob_nogogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
tree_entry.go On open repository open common cat file batch and batch-check (#15667) 2021-05-10 03:27:03 +02:00
tree_entry_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
tree_entry_mode.go format with gofumpt (#18184) 2022-01-20 18:46:10 +01:00
tree_entry_nogogit.go Propagate context and ensure git commands run in request context (#17868) 2022-01-19 23:26:57 +00:00
tree_entry_test.go Make git.OpenRepository accept Context (#19260) 2022-03-30 03:13:41 +08:00
tree_gogit.go Add bundle download for repository (#14538) 2021-08-24 11:47:09 -05:00
tree_nogogit.go Delete old git.NewCommand() and use it as git.NewCommandContext() (#18552) 2022-02-06 20:01:47 +01:00
utils.go Simplify parameter types (#18006) 2021-12-20 04:41:31 +00:00
utils_test.go Fix ref links in issue overviews for tags (#8742) 2020-05-14 18:55:43 -04:00

README.md

Git Module

This module is merged from https://github.com/go-gitea/git which is a Go module to access Git through shell commands. Now it's a part of gitea's main repository for easier pull request.