0
0
Fork 0
mirror of https://github.com/go-gitea/gitea synced 2024-12-29 15:34:46 +01:00
Commit graph

943 commits

Author SHA1 Message Date
wxiaoguang
580e21dd2e
Refactor LFS SSH and internal routers (#32473)
Gitea instance keeps reporting a lot of errors like "LFS SSH transfer connection denied, pure SSH protocol is disabled". When starting debugging the problem, there are more problems found. Try to address most of them:

* avoid unnecessary server side error logs (change `fail()` to not log them)
* figure out the broken tests/user2/lfs.git (added comments)
* avoid `migratePushMirrors` failure when a repository doesn't exist (ignore them)
* avoid "Authorization" (internal&lfs) header conflicts, remove the tricky "swapAuth" and use "X-Gitea-Internal-Auth"
* make internal token comparing constant time (it wasn't a serous problem because in a real world it's nearly impossible to timing-attack the token, but good to fix and backport)
* avoid duplicate routers (introduce AddOwnerRepoGitLFSRoutes)
* avoid "internal (private)" routes using session/web context (they should use private context)
* fix incorrect "path" usages (use "filepath")
* fix incorrect mocked route point handling (need to check func nil correctly)
* split some tests from "git general tests" to "git misc tests" (to keep "git_general_test.go" simple)

Still no correct result for Git LFS SSH tests. So the code is kept there
(`tests/integration/git_lfs_ssh_test.go`) and a FIXME explains the details.
2024-11-12 02:38:22 +00:00
wangjingcun
3d6ccbac3f
chore: fix some function names in comment (#32300)
fix some function names in comment
2024-10-22 08:41:05 +08:00
wxiaoguang
afa8dd45af
Make git push options accept short name (#32245)
Just like what most CLI parsers do: `--opt` means `opt=true`

Then users could use `-o force-push` as `-o force-push=true`
2024-10-12 05:42:10 +00:00
ConcurrentCrab
8a9fd7f771
Add pure SSH LFS support (#31516)
Fixes #17554
/claim #17554

Docs PR https://gitea.com/gitea/docs/pulls/49

To test, run pushes like: `GIT_TRACE=1` git push. The trace output
should mention "pure SSH connection".
2024-09-27 10:27:37 -04:00
Lunny Xiao
3f2d8f8730
Fix panic when cloning with wrong ssh format. (#32076) 2024-09-24 01:30:05 +00:00
techknowlogick
d9a7748cdc
bump to go 1.23 (#31855) 2024-09-10 02:23:07 +00:00
techknowlogick
f183783baa
Save initial signup information for users to aid in spam prevention (#31852)
This will allow instance admins to view signup pattern patterns for
public instances. It is modelled after discourse, mastodon, and
MediaWiki's approaches.

Note: This has privacy implications, but as the above-stated open-source
projects take this approach, especially MediaWiki, which I have no doubt
looked into this thoroughly, it is likely okay for us, too. However, I
would be appreciative of any feedback on how this could be improved.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2024-09-09 17:05:16 -04:00
Lunny Xiao
ac34449747
Prevent update pull refs manually and will not affect other refs update (#31931)
All refs under `refs/pull` should only be changed from Gitea inside but
not by pushing from outside of Gitea.
This PR will prevent the pull refs update but allow other refs to be
updated on the same pushing with `--mirror` operations.

The main changes are to add checks on `update` hook but not
`pre-receive` because `update` will be invoked by every ref but
`pre-receive` will revert all changes once one ref update fails.
2024-09-02 07:38:27 +00:00
Rowan Bohde
8de8972baf
fix: allow actions artifacts storage migration to complete succesfully (#31251)
Change the copy to use `ActionsArtifact.StoragePath` instead of the
`ArtifactPath`. Skip artifacts that are expired, and don't error if the
file to copy does not exist.

---

When trying to migrate actions artifact storage from local to MinIO, we
encountered errors that prevented the process from completing
successfully:

* The migration tries to copy the files using the per-run
`ArtifactPath`, instead of the unique `StoragePath`.
* Artifacts that have been marked expired and had their files deleted
would throw an error
* Artifacts that are pending, but don't have a file uploaded yet will
throw an error.

This PR addresses these cases, and allow the process to complete
successfully.
2024-06-05 12:00:56 +08:00
Lunny Xiao
fb7b743bd0
Azure blob storage support (#30995)
This PR implemented object storages(LFS/Packages/Attachments and etc.)
for Azure Blob Storage. It depends on azure official golang SDK and can
support both the azure blob storage cloud service and azurite mock
server.

Replace #25458
Fix #22527 

- [x] CI Tests
- [x] integration test, MSSQL integration tests will now based on
azureblob
  - [x] unit test 
- [x] CLI Migrate Storage
- [x] Documentation for configuration added

------

TODO (other PRs):
- [ ] Improve performance of `blob download`.

---------

Co-authored-by: yp05327 <576951401@qq.com>
2024-05-30 07:33:50 +00:00
dicarne
d0d6aad85f
Supports forced use of S3 virtual-hosted style (#30969)
Add a configuration item to enable S3 virtual-hosted style (V2) to solve
the problem caused by some S3 service providers not supporting path
style (V1).
2024-05-15 08:56:17 -05:00
Lunny Xiao
ebf0c96940
Move database operations of merging a pull request to post receive hook and add a transaction (#30805)
Merging PR may fail because of various problems. The pull request may
have a dirty state because there is no transaction when merging a pull
request. ref
https://github.com/go-gitea/gitea/pull/25741#issuecomment-2074126393

This PR moves all database update operations to post-receive handler for
merging a pull request and having a database transaction. That means if
database operations fail, then the git merging will fail, the git client
will get a fail result.

There are already many tests for pull request merging, so we don't need
to add a new one.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2024-05-07 07:36:48 +00:00
Lunny Xiao
6ad77125ca
Fix missing migrate actions artifacts (#30874)
The actions artifacts should be able to be migrate to the new storage
place.
2024-05-07 06:45:30 +00:00
wxiaoguang
7c613f100e
Make sure git version&feature are always prepared (#30877)
Otherwise there would be more similar issues like #29287
2024-05-06 18:34:16 +02:00
wxiaoguang
d3cdef88ad
Add some tests to clarify the "must-change-password" behavior (#30693)
Follow  #30472:

When a user is created by command line `./gitea admin user create`:

Old behavior before #30472: the first user (admin or non-admin) doesn't
need to change password.

Revert to the old behavior before #30472
2024-04-27 12:23:37 +00:00
silverwind
74f0c84fa4
Enable more revive linter rules (#30608)
Noteable additions:

- `redefines-builtin-id` forbid variable names that shadow go builtins
- `empty-lines` remove unnecessary empty lines that `gofumpt` does not
remove for some reason
- `superfluous-else` eliminate more superfluous `else` branches

Rules are also sorted alphabetically and I cleaned up various parts of
`.golangci.yml`.
2024-04-22 11:48:42 +00:00
Kemal Zebari
0606284fcf
Add --skip-db option to dump command (#30613)
Attempts to resolve #28720.

---

Note that I am not a Gitea administrator so I don't normally use the
gitea CLI. Just saw this issue and wanted an opportunity to understand
how this subcommand works and see if I can add this feature :^)

I tested both with `--skip-db` and without and it appears to not add any
database-specific files to the generated archive i.e. I don't see a
`gitea-db.sql` or `gitea.db` file:
```console
$ TAGS="bindata sqlite sqlite_unlock_notify" make backend
Running go generate...
bindata for migration already up-to-date
bindata for options already up-to-date
bindata for public already up-to-date
bindata for templates already up-to-date
$ ./gitea dump --skip-db
2024/04/20 01:16:11 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/04/20 01:16:11 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/attachments
2024/04/20 01:16:11 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/avatars
2024/04/20 01:16:11 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-avatars
2024/04/20 01:16:11 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/lfs
2024/04/20 01:16:11 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/repo-archive
2024/04/20 01:16:11 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/packages
2024/04/20 01:16:11 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_log
2024/04/20 01:16:11 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/04/20 01:16:11 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /workspaces/gitea/data/actions_artifacts
2024/04/20 01:16:11 cmd/dump.go:172:runDump() [I] Dumping local repositories... /workspaces/gitea/data/gitea-repositories
2024/04/20 01:16:11 cmd/dump.go:195:runDump() [I] Skipping database
2024/04/20 01:16:11 cmd/dump.go:229:runDump() [I] Adding custom configuration file from /workspaces/gitea/custom/conf/app.ini
2024/04/20 01:16:11 cmd/dump.go:256:runDump() [I] Packing data directory.../workspaces/gitea/data
2024/04/20 01:16:11 cmd/dump.go:335:runDump() [I] Finish dumping in file /workspaces/gitea/gitea-dump-1713575771.zip
$ unzip /workspaces/gitea/gitea-dump-1713575771.zip -d example
Archive:  /workspaces/gitea/gitea-dump-1713575771.zip
. . .
$ ls example/
app.ini  custom  data  repos
$ ls example/data/
actions_artifacts  actions_log  avatars  home  indexers  jwt  queues  repo-archive  repo-avatars  tmp
```

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-21 23:32:12 +02:00
Cheng
9de443ced2
chore: use errors.New to replace fmt.Errorf with no parameters will much better (#30621)
use errors.New to replace fmt.Errorf with no parameters will much better
2024-04-21 19:44:03 +00:00
wxiaoguang
4c6e2da088
Improve "must-change-password" logic and document (#30472)
Unify the behaviors of "user create" and "user change-password".

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2024-04-14 17:22:14 +00:00
Lunny Xiao
263a716cb5
Performance optimization for git push (#30104)
Agit returned result should be from `ProcReceive` hook but not
`PostReceive` hook. Then for all non-agit pull requests, it will not
check the pull requests for every pushing `refs/pull/%d/head`.
2024-04-09 03:43:17 +00:00
wxiaoguang
83f83019ef
Clean up log messages (#30313)
`log.Xxx("%v")` is not ideal, this PR adds necessary context messages.
Remove some unnecessary logs.

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-07 19:17:06 +08:00
wxiaoguang
654cfd1dfb
Refactor "dump" sub-command (#30240)
Major changes:

* Move some functions like "addReader" / "isSubDir" /
"addRecursiveExclude" to a separate package, and add tests
* Clarify the filename&dump type logic and add tests
* Clarify the logger behavior and remove FIXME comments

Co-authored-by: Giteabot <teabot@gitea.io>
2024-04-03 02:16:46 +00:00
Lunny Xiao
e2277d07ca
Move some asymkey functions to service layer (#28894)
After the moving, all models will not depend on `util.Rename` so that I
can do next step refactoring.
2024-03-04 08:57:39 +00:00
6543
7fbdb60fc1
Start to migrate from util.OptionalBool to optional.Option[bool] (#29329)
just create transition helper and migrate two structs
2024-02-23 02:18:33 +00:00
wxiaoguang
e9b13732f3
Refactor cmd setup and remove deadcode (#29313)
* use `setup(ctx, c.Bool("debug"))` like all other callers
* `setting.RunMode = "dev"` is a no-op.
* `if _, err := os.Stat(setting.RepoRootPath); err != nil` could be
simplified
2024-02-22 07:04:30 +00:00
wxiaoguang
45c15387b2
Refactor JWT secret generating & decoding code (#29172)
Old code is not consistent for generating & decoding the JWT secrets.

Now, the callers only need to use 2 consistent functions:
NewJwtSecretWithBase64 and DecodeJwtSecretBase64

And remove a non-common function Base64FixedDecode from util.go
2024-02-16 15:18:30 +00:00
Lunny Xiao
96ad1d6340
Fix push to create with capitalize repo name (#29090)
Fix #29073
2024-02-08 13:25:09 +00:00
KN4CK3R
f8b471ace1
Unify user update methods (#28733)
Fixes #28660
Fixes an admin api bug related to `user.LoginSource`
Fixed `/user/emails` response not identical to GitHub api

This PR unifies the user update methods. The goal is to keep the logic
only at one place (having audit logs in mind). For example, do the
password checks only in one method not everywhere a password is updated.

After that PR is merged, the user creation should be next.
2024-02-04 13:29:09 +00:00
KN4CK3R
9bea276055
Add must-change-password cli parameter (#27626)
This PR adds a new `must-change-password` parameter to the
`change-password` cli command.
We already have the `must-change-password` command but it feels natural
to have this integrated into the `change-password` cli command.

---------

Co-authored-by: 6543 <6543@obermui.de>
2024-02-03 19:53:27 +02:00
Lunny Xiao
5f82ead13c
Simplify how git repositories are opened (#28937)
## Purpose
This is a refactor toward building an abstraction over managing git
repositories.
Afterwards, it does not matter anymore if they are stored on the local
disk or somewhere remote.

## What this PR changes
We used `git.OpenRepository` everywhere previously.
Now, we should split them into two distinct functions:

Firstly, there are temporary repositories which do not change:

```go
git.OpenRepository(ctx, diskPath)
```

Gitea managed repositories having a record in the database in the
`repository` table are moved into the new package `gitrepo`:

```go
gitrepo.OpenRepository(ctx, repo_model.Repo)
```

Why is `repo_model.Repository` the second parameter instead of file
path?
Because then we can easily adapt our repository storage strategy.
The repositories can be stored locally, however, they could just as well
be stored on a remote server.

## Further changes in other PRs
- A Git Command wrapper on package `gitrepo` could be created. i.e.
`NewCommand(ctx, repo_model.Repository, commands...)`. `git.RunOpts{Dir:
repo.RepoPath()}`, the directory should be empty before invoking this
method and it can be filled in the function only. #28940
- Remove the `RepoPath()`/`WikiPath()` functions to reduce the
possibility of mistakes.

---------

Co-authored-by: delvh <dev.lh@web.de>
2024-01-27 21:09:51 +01:00
Lunny Xiao
7711db0a71
Fix migrate storage bug (#28830) 2024-01-20 23:27:31 +08:00
Lunny Xiao
62f995203a
Move doctor package from modules to services (#28856) 2024-01-20 10:07:31 +08:00
yp05327
b60a7c3358
Return responseText instead of string in some functions (#28836)
Follow
https://github.com/go-gitea/gitea/pull/28796#issuecomment-1891727591
2024-01-19 10:45:23 +08:00
Lunny Xiao
70c4aad8e1
Move more functions to db.Find (#28419)
Following #28220

This PR move more functions to use `db.Find`.

---------

Co-authored-by: delvh <dev.lh@web.de>
2024-01-15 02:19:25 +00:00
wxiaoguang
2df7563f31
Recommend/convert to use case-sensitive collation for MySQL/MSSQL (#28662)
Mainly for MySQL/MSSQL.

It is important for Gitea to use case-sensitive database charset
collation. If the database is using a case-insensitive collation, Gitea
will show startup error/warning messages, and show the errors/warnings
on the admin panel's Self-Check page.

Make `gitea doctor convert` work for MySQL to convert the collations of
database & tables & columns.

* Fix #28131

## ⚠️ BREAKING ⚠️

It is not quite breaking, but it's highly recommended to convert the
database&table&column to a consistent and case-sensitive collation.
2024-01-10 11:03:23 +00:00
Lunny Xiao
4eb2a29910
Improve ObjectFormat interface (#28496)
The 4 functions are duplicated, especially as interface methods. I think
we just need to keep `MustID` the only one and remove other 3.

```
MustID(b []byte) ObjectID
MustIDFromString(s string) ObjectID
NewID(b []byte) (ObjectID, error)
NewIDFromString(s string) (ObjectID, error)
```

Introduced the new interfrace method `ComputeHash` which will replace
the interface `HasherInterface`. Now we don't need to keep two
interfaces.

Reintroduced `git.NewIDFromString` and `git.MustIDFromString`. The new
function will detect the hash length to decide which objectformat of it.
If it's 40, then it's SHA1. If it's 64, then it's SHA256. This will be
right if the commitID is a full one. So the parameter should be always a
full commit id.

@AdamMajer Please review.
2023-12-19 07:20:47 +00:00
wxiaoguang
047c69bd85
Improve CLI code and descriptions (#28482)
* Close #28444 
* Actually, it doesn't need to use that trick because it looks like it
is not necessary, no user really needs it
* Remove the hidden (legacy) "doctor" subcommand and update documents
* Fix "actions" usage


![image](https://github.com/go-gitea/gitea/assets/2114189/3c2b34a7-4f92-4a6c-96fd-9505e413d4ec)
2023-12-15 15:49:01 +00:00
Adam Majer
cbf923e87b
Abstract hash function usage (#28138)
Refactor Hash interfaces and centralize hash function. This will allow
easier introduction of different hash function later on.

This forms the "no-op" part of the SHA256 enablement patch.
2023-12-13 21:02:00 +00:00
wxiaoguang
f2a309e6c8
Improve doctor cli behavior (#28422)
1. Do not sort the "checks" slice again and again when "Register", it
just wastes CPU when the Gitea instance runs
2. If a check doesn't exist, tell the end user
3. Add some tests
2023-12-11 15:55:10 +00:00
Lunny Xiao
df1e7d0067
Use db.Find instead of writing methods for every object (#28084)
For those simple objects, it's unnecessary to write the find and count
methods again and again.
2023-11-24 03:49:41 +00:00
Anudeep Reddy
5380fd5d6d
fixed duplicate attachments on dump on windows (#28019)
Hi,

This PR fixes #27988. The use of `path.join`(which uses `/` as the file
separator) to construct paths and comparing them with paths constructed
using `filepath.join`(which uses platform specific file separator) is
the root cause of this issue.

The desired behavior is to ignore attachments when dumping data
directory. Due to the what's mentioned above, the function
`addRecursiveExclude` is not actually ignoring the attachments directory
and is being written to the archive. The attachment directory is again
added to the archive (with different file separator as mentioned in the
issue) causing a duplicate entry on windows.

The solution is to use `filepath.join` in `addResursiveExclude` to
construct `currentAbsPath`.
2023-11-14 02:15:45 +01:00
Lunny Xiao
1bf5527eac
Refactor Find Sources and fix bug when view a user who belongs to an unactive auth source (#27798)
The steps to reproduce it.

First, create a new oauth2 source.
Then, a user login with this oauth2 source.
Disable the oauth2 source.
Visit users -> settings -> security, 500 will be displayed.
This is because this page only load active Oauth2 sources but not all
Oauth2 sources.
2023-11-03 01:41:00 +00:00
JakobDev
76a85a4ce9
Final round of db.DefaultContext refactor (#27587)
Last part of #27065
2023-10-14 08:37:24 +00:00
JakobDev
ebe803e514
Penultimate round of db.DefaultContext refactor (#27414)
Part of #27065

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-10-11 04:24:07 +00:00
Lunny Xiao
673cf6af76
make writing main test easier (#27270)
This PR removed `unittest.MainTest` the second parameter
`TestOptions.GiteaRoot`. Now it detects the root directory by current
working directory.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-28 01:38:53 +00:00
Nanguan Lin
70f8d10fc5
move the gitea admin subcommands into separate files (#27307)
As title.
Probably it's better to put those sub cmd to different dirs. Will do
that in the future.
2023-09-27 12:25:38 +00:00
JakobDev
7047df36d4
Another round of db.DefaultContext refactor (#27103)
Part of #27065

---------

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-09-25 13:17:37 +00:00
JakobDev
c548dde205
More refactoring of db.DefaultContext (#27083)
Next step of #27065
2023-09-15 06:13:19 +00:00
JakobDev
76659b1114
Reduce usage of db.DefaultContext (#27073)
Part of #27065

This reduces the usage of `db.DefaultContext`. I think I've got enough
files for the first PR. When this is merged, I will continue working on
this.

Considering how many files this PR affect, I hope it won't take to long
to merge, so I don't end up in the merge conflict hell.

---------

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2023-09-14 17:09:32 +00:00
MomentQYC
598465eb43
fix: typo (#27009) 2023-09-11 09:30:18 +00:00