MirrorHub/gitea
0
0
Fork 0
mirror of https://github.com/go-gitea/gitea synced 2024-12-27 12:24:29 +01:00
Code Issues Releases Wiki Activity
11553 commits 17 branches 214 tags 773 MiB
Commit graph

31 commits

Author SHA1 Message Date
zeripath
ba526ceffe
Multiple Queue improvements: LevelDB Wait on empty, shutdown empty shadow level queue, reduce goroutines etc (#15693) 
* move shutdownfns, terminatefns and hammerfns out of separate goroutines

Coalesce the shutdownfns etc into a list of functions that get run at shutdown
rather then have them run at goroutines blocked on selects.

This may help reduce the background select/poll load in certain
configurations.

* The LevelDB queues can actually wait on empty instead of polling

Slight refactor to cause leveldb queues to wait on empty instead of polling.

* Shutdown the shadow level queue once it is empty

* Remove bytefifo additional goroutine for readToChan as it can just be run in run

* Remove additional removeWorkers goroutine for workers

* Simplify the AtShutdown and AtTerminate functions and add Channel Flusher

* Add shutdown flusher to CUQ

* move persistable channel shutdown stuff to Shutdown Fn

* Ensure that UPCQ has the correct config

* handle shutdown during the flushing

* reduce risk of race between zeroBoost and addWorkers

* prevent double shutdown

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2021-05-15 16:22:26 +02:00
zeripath
aa65a607e4
Queue manager FlushAll can loop rapidly - add delay (#15733) 
* Queue manager FlushAll can loop rapidly - add delay

Add delay within FlushAll to prevent rapid loop when workers are busy

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per lunny

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: 6543 <6543@obermui.de>
 2021-05-12 00:22:08 +01:00
zeripath
e22ee468cf
Exponential Backoff for ByteFIFO (#15724) 
This PR is another in the vein of queue improvements. It suggests an
exponential backoff for bytefifo queues to reduce the load from queue
polling. This will mostly be useful for redis queues.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
 2021-05-08 17:29:47 +01:00
zeripath
d11b9fbcce
Prevent race in TestChannelQueue_Batch (#15703) 
There is a potential race in TestChannelQueue_Batch due to boost workers starting up

This PR simply removes the boosts from this test.

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2021-05-03 17:16:59 +01:00
zeripath
0590176a23
Only use boost workers for leveldb shadow queues (#15696) 
* The leveldb shadow queue of a persistable channel queue should always start with 0
workers and just use boost to add additional workers if necessary.

* create a zero boost so that if there are no workers in a pool - boost to start the workers

* actually set timeout appropriately on boosted workers

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2021-05-02 08:22:30 +01:00
zeripath
84f5a0bc62
Always set the merge base used to merge the commit (#15352) 
The issue is that the TestPatch will reset the PR MergeBase - and it is possible for TestPatch to update the MergeBase whilst a merge is ongoing. The ensuing merge will then complete but it doesn't re-set the MergeBase it used to merge the PR.

Fixes the intermittent error in git test.

Signed-off-by: Andrew Thornton art27@cantab.net
 2021-04-10 09:27:29 +01:00
6543
9c4601bdf8
Code Formats, Nits & Unused Func/Var deletions (#15286) 
* _ to unused func options

* rm useless brakets

* rm trifial non used models functions

* rm dead code

* rm dead global vars

* fix routers/api/v1/repo/issue.go

* dont overload import module
 2021-04-09 09:40:34 +02:00
zeripath
f0e15250b9
Migrate to use jsoniter instead of encoding/json (#14841) 
* Migrate to use jsoniter

* fix tests

* update gitea.com/go-chi/binding

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
 2021-03-01 22:08:10 +01:00
zeripath
b3c2e23cbb
Prevent race in PersistableChannelUniqueQueue.Has (#14651) 
There is potentially a race with a slow starting internal
queue causing a NPE if Has is checked before the internal
queue has been setup.

This PR adds a lock on the Has() fn.

Fix #14311

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2021-02-13 20:02:09 +01:00
6543
ac97ea573c
[Vendor] Update go-redis to v8.5.0 (#13749) 
* Update go-redis to v8.4.0

* github.com/go-redis/redis/v8  v8.4.0 -> v8.5.0

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* TODO

* Use the Queue termination channel as the default context for pushes

Signed-off-by: Andrew Thornton <art27@cantab.net>

* missed one

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
 2021-02-10 21:28:32 +00:00
zeripath
c8f7a6b774
Slightly simplify the queue settings code to help reduce the risk of problems (#12976) 
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
 2020-10-15 17:40:03 -04:00
zeripath
5cfc1f573f
Fix the issue reported on #12385 (#12969) 
Missed setting ConnectionString on queuesettings

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2020-09-28 19:00:54 -04:00
zeripath
7f8e3192cd
Allow common redis and leveldb connections (#12385) 
* Allow common redis and leveldb connections

Prevents multiple reopening of redis and leveldb connections to the same
place by sharing connections.

Further allows for more configurable redis connection type using the
redisURI and a leveldbURI scheme.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add unit-test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* as per @lunny

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update modules/cache/cache_redis.go

* Update modules/queue/queue_disk.go

* Update modules/cache/cache_redis.go

* Update modules/cache/cache_redis.go

* Update modules/queue/unique_queue_disk.go

* Update modules/queue/queue_disk.go

* Update modules/queue/unique_queue_disk.go

* Update modules/session/redis.go

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
 2020-09-28 00:09:46 +03:00
Lunny Xiao
91e7ad569a
Add queue for code indexer (#10332) 
* Add queue for code indexer

* Fix lint

* Fix test

* Fix lint

* Fix bug

* Fix bug

* Fix lint

* Add noqueue

* Fix tests

* Rename noqueue to immediate
 2020-09-07 23:05:08 +08:00
zeripath
69b8d7ba19
use assignment in tests (#12734) 
Signed-off-by: Andrew Thornton <art27@cantab.net>
 2020-09-06 01:50:57 +03:00
zeripath
74bd9691c6
Re-attempt to delete temporary upload if the file is locked by another process (#12447) 
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.

Fix #12339

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
 2020-08-11 21:05:34 +01:00
zeripath
217647f331
Multiple small admin dashboard fixes (#12153) 
* Remove spurious spacing between Maintenance Operations and its table on dashboard
* Prevent (EXTRA string) comments in Task headers
* Redirect tasks started from monitor page back to monitor
* Fix #12107 - redirects from process cancel should use AppSubUrl
* When wrapping queues set the name correctly

Signed-off-by: Andrew Thornton <art27@cantab.net>
 2020-07-05 22:38:03 +03:00
zeripath
c58bc4bf80
Prevent timer leaks in Workerpool and others (#11333) 
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.

Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.

Unfortunately in `Workerpool` these were checked the wrong way round.

However, there were a few other places that were not being checked.

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
 2020-05-08 16:46:05 +01:00
Lunny Xiao
fcc8cdd446
Improve config logging when WrappedQueue times out (#11174) 
Before
```sh
Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg []byte{0x7b, 0x22, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x36, 0x33, 0x37, 0x39, 0x22, 0x2c, 0x22, 0x42, 0x61, 0x74, 0x63, 0x68, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x22, 0x3a, 0x32, 0x30, 0x2c, 0x22, 0x42, 0x6c, 0x6f, 0x63, 0x6b, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x31, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x2c, 0x22, 0x42, 0x6f, 0x6f, 0x73, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x33, 0x30, 0x30, 0x30, 0x30, 0x30
......
```

After
```sh
Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg "{\"Addresses\":\"127.0.0.1:6379\",\"BatchLength\":20,\"BlockTimeout\":1000000000,\"BoostTimeout\":300000000000,\"BoostWorkers\":5,\"DBIndex\":0,\"DataDir\":\".../data/queues/mail\",\"MaxWorkers\":10,\"Name\":\"mail\",\"Network\":\"\",\"Password\":\"\",\"QueueLength\":20,\"QueueName\":\"mail_queue\",\"SetName\":\"\",\"Workers\":1}" in
```
 2020-04-22 13:38:40 +01:00
zeripath
e83daf77ad
Avoid logging []byte in queue failures - convert to string first (#10865) 
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
 2020-03-29 15:12:15 +08:00
Lunny Xiao
cf7ece6245
Fix queue log param (#10733) 2020-03-16 16:59:21 +08:00
zeripath
88986746d5
Fix Workerpool deadlock (#10283) 
* Prevent deadlock on boost

* Force a boost in testchannelqueue
 2020-02-15 18:44:58 +00:00
Lunny Xiao
3d69bbd58f
Fix queue pop error and stat empty repository error (#10248) 
* Fix queue pop error and stat empty repository error

* Fix error
 2020-02-12 18:12:27 +08:00
zeripath
2c903383b5
Add Unique Queue infrastructure and move TestPullRequests to this (#9856) 
* Upgrade levelqueue to version 0.2.0

This adds functionality for Unique Queues

* Add UniqueQueue interface and functions to create them

* Add UniqueQueue implementations

* Move TestPullRequests over to use UniqueQueue

* Reduce code duplication

* Add bytefifos

* Ensure invalid types are logged

* Fix close race in PersistableChannelQueue Shutdown
 2020-02-02 23:19:58 +00:00
zeripath
9b9dd19d7d
Fix broken FlushAll (#10101) 
* go function contexting is not what you expect

* Apply suggestions from code review

Co-Authored-By: Lauris BH <lauris@nix.lv>

Co-authored-by: Lauris BH <lauris@nix.lv>
 2020-02-01 23:43:50 +00:00
Lunny Xiao
eac5142ac7
Fix leveldb test race (#10054) 
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
 2020-01-30 11:09:39 -05:00
zeripath
c01221e70f
Queue: Make WorkerPools and Queues flushable (#10001) 
* Make WorkerPools and Queues flushable

Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Change to for loop

* Add IsEmpty and begin just making the queues composed WorkerPools

* subsume workerpool into the queues and create a flushable interface

* Add manager command

* Move flushall to queue.Manager and add to testlogger

* As per @guillep2k

* as per @guillep2k

* Just make queues all implement flushable and clean up the wrapped queue flushes

* cope with no timeout

Co-authored-by: Lauris BH <lauris@nix.lv>
 2020-01-28 20:01:06 -05:00
zeripath
8cb9b2f016
Update queue_redis.go (#9910) 
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
 2020-01-21 12:50:27 +00:00
zeripath
960ac36099
Remove unused lock (#9709) 2020-01-11 19:06:35 +00:00
zeripath
edd31770aa
Missed q.lock.Unlock() will cause panic (#9705) 2020-01-11 17:05:07 +00:00
zeripath
62eb1b0f25 Graceful Queues: Issue Indexing and Tasks (#9363) 
* Queue: Add generic graceful queues with settings

* Queue & Setting: Add worker pool implementation

* Queue: Add worker settings

* Queue: Make resizing worker pools

* Queue: Add name variable to queues

* Queue: Add monitoring

* Queue: Improve logging

* Issues: Gracefulise the issues indexer

Remove the old now unused specific queues

* Task: Move to generic queue and gracefulise

* Issues: Standardise the issues indexer queue settings

* Fix test

* Queue: Allow Redis to connect to unix

* Prevent deadlock during early shutdown of issue indexer

* Add MaxWorker settings to queues

* Merge branch 'master' into graceful-queues

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update modules/queue/queue_disk.go

* Update modules/queue/queue_disk_channel.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Rename queue.Description to queue.ManagedQueue as per @guillep2k

* Cancel pool workers when removed

* Remove dependency on queue from setting

* Update modules/queue/queue_redis.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* As per @guillep2k add mutex locks on shutdown/terminate

* move unlocking out of setInternal

* Add warning if number of workers < 0

* Small changes as per @guillep2k

* No redis host specified not found

* Clean up documentation for queues

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

* Update modules/indexer/issues/indexer_test.go

* Ensure that persistable channel queue is added to manager

* Rename QUEUE_NAME REDIS_QUEUE_NAME

* Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"

This reverts commit 1f83b4fc9b.

Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
 2020-01-07 12:23:09 +01:00