0
0
Fork 0
mirror of https://github.com/go-gitea/gitea synced 2024-11-15 06:21:23 +01:00
gitea/modules/setting
Jason Song 33cc5837a6
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
2024-08-09 10:10:30 +08:00
..
config Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
actions.go Support compression for Actions logs (#31761) 2024-08-09 10:10:30 +08:00
actions_test.go Restrict [actions].DEFAULT_ACTIONS_URL to only github or self (#25581) 2023-06-30 07:26:36 +00:00
admin.go Allow disabling authentication related user features (#31535) 2024-07-09 17:36:31 +00:00
api.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
asset_dynamic.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
asset_static.go Use a general approach to access custom/static/builtin assets (#24022) 2023-04-12 18:16:45 +08:00
attachment.go Update allowed attachment types (#29688) 2024-03-09 12:41:32 +00:00
attachment_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
cache.go Always enable caches (#28527) 2023-12-19 09:29:05 +00:00
camo.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
config.go Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
config_env.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
config_env_test.go Fix environment-to-ini inherited key bug (#27543) 2023-10-10 01:10:37 +08:00
config_provider.go Initial support for colorblindness-friendly themes (#30625) 2024-04-24 00:18:41 +08:00
config_provider_test.go Fix INI parsing for value with trailing slash (#26995) 2023-09-10 16:15:51 +00:00
cors.go Refactor CORS handler (#28587) 2023-12-25 20:13:18 +08:00
cron.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
cron_test.go Rewrite queue (#24505) 2023-05-08 19:49:59 +08:00
database.go Add slow SQL query warning (#27545) 2024-02-23 00:57:24 +00:00
database_sqlite.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
database_test.go Fix incorrect PostgreSQL connection string for Unix sockets (#28865) 2024-01-20 16:04:47 +00:00
federation.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
git.go Use [git.config] for reflog cleaning up (#24958) 2023-05-28 01:07:14 +00:00
git_test.go Use [git.config] for reflog cleaning up (#24958) 2023-05-28 01:07:14 +00:00
glob.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
global.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
highlight.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
i18n.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
incoming_email.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-04-29 08:47:56 +00:00
indexer.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
indexer_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
lfs.go Use general token signing secret (#29205) 2024-02-18 17:39:04 +00:00
lfs_test.go Display deprecated warning in admin panel pages as well as in the log file (#26094) 2023-07-26 03:53:37 +00:00
log.go Refactor "dump" sub-command (#30240) 2024-04-03 02:16:46 +00:00
log_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
mailer.go Add option to change mail from user display name (#31528) 2024-07-14 23:27:00 +02:00
mailer_test.go Remove unnecessary code (#24610) 2023-05-10 04:57:06 +00:00
markup.go Add .livemd as a markdown extension (#22730) 2023-04-26 11:22:54 -04:00
metrics.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
migrations.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mime_type_map.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
mirror.go Avoid polluting the config (#25345) 2023-06-18 16:10:44 +00:00
oauth2.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
oauth2_test.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
other.go Add [other].SHOW_FOOTER_POWERED_BY setting to hide Powered by (#30253) 2024-04-03 16:01:50 +00:00
packages.go Add signature support for the RPM module (#27069) 2024-08-06 09:03:33 -04:00
packages_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
path.go Do not allow different storage configurations to point to the same directory (#30169) 2024-03-31 03:03:24 +00:00
path_test.go Refactor path & config system (#25330) 2023-06-21 13:50:26 +08:00
picture.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
project.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
proxy.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00
queue.go Increase queue length (#27555) 2023-10-10 18:47:49 +08:00
repository.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
repository_archive.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
repository_archive_test.go Fix all possible setting error related storages and added some tests (#23911) 2023-06-14 11:42:38 +08:00
security.go Remember login for a month by default (#30150) 2024-03-28 03:13:42 +00:00
server.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
service.go Add reverseproxy auth for API back with default disabled (#26703) 2023-09-07 08:31:46 +00:00
service_test.go Fix allowed user types setting problem (#26200) 2023-07-28 12:15:39 -04:00
session.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
setting.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
setting_test.go Implement FSFE REUSE for golang files (#21840) 2022-11-27 18:20:29 +00:00
ssh.go Expanded minimum RSA Keylength to 3072 (#26604) 2023-08-28 00:53:16 +00:00
storage.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
storage_test.go Azure blob storage support (#30995) 2024-05-30 07:33:50 +00:00
task.go handle deprecated settings (#22992) 2023-02-20 16:18:26 -06:00
time.go Enable more revive linter rules (#30608) 2024-04-22 11:48:42 +00:00
ui.go Add MAX_ROWS option for CSV rendering (#30268) 2024-06-06 16:06:59 +08:00
webhook.go Refactor the setting to make unit test easier (#22405) 2023-02-20 00:12:01 +08:00