0
0
Fork 0
mirror of https://github.com/matrix-construct/construct synced 2024-10-31 19:08:59 +01:00

Compare commits

...

201 commits

Author SHA1 Message Date
Jason Volk
0624b69246 modules/web_hook: Add post-push handler for reacting to push events. 2023-05-01 12:45:37 -07:00
Jason Volk
e5ae70c521 modules/web_hook: Add requests for workflows and dispatching. 2023-05-01 12:45:37 -07:00
Jason Volk
759871ce8e ircd::versions: Minor cleanup instance_list related. 2023-05-01 12:39:06 -07:00
Jason Volk
3c9a4f8c57 ircd:Ⓜ️ Assertions for non-empty id constructions. 2023-05-01 10:48:07 -07:00
Jason Volk
f6b3b8b758 ircd:Ⓜ️:event::append: Add branch for bundled relations; add m.replace bundle. 2023-05-01 10:48:07 -07:00
Jason Volk
4fe85805bf ircd:Ⓜ️:relates: Add conf item for non-type coarse control of sorting field. 2023-05-01 10:48:07 -07:00
Jason Volk
b6cb1180f7 ircd:Ⓜ️:user::keys: Basis for verification cross-signatures between users. 2023-05-01 10:48:07 -07:00
Jason Volk
741304271e ircd:Ⓜ️:event::append: Optimize opts, eliminate double-indirection; various reorg. 2023-05-01 10:48:07 -07:00
Jason Volk
cd2b92b8fe modules/client/keys/claim: Restore when_any() loop to smooth remote handling. 2023-04-29 13:57:39 -07:00
Jason Volk
feb4ac1fd3 ircd:Ⓜ️:event::append: Move static objects into class. 2023-04-29 13:57:39 -07:00
Jason Volk
d4ba215a3b modules/client/user: Relax matching user_id in URL to requestor for bridges. 2023-04-29 13:57:39 -07:00
Jason Volk
8a705f77a9 ircd::db: Fixes for RocksDB 8.1.1 interface changes. 2023-04-29 13:57:37 -07:00
Jason Volk
f86fddc3a3 configure: Error on -Wabstract-final-class (clang) 2023-04-28 13:39:35 -07:00
Jason Volk
8eb10c5d1a modules/client/rooms/event: Replace raw event response with client format. 2023-04-28 13:04:06 -07:00
Jason Volk
a7a3275817 modules/client/keys/claim: Handle local users without loopbacking. 2023-04-27 20:41:18 -07:00
Jason Volk
82feeb5274 ircd:Ⓜ️:user::keys: Centralize key claiming for user device and algorithm. 2023-04-27 20:41:18 -07:00
Jason Volk
8af781c5d7 modules/client/keys/query: Handle local users without loopbacking. 2023-04-27 20:41:18 -07:00
Jason Volk
29f3ddfc3e ircd:Ⓜ️:user::devices: Upgrade to polymorphic closure_bool. 2023-04-27 20:41:18 -07:00
Jason Volk
0b888b662a ircd:Ⓜ️:user::keys: Further abstract callstack through type string argument. 2023-04-27 20:41:18 -07:00
Jason Volk
bbed809975 ircd:Ⓜ️:user::keys: Consolidate cross signing keys update. 2023-04-27 20:41:18 -07:00
Jason Volk
f9aeae5516 ircd:Ⓜ️ Add user_signing_key to signing_key_update for completeness. 2023-04-27 20:41:18 -07:00
Jason Volk
9301980f9d ircd:Ⓜ️:user::keys: Implement m.signing_key_update broadcast interface. 2023-04-27 20:41:18 -07:00
Jason Volk
ca80d66e85 ircd:Ⓜ️:user::devices: Complete the m.device_list_update broadcast w/ keys; reinterface. 2023-04-27 20:41:18 -07:00
Jason Volk
e7ad503f8c modules/federation/user_keys_claim: Implement claimancy; check user exists; minor cleanup. 2023-04-27 17:18:17 -07:00
Jason Volk
f70d837258 modules/client/keys/claim: Enforce remote authority over results; relax log level. 2023-04-27 17:18:17 -07:00
Jason Volk
6f5121dc6a modules/client/keys/query: Refactor receive handling for properly composed response. 2023-04-27 17:18:17 -07:00
Jason Volk
91a8fcbe43 ircd::ctx::future: Add what() exception peeking convenience to interface. 2023-04-27 11:36:31 -07:00
Jason Volk
b4b26484ec modules/console: Add wildcard for all devices to user devices update cmd. 2023-04-26 19:27:04 -07:00
Jason Volk
faf796a56b ircd:Ⓜ️:pretty: Increase widths; improve formatting for stateline. 2023-04-26 19:27:04 -07:00
Jason Volk
48e0755a79 modules/web_hook: Search for reactions by contents rather than endswith. 2023-04-26 19:27:04 -07:00
Jason Volk
5f5be52fa9 ircd::net::dns::cache: Fix case sensitivities on several key mappings. (956aa7682f) 2023-04-26 19:27:04 -07:00
Jason Volk
5f0b98e5d1 modules/console: Add well_known matrix client diagnostic. 2023-04-26 19:27:04 -07:00
Jason Volk
62a1c850cc ircd::rest: Add get overload w/ default opts. 2023-04-26 19:27:04 -07:00
Jason Volk
e874e28473 modules/console: Add raw override argument to fed user keys query and device cmds. 2023-04-26 14:50:26 -07:00
Jason Volk
55f332c821 modules: Replace any http::CREATED with http::OK. (5a06c006) 2023-04-26 14:50:15 -07:00
Jason Volk
44a4f33e45 modules/federation/user_devices: Add signed keys results to response. 2023-04-26 14:50:14 -07:00
Jason Volk
04025af961 ircd:Ⓜ️:user::keys: Abstract key appendix; attach signatures to all keys. 2023-04-26 14:50:14 -07:00
Jason Volk
d1b0722169 ircd:Ⓜ️:user::keys: Move amalgamated keys and sigs output into central interface. 2023-04-26 14:50:14 -07:00
Jason Volk
0943cfd69c modules/federation/user_keys_query: Merge additional signatures into master key object. 2023-04-24 16:37:52 -07:00
Jason Volk
8b0cf48578 modules/client/keys/signatures: Re-schematize signatures uploaded for keys and devices. 2023-04-24 16:37:49 -07:00
Jason Volk
8eb4de920c modules: Rename cross_signing types out of device namespace. 2023-04-24 14:21:15 -07:00
Giovanni Bottaro
5a06c0066e modules/client/user/filter: Set the HTTP response status code according to Matrix spec 2023-04-24 14:02:24 -07:00
Jason Volk
4ebc9fefaf modules/client/rooms/relations: Condition path param decode on non-empty args. 2023-04-24 14:02:24 -07:00
Jason Volk
5f3398bf52 github/workflows/docker: Add manual dispatch arguments. 2023-04-24 14:02:24 -07:00
Jason Volk
dabc8b4304 modules/federation/user_keys_query: Cross signatures. 2023-04-22 22:04:59 -07:00
Jason Volk
d80f29b65a ircd::json::stack::member: Conversion constructions from other member categories. 2023-04-22 21:44:55 -07:00
Jason Volk
6c3420afbc modules/client/sync/device_lists: Fix structure; indicate changes to own device. 2023-04-22 00:21:29 -07:00
Jason Volk
6072229dcc modules/client/sync: Stub device_unused_fallback_key_types. 2023-04-22 00:21:29 -07:00
Jason Volk
956aa7682f ircd::net::dns::resolver: Fix case sensitivity of the tag host. 2023-04-22 00:21:29 -07:00
Jason Volk
238cc10489 ircd:Ⓜ️:rooms: Implement matrix-org/matrix-spec-proposals#3827 2023-04-21 21:40:50 -07:00
Jason Volk
1a032b28b7 ircd:Ⓜ️:room: Add boolean query for room type. 2023-04-21 21:40:50 -07:00
Jason Volk
f08b9e85cd ircd:Ⓜ️:createroom: Bump the default version. 2023-04-21 21:40:50 -07:00
Jason Volk
9ce44aadd5 modules/client/devices: Add de facto user_id to response. 2023-04-21 17:45:57 -07:00
Jason Volk
78d6e4ce03 modules/client/capabilities: List additional capabilities. 2023-04-21 17:42:27 -07:00
Jason Volk
882629ab53 modules/federation/user_devices: Fix property name; additional for the loopback. 2023-04-21 17:42:27 -07:00
Jason Volk
8ade7c814e modules/client/room_keys: Add missing count and etag to responses. 2023-04-20 18:22:02 -07:00
Jason Volk
4d5d99ab2c modules/client/room_keys: Unify version and keys into single module. 2023-04-20 16:51:09 -07:00
Jason Volk
502a03a8cf modules/console: Add redact state convenience; fix missing newline. 2023-04-20 13:34:47 -07:00
Jason Volk
1e42fa16a7 modules/client/room_keys/keys: Add missing DELETE for all rooms and sessions. 2023-04-20 13:34:47 -07:00
Jason Volk
0a151a3a90 modules/client/room_keys/keys: Fix sessions request and response structure.
closes #142
2023-04-20 13:34:44 -07:00
Jason Volk
fa46231e3a modules/client/room_keys/keys: Add util to split state_key parts. 2023-04-20 13:34:23 -07:00
Jason Volk
336026dde6 modules/client/keys/signatures/upload: Fix inhibition to interactive verification.
closes #162
2023-04-20 13:33:08 -07:00
Jason Volk
d5df04a183 modules/federation/user_keys_query: Add missing user_signing_keys on the loopback. 2023-04-20 00:54:01 -07:00
Jason Volk
080748f0af modules/client/keys/signatures/upload: Fix POST interpretation. 2023-04-19 15:01:24 -07:00
Jason Volk
321ea3d641 modules/client/room_keys/version: Implement missing PUT replacement functionality. 2023-04-19 15:01:24 -07:00
Jason Volk
d41926bc6f modules/client/versions: Add more version. 2023-04-19 14:56:11 -07:00
Jason Volk
58476f59c3 modules/console: Overload parameters of room events cmd to filter by type. 2023-04-19 14:56:11 -07:00
Jason Volk
8dc50db2dc modules/console: Add user devices delete cmd. 2023-04-19 14:56:11 -07:00
Jason Volk
2122412c6d modules/console: Split user tokens clear into separate cmd. 2023-04-19 14:56:11 -07:00
Jason Volk
15cb6bfdca ircd:Ⓜ️🆔 de facto device id. 2023-04-19 14:56:11 -07:00
Jason Volk
fa0365fa31 ircd:Ⓜ️ Add convenience interface for querying m.replace relation. 2023-04-18 20:11:29 -07:00
Jason Volk
02e09728a5 ircd::rest: Add option to supply temporary/headers buffer in lieu of allocating. 2023-04-18 20:11:29 -07:00
Jason Volk
dc13381822 ircd::json: Reduce replace() overloads to single linked procedure. 2023-04-18 20:11:29 -07:00
Jason Volk
22b9cf515c ircd::json: Add overload to insert multiple members. 2023-04-18 19:43:33 -07:00
Jason Volk
a10992b813 ircd::json: Split back into tool.h 2023-04-18 19:43:33 -07:00
Jason Volk
c77df219b5 ircd::json: Optimize json::type() ABI.
ircd::json: Fix indentation; minor cleanup.
2023-04-17 19:31:44 -07:00
Jason Volk
55a73624d2 ircd:🆑 MESA_GLSL_CACHE_DISABLE is deprecated. 2023-04-16 19:33:31 -07:00
Jason Volk
cab2b4c822 github/workflows/docker: Tweak concurrency group; prime images fail fast; comments/cleanup. 2023-04-16 15:42:20 -07:00
Jason Volk
79f5e4fd8d modules/web_hook: Add hook/shot request suite. 2023-04-16 15:42:20 -07:00
Jason Volk
4c06793980 ircd::db: Make SeekForPrev() reachable via internal callstack; remove cruft. 2023-04-16 15:42:20 -07:00
Jason Volk
ef27ae50dc ircd::db: Fix workaround for reverse prefix iteration. 2023-04-16 15:42:20 -07:00
Jason Volk
6e7d63ce6d ircd::db: Assert valid iterator prior to all relevant operations. 2023-04-16 15:42:18 -07:00
Jason Volk
73a3a4cb35 docker: Add --no-cache on built stage to force rebuild on runners. 2023-04-13 22:22:11 -07:00
Jason Volk
b40c21545a modules/m_command: Add watch support for server-side command line. 2023-04-13 22:21:38 -07:00
Jason Volk
b4e75dfdf0 modules/console: Allow wildcard for all users in user read cmd. 2023-04-13 22:21:38 -07:00
Jason Volk
ba6030f4ce modules/console: Improve formatting for feds head output when missing local event. 2023-04-13 22:21:19 -07:00
Jason Volk
dd6d17433e ircd:🆑 Stop propagation of platform and device initialization errors. 2023-04-10 18:00:14 -07:00
Jason Volk
7904fa0563 modules/webhook: Render markdown to html for dependabot alerts. 2023-04-10 01:08:11 -07:00
Jason Volk
1f35421cec docker/ubuntu: Disable additional rocksdb built accessories. 2023-04-10 01:08:11 -07:00
Jason Volk
af783c8dfe docker: Add build arg for extra packages; move opencl/libclc. 2023-04-10 01:08:11 -07:00
Jason Volk
e30c77283c ircd/Makefile: Platform-conditional compilation for amdgcn/r600. 2023-04-10 01:08:11 -07:00
Jason Volk
8c629f20a6 docker: Workaround buildkit ignoring --cpuset-cpus. 2023-04-10 01:08:11 -07:00
Jason Volk
b2d3ae4d0d Add docker image workflows. 2023-04-10 01:08:11 -07:00
Jason Volk
c8ff5bbc37 modules/web_hook: Aggregate status updates on cancel to reduce edit load. 2023-04-09 21:48:34 -07:00
Jason Volk
715f21564e configure: Add conditionals for hardware platform / host_cpu. 2023-04-09 21:48:34 -07:00
Jason Volk
50481585b0 Add *.save to gitignore. 2023-04-09 21:48:34 -07:00
Jason Volk
5d40121d53 docker/ubuntu: Remove cruft. (792973397) 2023-04-09 21:48:34 -07:00
Jason Volk
b906650946 modules/console: Add redact last cmd to supplement the feature in E!ement. 2023-04-09 21:48:34 -07:00
Jason Volk
11d0ab9eaf modules/web_hook: Add stream for alt text; add alt msgs for workflow errors. 2023-04-09 21:48:34 -07:00
Jason Volk
24cb1b804f docker: Add appropriate libclc-dev for the specific LLVM epoch. (ubuntu/clang) 2023-04-09 21:48:34 -07:00
Jason Volk
13d21ca83c docker: Reduce extra_packages_devX to single build argument. 2023-04-09 21:48:33 -07:00
Jason Volk
7501f983a6 Disable LTO for matrix.so due to issue w/ ld.gold for llvm-14. 2023-04-09 21:48:33 -07:00
Jason Volk
193d87ef76 configure: Prevent unsupported march=native on clang-14 aarch64. 2023-04-09 21:48:33 -07:00
Jason Volk
922db35dd5 configure: Fix emitting option for CET on aarch64. (gcc) 2023-04-09 21:48:33 -07:00
Jason Volk
02dd645748 configure: Use $host_cpu rather than $target_cpu; cleanup quotings. 2023-04-09 21:48:33 -07:00
Jason Volk
a67227f28f docker/ubuntu: Append the config.log after a failed configure; quiet rocksdb untar. 2023-04-09 21:48:33 -07:00
Jason Volk
b21558fada modules/web_hook: Add run-deletion to control panel. 2023-04-09 21:48:33 -07:00
Jason Volk
97692b7330 modules/web_hook: Aggregate job-queued updates until first progress; various. 2023-04-09 21:48:33 -07:00
Jason Volk
fc91ace4f2 modules/web_hook: Make dockerhub handler output conditional; mute for now. 2023-04-09 21:48:33 -07:00
Jason Volk
219571e69c modules/web_hook: Add control panel on the status bar. 2023-04-09 21:48:33 -07:00
Jason Volk
d2c47b0d9e modules/web_hook: Add github request stack. 2023-04-09 21:48:33 -07:00
Jason Volk
8cbeb98b59 modules/web_hook: Add github workflow handlers. 2023-04-09 21:48:33 -07:00
Jason Volk
25ed05429b ircd::db: Inline domain::const_iterator seek(string_view). 2023-04-09 21:48:33 -07:00
Jason Volk
4518cf104a modules/console: Fix db files cmd table header alignment (c807550ca7) 2023-04-09 21:48:33 -07:00
Jason Volk
c02b3d845f ircd::rest: Add web request interface which isn't orchestrally complicated. 2023-04-09 21:48:33 -07:00
Jason Volk
32d4d44662 ircd::buffer: Add unique_buffer ctor for mutable to const move conversion. 2023-04-04 15:49:24 -07:00
Jason Volk
b260bd85a7 ircd::rfc3986::uri: Add path+query extractor convenience. 2023-04-04 15:49:24 -07:00
Jason Volk
d4d8063dee ircd::http: Allow empty header values. 2023-04-04 15:49:23 -07:00
Jason Volk
7be2582714 ircd::util::bitset: Make set() invertible to false. 2023-04-03 11:47:32 -07:00
Jason Volk
6264071248 ircd::simd: Remove unused attribute (clang) (38e77c64c6). 2023-04-03 11:47:30 -07:00
Jason Volk
602833d0ef modules/web_hook: Add dependabot alert handler. 2023-04-01 19:15:11 -07:00
Jason Volk
1b933f8b8f modules/web_hook: Abstract the push event id finder. 2023-04-01 12:42:26 -07:00
Jason Volk
2db42a7e70 appveyor: Isolate appveyor to specific branch (disable appveyor). 2023-04-01 12:42:24 -07:00
Jason Volk
7929733978 docker: Refactor into build matrix. [ci skip] 2023-03-30 21:02:37 -07:00
Jason Volk
1b67a12a5f configure: Refactor all --disable options to --enable w/ proper enablevar. 2023-03-29 12:19:17 -07:00
Jason Volk
38c9a77a8f configure: Improve generic mode and machine tuning related options. 2023-03-28 22:09:11 -07:00
Jason Volk
8f4fe45034 configure: Fix missing result message for enable every warning. 2023-03-28 21:02:50 -07:00
Jason Volk
c7a89fcd89 tools/m4: Add size check comparison macro. 2023-03-28 21:02:04 -07:00
Jason Volk
5303e170d3 modules/console: Add remote arg, use params for join cmd. 2023-03-26 20:55:09 -07:00
Jason Volk
ffecc8bd22 ircd::info: Fix leaf for avx2/avx512f; additional tsc related. 2023-03-26 18:23:30 -07:00
Jason Volk
d4b3a0db66 ircd::b64: Optimize outer loop vectorized codegen. (clang/AVX512) 2023-03-26 17:47:24 -07:00
Jason Volk
38e77c64c6 ircd::simd: Fix 'return type cannot be vectorized' inhibition (clang). 2023-03-26 16:14:06 -07:00
Jason Volk
b1c2576c20 ircd::util: Add sign cast convenience suite. 2023-03-25 18:37:56 -07:00
Jason Volk
ef930635de tools/boostrap: Add required mp11. (boost 1.82.0.beta1) 2023-03-24 22:47:01 -07:00
Jason Volk
02c862e4bc ircd::db: Add define for versions w/ buggy MultiRead. 2023-03-24 22:47:01 -07:00
Jason Volk
674b6489a4 configure: Suppress deprecated/uninitialized warnings in release mode. 2023-03-24 21:23:56 -07:00
Jason Volk
999ef88b7f docker/ubuntu/22.04: Fetch RocksDB source as needed without persisting into image.
Thanks to Giovanni Bottaro.
2023-03-24 18:36:02 -07:00
Jason Volk
7046010ff9 docker/ubuntu/22.04: Delete unused librocksdb.a build artifact from base image.
Thanks to Giovanni Bottaro.
2023-03-24 17:32:08 -07:00
Jason Volk
92da7381ce docker/ubuntu/22.04: Use exact boost package names to prevent regex matching dev packages.
Thanks to Giovanni Bottaro.
2023-03-24 16:44:24 -07:00
Jason Volk
4ecf1ef037 Update README; link to wiki for docker grid. [ci skip] 2023-03-23 23:43:52 -07:00
Jason Volk
3ec562ce0b modules/console: Display m.* state in room overview cmd rather than m.room.* 2023-03-23 18:53:52 -07:00
Jason Volk
a8c16e0fd0 ircd::json: Fix mismatched serialized size propagating through value copy. 2023-03-23 18:51:33 -07:00
Jason Volk
66629046e1 ircd:Ⓜ️:vm: Add reverse reflection of phase enum. 2023-03-23 03:29:57 -07:00
Jason Volk
4339639732 modules/console: Add mass-query for room_id/alias argument to key get cmd. 2023-03-22 22:57:18 -07:00
Jason Volk
95d5361c20 ircd:Ⓜ️:keys: Add query overload for supplied buffer; add conf item defaults. 2023-03-22 22:57:18 -07:00
Jason Volk
95274dd0cd ircd::asio: Fix missing condition for newer boost systems w/o liburing. 2023-03-21 19:27:23 -07:00
Jason Volk
bffa445d37 ircd::server: Add bounds for wait all w/ conf items; promote warning; rename item. 2023-03-21 10:39:24 -07:00
Jason Volk
1b8ad3e160 ircd::resource: Optimize response w/ iov for fused head and content. 2023-03-20 20:56:56 -07:00
Jason Volk
04acaabf91 ircd::resource: Minor cleanup response overload stack for followability. 2023-03-20 14:42:10 -07:00
Jason Volk
575211d37e ircd::net: Add close option to skip shutdown syscall prior to close. 2023-03-20 14:01:42 -07:00
Jason Volk
6d73a65867 ircd::net: Add translate/reflect for disconnect type. 2023-03-20 14:01:42 -07:00
Jason Volk
b3832541ff ircd::net: Allow designated and implicit constructions for close_opts. 2023-03-20 14:01:42 -07:00
Jason Volk
14f55f6110 ircd::ios: ABI simplify; fix double-indirection, AGU load, code size on context switch. 2023-03-20 10:31:09 -07:00
Jason Volk
82482278fb construct: Handle SIGUSR2 to refresh slave. 2023-03-20 00:49:11 -07:00
Jason Volk
fc495f06ef ircd:Ⓜ️ Add slave refresh interface. 2023-03-20 00:49:11 -07:00
Jason Volk
1973e2c086 construct: Fix the conf rehash interface for SIGUSR1. 2023-03-19 22:21:27 -07:00
Jason Volk
63c70ecdfc construct: Support IORING_SETUP_SINGLE_ISSUER (Linux 6.0) 2023-03-19 20:02:17 -07:00
Jason Volk
13f3d1ebc6 ircd::net: Add state for nodelay in socket w/ condition to elide; interface overloads. 2023-03-19 19:56:14 -07:00
Jason Volk
b07fa8c110 ircd::net: Add interface for native_non_blocking(); mitigate unconditional ioctl. 2023-03-19 19:47:53 -07:00
Jason Volk
4d2478f814 ircd::net: Minor sock_opts interface ABI simplifications. 2023-03-19 18:02:53 -07:00
Jason Volk
f2626d39a7 ircd::net::acceptor: Eliminate getpeername(2) for address already returned on accept(2). 2023-03-19 14:12:00 -07:00
Jason Volk
c3c73fcbe7 ircd: Promote slave-mode to a non-maintenance mode allowing listeners. 2023-03-19 13:21:11 -07:00
Jason Volk
19462b5fae ircd: Remove write_avoid mode replaced by maintenance mode. 2023-03-19 13:21:11 -07:00
Jason Volk
96b1d68933 ircd::db: Add paranoid check conf suite; open stats confs; feature defines. 2023-03-19 13:21:11 -07:00
Jason Volk
c807550ca7 modules/console: Add column for checkpoint to db files cmd. 2023-03-18 20:44:22 -07:00
Jason Volk
3709cda233 modules/console: Add argument to peer version cmd to filter by expression. 2023-03-18 20:43:10 -07:00
Jason Volk
eca02723b3 ircd::net: Consolidate SSL_pending related behind interface. 2023-03-18 14:08:20 -07:00
Jason Volk
ee31b5a59f ircd::net::listener: Support non-SSL listening sockets. 2023-03-18 14:08:20 -07:00
Jason Volk
4e16f1849b ircd::net::socket: Support non-SSL sockets. 2023-03-18 14:08:20 -07:00
Jason Volk
3fcfaddc3e ircd::net::acceptor: Move alpn/sni init config into method; minor reorg. 2023-03-18 11:52:51 -07:00
Jason Volk
abf1ed47c6 modules/console: Increase various output table column widths. 2023-03-18 11:52:51 -07:00
Jason Volk
cc36c17c03 ircd::db: Update for Cache interface refactor; reflections. (RocksDB 8.0) 2023-03-17 21:25:15 -07:00
Jason Volk
1e4f44f41d ircd::db: Deprecate compressed block cache. (RocksDB 8.0) 2023-03-17 21:25:15 -07:00
Jason Volk
5994475542 ircd::db: Update for required ConfigOptions. (RocksDB 8.0) 2023-03-17 21:25:10 -07:00
Jason Volk
eff25f45f1 modules/console: Show totals specific to args passed to db files cmd. 2023-03-17 15:26:22 -07:00
Jason Volk
c6ce4d3229 appveyor: Skip extra builds for tags; minor cleanup. 2023-03-16 22:06:01 -07:00
Jason Volk
76def3378f ircd:Ⓜ️:vm::fetch: Add stats items. 2023-03-16 21:44:20 -07:00
Jason Volk
767f6cbae5 ircd:Ⓜ️:vm::fetch: Eliminate the remaining coarse sleep for prev fetch+eval. 2023-03-16 21:44:20 -07:00
Jason Volk
dcfae310ab ircd:Ⓜ️:vm::notify: Add promise/future based notify interface. 2023-03-16 19:32:14 -07:00
Jason Volk
2331de3f3e ircd:Ⓜ️:vm::notify: Use ctx::future/promise rather than direct context notify. 2023-03-16 19:25:45 -07:00
Jason Volk
0b6669ca20 ircd::util::unique_iterator: Add deref operator passthrus; minor reorg. 2023-03-16 17:23:28 -07:00
Jason Volk
43838608fd ircd::fs::dev: Add stats gather struct w/ console cmd. 2023-03-16 12:03:32 -07:00
Jason Volk
f85781b65a ircd::fs::dev: Move dev::blk related into class nested. 2023-03-16 11:42:49 -07:00
Jason Volk
d01e937f3e modules/console: Add arg to check specific file in db check cmd. 2023-03-16 10:21:10 -07:00
Jason Volk
9558637c20 modules/key/server: Selective key occlusion blacklisting w/ conf items. 2023-03-15 10:50:26 -07:00
Jason Volk
e3edcefe17 modules/client/register_available: Fix and remove stale dynamic module links. 2023-03-15 10:50:26 -07:00
Jason Volk
ac3b85114b ircd::net::dns::resolver: Add conf item to ease off ServFail retries. 2023-03-14 22:34:58 -07:00
Jason Volk
684dd18497 ircd:Ⓜ️:event: Split signature generating overload w/ custom sk/pk. 2023-03-14 20:51:46 -07:00
Jason Volk
fb9b68b4e3 ircd:Ⓜ️:push: Reenable .m.rule.message default underride (with dont_notify e7089e8e7f). 2023-03-14 13:56:34 -07:00
Jason Volk
00094e272f ircd:Ⓜ️:vm::inject: Add membership prior to auth_events generation.
Fixes m.room.join_rules added to membership=leave auth_events.

(regression matrix-org/synapse#13088 matrix-org/synapse@d4b1c0d800)
2023-03-13 19:09:08 -07:00
Jason Volk
98e366e012 ircd:Ⓜ️:room::auth: Perform relative check against state without the event included. 2023-03-13 12:54:08 -07:00
Jason Volk
b5932ba33c modules/console: Add fed auth space cmd to the 'proper toolset and diagnostic equipment' 2023-03-12 19:59:24 -07:00
Jason Volk
70b49b96d2 Drop gcc-9; minimum required: gcc-10. 2023-03-12 13:38:45 -07:00
Jason Volk
ac8889b74c modules/client/sync: Fix crazyloading sync property ordering bugging riot. (regression 9836f65c05) 2023-03-12 12:53:02 -07:00
Jason Volk
e7089e8e7f ircd:Ⓜ️:push: Default to dont_notify for common underrides. 2023-03-11 17:28:13 -08:00
Jason Volk
7e28a27549 modules/console: Refactor feds auth output into grid report; add type filter. 2023-03-11 15:24:44 -08:00
Jason Volk
fb9f2b0bcc modules/console: Add param to room auth and fed auth cmds to filter by type. 2023-03-11 14:48:16 -08:00
Jason Volk
8ebd2089f2 ircd:Ⓜ️:media: Add optional file name to file room state. 2023-03-11 14:14:35 -08:00
Jason Volk
96119166fa ircd::json::tuple: Modernize constant evaluation selection list constructions. 2023-03-11 13:48:37 -08:00
Jason Volk
527af013f7 ircd::string_view: Modernize constant evaluation comparisons. 2023-03-11 13:48:07 -08:00
Jason Volk
cefcc75943 ircd::util::bitset: Fix assertion made under constant evaluation. 2023-03-11 13:47:22 -08:00
186 changed files with 8546 additions and 2582 deletions

View file

@ -1,15 +1,20 @@
version: master_{build} version: master_{build}
clone_depth: 1 clone_depth: 1
skip_tags: true
services: services:
- docker - docker
branches:
only:
- appveyor
environment: environment:
COMMAND: if [[ "${APPVEYOR_REPO_COMMIT_MESSAGE@Q}" == *"[ci debug]"* ]]; then export DEBUG_FLAGS="--enable-debug"; fi; if [[ "${APPVEYOR_REPO_COMMIT_MESSAGE@Q}" == *"[ci gdb]"* ]]; then apt-get update && apt-get -y install gdb && export CONSTRUCT="gdb --batch -ex r -ex bt --return-child-result --args construct"; else export CONSTRUCT="construct"; fi && rmdir -v deps/rocksdb && ln -sv /usr/src/rocksdb deps && ./autogen.sh && ./configure --enable-assert ${DEBUG_FLAGS} && make install && ${CONSTRUCT} -smoketest -debug localhost COMMAND: if [[ "${APPVEYOR_REPO_COMMIT_MESSAGE@Q}" == *"[ci debug]"* ]]; then export DEBUG_FLAGS="--enable-debug"; fi; if [[ "${APPVEYOR_REPO_COMMIT_MESSAGE@Q}" == *"[ci gdb]"* ]]; then apt-get update && apt-get -y install gdb && export CONSTRUCT="gdb --batch -ex r -ex bt --return-child-result --args construct"; else export CONSTRUCT="construct"; fi && rmdir -v deps/rocksdb && ln -sv /usr/src/rocksdb deps && ./autogen.sh && ./configure --enable-assert ${DEBUG_FLAGS} && make install && ${CONSTRUCT} -smoketest -debug localhost
matrix: matrix:
- APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu1804 - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu1804
DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-base-build-gcc-9-amd64 DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-base-build-gcc-10-amd64
- APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu1804 - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu1804
DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-base-build-gcc-12-amd64 DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-base-build-gcc-12-amd64
@ -18,8 +23,8 @@ environment:
DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-full-build-clang-15-amd64 DOCKER_IMAGE: jevolk/construct:ubuntu-22.04-full-build-clang-15-amd64
for: for:
- -
build_script: build_script:
- docker run -v `pwd`:/build "${DOCKER_IMAGE}" /bin/bash -c "${COMMAND}" - docker run -v `pwd`:/build "${DOCKER_IMAGE}" /bin/bash -c "${COMMAND}"
matrix: matrix:

98
.github/workflows/docker.yml vendored Normal file
View file

@ -0,0 +1,98 @@
#
# Build docker images
#
name: Docker Images
on:
workflow_dispatch:
inputs:
features:
type: string
description: JSON array of feature-set names to build images for.
distros:
type: string
description: JSON array of operating system distros to build for.
machines:
type: string
description: JSON array of machines to build for.
toolchains:
type: string
description: JSON array of compiler toolchains to build for.
concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
env:
ctor_id: ${{vars.DOCKER_ID}}
ctor_url: https://github.com/${{github.repository}}
jobs:
# Build the base-feature intermediate images (cached and not shipped).
base:
uses: ./.github/workflows/docker_prime.yml
with:
id: ${{github.env.ctor_id}}
url: ${{github.env.ctor_url}}
features: '["base"]'
distros: ${{github.event.inputs.distros || vars.DOCKER_DISTROS}}
machines: ${{github.event.inputs.machines || vars.DOCKER_MACHINES}}
test: ${{contains(github.events.push.commits[0].message, '[ci test]')}}
# Build the full-feature intermediate images (cached and not shipped).
full:
uses: ./.github/workflows/docker_prime.yml
needs: [base]
with:
id: ${{github.env.ctor_id}}
url: ${{github.env.ctor_url}}
features: '["full"]'
distros: ${{github.event.inputs.distros || vars.DOCKER_DISTROS}}
machines: ${{github.event.inputs.machines || vars.DOCKER_MACHINES}}
test: ${{contains(github.events.push.commits[0].message, '[ci test]')}}
# Build the leaf images (shipped and not cached)
built:
needs: [base, full]
runs-on: ${{matrix.machine}}
strategy:
fail-fast: false
matrix:
feature: ${{fromJSON(github.event.inputs.features || vars.DOCKER_FEATURES)}}
distro: ${{fromJSON(github.event.inputs.distros || vars.DOCKER_DISTROS)}}
machine: ${{fromJSON(github.event.inputs.machines || vars.DOCKER_MACHINES)}}
toolchain: ${{fromJSON(github.event.inputs.toolchains || vars.DOCKER_TOOLCHAINS)}}
exclude:
- distro: alpine-3.17
toolchain: gcc-10 # n/a on distro version
- distro: alpine-3.17
toolchain: gcc-11 # n/a on distro version
- distro: alpine-3.17
toolchain: clang-14 # n/a on distro version
- distro: ubuntu-22.04
toolchain: clang-15 # n/a on distro version
- machine: arm64
toolchain: gcc-12 # build hangs
- machine: arm64
toolchain: gcc-11 # build hangs
- machine: arm64
toolchain: gcc-10 # build hangs
- machine: arm64
distro: alpine-3.17
toolchain: clang-14 # smoketest crash
- machine: arm64
distro: alpine-3.17
toolchain: clang-15 # smoketest crash
steps:
- uses: actions/checkout@v3
- name: build
env:
ctor_features: ${{matrix.feature}}
ctor_distros: ${{matrix.distro}}
ctor_machines: ${{matrix.machine}}
ctor_toolchains: ${{matrix.toolchain}}
ctor_test: ${{contains(github.events.push.commits[0].message, '[ci test]')}}
run: |
docker/build-and-push-images.sh

62
.github/workflows/docker_prime.yml vendored Normal file
View file

@ -0,0 +1,62 @@
#
# Build intermediate images
#
# Called to build lower-layer images which other images depend on. These are
# cached for use by the next layer but not shipped to users.
#
name: Docker Images Prime
on:
workflow_call:
inputs:
id:
type: string
description: Dockerhub acct/repo identity.
url:
type: string
description: Git repository for checkout.
features:
type: string
description: JSON array of feature-set names to build images for.
distros:
type: string
description: JSON array of operating system distros to build for.
machines:
type: string
description: JSON array of machines to build for.
test:
type: boolean
default: false
required: false
description: Echo all docker commands rather than invoking them.
concurrency:
group: ${{github.workflow}}-${{inputs.features}}
cancel-in-progress: false
env:
ctor_id: ${{inputs.id}}
ctor_url: ${{inputs.url}}
jobs:
prime:
runs-on: ${{matrix.machine}}
strategy:
fail-fast: true
matrix:
feature: ${{fromJSON(inputs.features)}}
distro: ${{fromJSON(inputs.distros)}}
machine: ${{fromJSON(inputs.machines)}}
steps:
- uses: actions/checkout@v3
- name: build
env:
ctor_features: ${{matrix.feature}}
ctor_distros: ${{matrix.distro}}
ctor_machines: ${{matrix.machine}}
ctor_toolchains: false
ctor_test: ${{inputs.test}}
run: |
docker/build-and-push-images.sh ${{matrix.feature}}

1
.gitignore vendored
View file

@ -19,6 +19,7 @@ Makefile
*.gch *.gch
*.cache *.cache
*.tmp *.tmp
*.save
*.profdata *.profdata
.deps .deps
.dirstamp .dirstamp

View file

@ -1,10 +1,10 @@
# This — is The **Construct** # This — is The **Construct**
#### Federated Messaging Server #### FEDERATED MESSAGING SERVER
[![Chat in #construct:zemos.net](https://img.shields.io/matrix/construct:zemos.net.svg?label=Chat%20in%20%23construct%3Azemos.net&logo=matrix&server_fqdn=matrix.org&style=for-the-badge&color=5965AF)](https://matrix.to/#/#construct:zemos.net) [![](https://img.shields.io/badge/License-BSD-5965AF.svg?label=%20license&style=for-the-badge)]() [![Chat in #construct:zemos.net](https://img.shields.io/matrix/construct:zemos.net.svg?label=Chat%20in%20%23construct%3Azemos.net&logo=matrix&server_fqdn=matrix.org&style=for-the-badge&color=5965AF)](https://matrix.to/#/#construct:zemos.net) [![](https://img.shields.io/badge/License-BSD-5965AF.svg?label=%20license&style=for-the-badge)]()
### 📦 RUN YOUR OWN ### 🚚 GET CONSTRUCT
- `git clone https://github.com/matrix-construct/construct` - `git clone https://github.com/matrix-construct/construct`
@ -13,19 +13,17 @@
[![](https://img.shields.io/github/directory-file-count/matrix-construct/construct.svg?type=dir&label=directories&logo=GitHub&style=flat-square&color=5965AF)](https://github.com/matrix-construct/construct) [![](https://img.shields.io/github/directory-file-count/matrix-construct/construct.svg?type=dir&label=directories&logo=GitHub&style=flat-square&color=5965AF)](https://github.com/matrix-construct/construct)
[![](https://img.shields.io/github/directory-file-count/matrix-construct/construct.svg?type=file&label=files&logo=GitHub&style=flat-square&color=5965AF)](https://github.com/matrix-construct/construct) [![](https://img.shields.io/github/directory-file-count/matrix-construct/construct.svg?type=file&label=files&logo=GitHub&style=flat-square&color=5965AF)](https://github.com/matrix-construct/construct)
| Fully Featured Builds | Minimal Dependencies | - 📦 [**DISTRIBUTION PACKAGES**](https://github.com/matrix-construct/construct/wiki/PACKAGE)
|:---|:---|
| [![](https://img.shields.io/docker/image-size/jevolk/construct/ubuntu-22.04-full-built-clang-15-amd64.svg?logoWidth=25&label=ubuntu%2022.04%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags) | [![](https://img.shields.io/docker/image-size/jevolk/construct/ubuntu-22.04-base-built-gcc-12-amd64.svg?logoWidth=25&label=ubuntu%2022.04%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags) - 🐋 [**DOCKER IMAGES**](https://github.com/matrix-construct/construct/wiki/DOCKER)
| [![](https://img.shields.io/docker/image-size/jevolk/construct/alpine-3.16-full-built-clang-amd64.svg?logoWidth=25&label=alpine%203.16%20clang%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags) | [![](https://img.shields.io/docker/image-size/jevolk/construct/alpine-3.16-base-built-clang-amd64.svg?logoWidth=25&label=alpine%203.16%20clang%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags)
| [![](https://img.shields.io/docker/image-size/jevolk/construct/alpine-3.16-full-built-gcc-amd64.svg?logoWidth=25&label=alpine%203.16%20gcc%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags) | [![](https://img.shields.io/docker/image-size/jevolk/construct/alpine-3.16-base-built-gcc-amd64.svg?logoWidth=25&label=alpine%203.16%20gcc%20amd64&logo=Docker&style=flat-square&color=5965AF)](https://registry.hub.docker.com/r/jevolk/construct/tags)
### 🗒️ INSTRUCTIONS ### 🗒️ INSTRUCTIONS
1. 🏗️ [BUILD](https://github.com/matrix-construct/construct/wiki/BUILD) instructions to compile Construct from source. 1. 🏗️ [**BUILD**](https://github.com/matrix-construct/construct/wiki/BUILD) to compile Construct from source.
2. 🪛 [SETUP](https://github.com/matrix-construct/construct/wiki/SETUP) instructions to run Construct for the first time. 2. 🪛 [**SETUP**](https://github.com/matrix-construct/construct/wiki/SETUP) to run Construct for the first time.
3. ⚡ [TUNING](https://github.com/matrix-construct/construct/wiki/TUNING) guide to optimize Construct for your deployment. 3. ⚡ [**TUNING**](https://github.com/matrix-construct/construct/wiki/TUNING) to optimize Construct for your deployment.
- 🙋 [TROUBLESHOOTING](https://github.com/matrix-construct/construct/wiki/Troubleshooting-problems) guide for solutions to possible problems. - 🙋 [TROUBLESHOOTING](https://github.com/matrix-construct/construct/wiki/Troubleshooting-problems) guide for solutions to possible problems.

View file

@ -83,6 +83,16 @@ dnl
dnl Platform dnl Platform
dnl dnl
dnl
dnl Hardwares
dnl
AM_CONDITIONAL([AMD64], [[[[ $host_cpu = *x86_64* ]]]])
AM_CONDITIONAL([X86_64], [[[[ $host_cpu = *x86_64* ]]]])
AM_CONDITIONAL([ARM64], [[[[ $host_cpu = *aarch64* ]]]])
AM_CONDITIONAL([AARCH64], [[[[ $host_cpu = *aarch64* ]]]])
dnl dnl
dnl Compiler brand dnl Compiler brand
dnl dnl
@ -151,21 +161,25 @@ dnl Debugging mode
dnl dnl
AC_MSG_CHECKING(whether to enable debugging) AC_MSG_CHECKING(whether to enable debugging)
AC_ARG_ENABLE(debug, RB_HELP_STRING([--enable-debug], [Enable debugging suite for development]), AC_ARG_ENABLE(debug, RB_HELP_STRING([--enable-debug=[[no]]], [Enable debugging suite for development]),
[
debug=$enableval
], [
debug="no"
])
AC_MSG_RESULT([$debug])
AM_CONDITIONAL([DEBUG], [[[[ "$debug" = "yes" ]]]])
AM_COND_IF([DEBUG],
[ [
debug="yes"
AC_MSG_RESULT([yes])
AC_SUBST(DEBUG, 1) AC_SUBST(DEBUG, 1)
RB_DEFINE_UNQUOTED([DEBUG], [1], [Not configured for release when lit.]) RB_DEFINE_UNQUOTED([DEBUG], [1], [Not configured for release when lit.])
RB_DEFINE_UNQUOTED([DEBUG_LEVEL], [1], [Defined to 0 for release; or > 0 otherwise]) RB_DEFINE_UNQUOTED([DEBUG_LEVEL], [1], [Defined to 0 for release; or > 0 otherwise])
], [ ], [
debug="no" RB_DEFINE_UNQUOTED([DEBUG_LEVEL], [0], [Defined to 0 for release; or > 0 otherwise])
AC_MSG_RESULT([no])
RB_DEFINE_UNQUOTED([DEBUG_LEVEL], [0], [Defined to 0 for release])
]) ])
AM_CONDITIONAL([DEBUG], [[[[ "$DEBUG" = "1" ]]]])
dnl dnl
dnl Compactness dnl Compactness
dnl dnl
@ -173,14 +187,19 @@ dnl
AC_MSG_CHECKING(Optimize for size; strip symbols; force no debugging) AC_MSG_CHECKING(Optimize for size; strip symbols; force no debugging)
AC_ARG_ENABLE(compact, RB_HELP_STRING([--enable-compact], [Optimize for size and compactness]), AC_ARG_ENABLE(compact, RB_HELP_STRING([--enable-compact], [Optimize for size and compactness]),
[ [
AC_MSG_RESULT([yes]) compact=$enableval
AC_SUBST(COMPACT, 1)
RB_DEFINE_UNQUOTED([COMPACT], [1], [Not configured for compactness when lit.])
], [ ], [
AC_MSG_RESULT([no]) compact="no"
]) ])
AM_CONDITIONAL([COMPACT], [[[[ "$COMPACT" = "1" ]]]]) AC_MSG_RESULT([$compact])
AM_CONDITIONAL([COMPACT], [[[[ "$compact" = "yes" ]]]])
AM_COND_IF([COMPACT],
[
AC_SUBST(COMPACT, 1)
RB_DEFINE_UNQUOTED([COMPACT], [1], [Not configured for compactness when lit.])
])
dnl dnl
dnl Explicit assert switch for still using assert() without --enable-debug dnl Explicit assert switch for still using assert() without --enable-debug
@ -189,26 +208,26 @@ dnl
AC_MSG_CHECKING(whether to explicitly enable assertions) AC_MSG_CHECKING(whether to explicitly enable assertions)
AC_ARG_ENABLE(assert, RB_HELP_STRING([--enable-assert], [Enable assertions without --enable-debug]), AC_ARG_ENABLE(assert, RB_HELP_STRING([--enable-assert], [Enable assertions without --enable-debug]),
[ [
AC_MSG_RESULT([yes]) assert=$enableval
AC_SUBST(ASSERT, 1) AC_MSG_RESULT([$assert])
], [ ], [
AM_COND_IF(DEBUG, AM_COND_IF(DEBUG,
[ [
AC_MSG_RESULT([no, but assertions are enabled anyway]) assert="yes"
AC_SUBST(ASSERT, 1) AC_MSG_RESULT([$assert, enabled by default with --enable-debug])
], [ ], [
AC_MSG_RESULT([no]) assert="no"
AC_MSG_RESULT([$assert])
]) ])
]) ])
AM_CONDITIONAL([ASSERT], [[[[ "$ASSERT" = "1" ]]]]) AM_CONDITIONAL([ASSERT], [[[[ "$assert" = "yes" ]]]])
AM_COND_IF(ASSERT, AM_COND_IF(ASSERT,
[ [
assert="yes"
assert_type="abort" assert_type="abort"
AC_SUBST(ASSERT, 1)
], [ ], [
assert="no"
CPPDEFINE([NDEBUG]) CPPDEFINE([NDEBUG])
]) ])
@ -216,18 +235,17 @@ dnl
dnl Switch to control the action of assert() dnl Switch to control the action of assert()
dnl dnl
AC_MSG_CHECKING(whether to change the behavior of assertions) AC_MSG_CHECKING(what the result of a failed assertion implies)
AC_ARG_WITH(assert, RB_HELP_STRING([--with-assert[[[=abort]]]], [Soften assertion behavior]), AC_ARG_WITH(assert, RB_HELP_STRING([--with-assert[[[=abort]]]], [Change assertion behavior]),
[ [
assert_type=$withval assert_type=$withval
AC_MSG_RESULT([yes, "$assert_type"])
AC_SUBST(ASSERT_TYPE, $assert_type)
], [
AC_MSG_RESULT([no])
]) ])
AM_COND_IF(ASSERT, AM_COND_IF(ASSERT,
[ [
AC_MSG_RESULT([$assert_type])
AC_SUBST(ASSERT_TYPE, $assert_type)
if [[ ! -z "$assert_type" ]]; then if [[ ! -z "$assert_type" ]]; then
if [[ "$assert_type" != "abort" ]]; then if [[ "$assert_type" != "abort" ]]; then
@ -243,6 +261,9 @@ AM_COND_IF(ASSERT,
fi fi
fi fi
], [
AC_MSG_RESULT([nothing without assertions enabled])
assert_type=""
]) ])
dnl dnl
@ -252,19 +273,25 @@ dnl
AC_MSG_CHECKING(whether to explicitly enable optimized build) AC_MSG_CHECKING(whether to explicitly enable optimized build)
AC_ARG_ENABLE(optimize, RB_HELP_STRING([--enable-optimize], [Enable optimization even with --enable-debug]), AC_ARG_ENABLE(optimize, RB_HELP_STRING([--enable-optimize], [Enable optimization even with --enable-debug]),
[ [
AC_MSG_RESULT([yes]) optimize=$enableval
AC_SUBST(OPTIMIZE, 1) AC_MSG_RESULT([$optimize])
], [ ], [
AM_COND_IF(DEBUG, AM_COND_IF(DEBUG,
[ [
AC_MSG_RESULT([no]) optimize="no"
AC_MSG_RESULT([$optimize, disabled by default with --enable-debug])
], [ ], [
AC_MSG_RESULT([no, but optimized build is enabled anyway]) optimize="yes"
AC_SUBST(OPTIMIZE, 1) AC_MSG_RESULT([$optimize])
]) ])
]) ])
AM_CONDITIONAL([OPTIMIZE], [[[[ "$OPTIMIZE" = "1" ]]]]) AM_CONDITIONAL([OPTIMIZE], [[[[ "$optimize" = "yes" ]]]])
AM_COND_IF([OPTIMIZE],
[
AC_SUBST(OPTIMIZE, 1)
])
dnl dnl
dnl Enable target clones code generation dnl Enable target clones code generation
@ -273,16 +300,19 @@ dnl
AC_MSG_CHECKING(whether to generate code for target clones) AC_MSG_CHECKING(whether to generate code for target clones)
AC_ARG_ENABLE(clones, RB_HELP_STRING([--enable-clones], [Enable target clones generation]), AC_ARG_ENABLE(clones, RB_HELP_STRING([--enable-clones], [Enable target clones generation]),
[ [
AC_MSG_RESULT([yes])
AC_SUBST(CLONES, 1)
RB_DEFINE([CLONES], [1], [Function multi-versioning for different architectures])
clones="yes" clones="yes"
], [ ], [
AC_MSG_RESULT([no])
clones="no" clones="no"
]) ])
AM_CONDITIONAL([CLONES], [[[[ "$CLONES" = "1" ]]]]) AC_MSG_RESULT([$clones])
AM_CONDITIONAL([CLONES], [[[[ "$clones" = "yes" ]]]])
AM_COND_IF([CLONES],
[
AC_SUBST(CLONES, 1)
RB_DEFINE([CLONES], [1], [Function multi-versioning for different architectures])
])
dnl dnl
dnl Explicit link-time-optimization switch dnl Explicit link-time-optimization switch
@ -309,7 +339,6 @@ if test "$lto" = "yes"; then
], [ ], [
lto="yes" lto="yes"
AC_MSG_RESULT([yes]) AC_MSG_RESULT([yes])
AC_SUBST(LTO, 1)
]) ])
], [ ], [
lto="no" lto="no"
@ -323,7 +352,12 @@ else
AC_MSG_RESULT([no]) AC_MSG_RESULT([no])
fi fi
AM_CONDITIONAL([LTO], [[[[ "$LTO" = "1" ]]]]) AM_CONDITIONAL([LTO], [[[[ "$lto" = "yes" ]]]])
AM_COND_IF([LTO],
[
AC_SUBST(LTO, 1)
])
dnl dnl
dnl Explicit optimization level switch dnl Explicit optimization level switch
@ -419,67 +453,74 @@ dnl Precompiled headers
dnl dnl
AC_MSG_CHECKING(whether to enable precompiled headers) AC_MSG_CHECKING(whether to enable precompiled headers)
AC_ARG_ENABLE(pch, RB_HELP_STRING([--disable-pch], [Disable precompiled header generation]), AC_ARG_ENABLE(pch, RB_HELP_STRING([--enable-pch], [Enable precompiled header generation]),
[ [
build_pch=$enableval build_pch=$enableval
AC_MSG_RESULT([$enableval])
], [ ], [
build_pch="yes" build_pch="yes"
CPPDEFINE([PCH])
AC_MSG_RESULT([yes])
]) ])
AC_MSG_RESULT([$build_pch])
AM_CONDITIONAL([BUILD_PCH], [[[[ "$build_pch" = "yes" ]]]]) AM_CONDITIONAL([BUILD_PCH], [[[[ "$build_pch" = "yes" ]]]])
AM_CONDITIONAL([CLANG_PCH], [[[[ "$build_pch" = "yes" ]] && [[ $CXX = clang* ]]]]) AM_CONDITIONAL([CLANG_PCH], [[[[ "$build_pch" = "yes" ]] && [[ $CXX = clang* ]]]])
AM_COND_IF([BUILD_PCH],
[
CPPDEFINE([PCH])
])
dnl dnl
dnl Generic Mode compilation dnl Generic Mode compilation
dnl dnl
AC_MSG_CHECKING(whether to enable generic mode or tune for this host) AC_MSG_CHECKING(whether to enable generic mode or emit for this host)
AC_ARG_ENABLE(generic, RB_HELP_STRING([--enable-generic], [Emit more generic code for pkg binaries]), AC_ARG_ENABLE(generic, RB_HELP_STRING([--enable-generic], [Emit more generic code for pkg binaries]),
[ [
enable_generic="yes" enable_generic="yes"
AC_MSG_RESULT([yes])
RB_DEFINE([GENERIC], [1], [Building binary tuned for generic architectures])
], [ ], [
enable_generic="no" enable_generic="no"
AC_MSG_RESULT([no])
]) ])
AC_MSG_RESULT([$enable_generic])
AM_CONDITIONAL([GENERIC], [[[[ "$enable_generic" = "yes" ]]]]) AM_CONDITIONAL([GENERIC], [[[[ "$enable_generic" = "yes" ]]]])
dnl AM_COND_IF([GENERIC],
dnl Untuned Mode compilation
dnl
AC_MSG_CHECKING(whether to enable fully untuned mode)
AC_ARG_ENABLE(untuned, RB_HELP_STRING([--enable-untuned], [Emit no special feature instructions]),
[ [
enable_untuned="yes" RB_DEFINE([GENERIC], [1], [Building binary tuned for generic architectures])
AC_MSG_RESULT([yes])
RB_DEFINE([UNTUNED], [1], [Building binary without extended-feature cpu instructions.])
], [
enable_untuned="no"
AC_MSG_RESULT([no])
]) ])
AM_CONDITIONAL([UNTUNED], [[[[ "$enable_untuned" = "yes" ]]]]) dnl
dnl Tuned compilation
dnl
machine=""
AC_MSG_CHECKING(whether to emit specific architecture features)
AC_ARG_WITH(machine, RB_HELP_STRING([--with-machine], [Emit special feature instructions]),
[
if [[ "$withval" != "yes" ]]; then
machine=$withval
fi
])
if [[ ! -z "$machine" ]]; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
fi
dnl dnl
dnl Disable third-party allocators dnl Disable third-party allocators
dnl dnl
AC_MSG_CHECKING(whether to allow available third-party allocator libraries from being used) AC_MSG_CHECKING(whether to allow available third-party allocator libraries from being used)
AC_ARG_ENABLE(malloc-libs, RB_HELP_STRING([--disable-malloc-libs], [Disable third-party dynamic memory libraries (jemalloc/tcmalloc/etc)]), AC_ARG_ENABLE(malloc-libs, RB_HELP_STRING([--enable-malloc-libs], [Enable third-party dynamic memory libraries (jemalloc/tcmalloc/etc)]),
[ [
use_malloc_libs=$enableval use_malloc_libs=$enableval
AC_MSG_RESULT([$enableval])
], [ ], [
use_malloc_libs="yes" use_malloc_libs="yes"
AC_MSG_RESULT([yes])
]) ])
AC_MSG_RESULT([$use_malloc_libs])
AM_CONDITIONAL([MALLOC_LIBS], [[[[ "$use_malloc_libs" = "yes" ]]]]) AM_CONDITIONAL([MALLOC_LIBS], [[[[ "$use_malloc_libs" = "yes" ]]]])
dnl dnl
@ -493,6 +534,7 @@ AC_ARG_ENABLE(lowmem-compile, RB_HELP_STRING([--enable-lowmem-compile], [Enable
lowmem_compile="no" lowmem_compile="no"
]) ])
AC_MSG_RESULT([$lowmem_compile])
AM_CONDITIONAL([LOWMEM_COMPILE], [[[[ $lowmem_compile = "yes" ]]]]) AM_CONDITIONAL([LOWMEM_COMPILE], [[[[ $lowmem_compile = "yes" ]]]])
dnl dnl
@ -737,16 +779,20 @@ AM_COND_IF([GCC],
[ [
RB_VAR_PREPEND([CXXFLAGS], ["-fstack-protector-explicit"]) RB_VAR_PREPEND([CXXFLAGS], ["-fstack-protector-explicit"])
dnl These flags should not be used on Intel-CET capable platforms AS_CASE([$host_cpu],
dnl TODO: XXX [x86_64],
AS_IF([test "$CXX_EPOCH" -ge "9"],
[ [
AM_COND_IF([GENERIC], dnl These flags should not be used on Intel-CET capable platforms
dnl TODO: XXX
AS_IF([test "$CXX_EPOCH" -ge "9"],
[ [
RB_VAR_PREPEND([CXXFLAGS], ["-fcf-protection=full"]) AM_COND_IF([GENERIC],
], [ [
RB_VAR_PREPEND([CXXFLAGS], ["-fcf-protection=none"]) RB_VAR_PREPEND([CXXFLAGS], ["-fcf-protection=return"])
RB_VAR_PREPEND([CXXFLAGS], ["-mmanual-endbr"]) ], [
RB_VAR_PREPEND([CXXFLAGS], ["-fcf-protection=none"])
RB_VAR_PREPEND([CXXFLAGS], ["-mmanual-endbr"])
])
]) ])
]) ])
]) ])
@ -760,40 +806,83 @@ dnl
dnl Machine Tuning dnl Machine Tuning
dnl dnl
TUNE_FLAGS="" MACHINE_FLAGS=""
AM_COND_IF([UNTUNED],
dnl Initial machine
machine_arch="native"
machine_tune="native"
AM_COND_IF([CLANG],
[ [
machine_tuning="generic ${host_cpu} and untuned" AS_IF([test "$CXX_EPOCH" -lt "15"],
RB_VAR_PREPEND([TUNE_FLAGS], ["-mno-default"])
], [
AM_COND_IF([GENERIC],
[ [
machine_tuning="generic ${host_cpu}" AS_IF([test "$host_cpu" = "aarch64"],
RB_VAR_PREPEND([TUNE_FLAGS], ["-mtune=generic"])
], [
machine_tuning="${host_cpu} native"
AS_CASE([$host_cpu],
[x86_64],
[ [
dnl AMD K10's SSE4a doesn't work with valgrind machine_arch=""
RB_VAR_PREPEND([TUNE_FLAGS], ["-mno-sse4a"])
dnl Not accepted by clang on aarch64
RB_VAR_PREPEND([TUNE_FLAGS], ["-march=native"])
]) ])
RB_VAR_PREPEND([TUNE_FLAGS], ["-mtune=native"])
]) ])
]) ])
RB_VAR_PREPEND([CXXFLAGS], ["$TUNE_FLAGS"]) AM_COND_IF([GENERIC],
[
machine_tune="generic"
AS_CASE([$host_cpu],
[x86_64],
[
machine_arch="x86-64"
])
])
dnl Specific extension specification
machine_feat=""
for feature in $machine; do
if [[ $(echo $feature | cut -d"=" -f1) == "arch" ]]; then
machine_arch=$(echo $feature | cut -d"=" -f2)
elif [[ $(echo $feature | cut -d"=" -f1) == "tune" ]]; then
machine_tune=$(echo $feature | cut -d"=" -f2)
else
machine_feat="$feature $machine_feat"
RB_VAR_PREPEND([MACHINE_FLAGS], [-m${feature}])
fi
done
dnl Ensures mtune=generic accompanies one of the generic march= options
dnl even if --enable-generic wasn't given, otherwise gcc will reject.
AM_COND_IF_NOT([GENERIC],
[
AS_CASE([$machine_arch],
[x86-64*],
[
machine_tune="generic"
], [
machine_tune=$machine_arch
])
])
dnl Specific extension underrides
AS_CASE([$host_cpu],
[x86_64],
[
dnl AMD K10's SSE4a doesn't work with valgrind
RB_VAR_PREPEND([MACHINE_FLAGS], [-mno-sse4a])
])
if [[ ! -z "$machine_tune" ]]; then
RB_VAR_PREPEND([MACHINE_FLAGS], [-mtune=$machine_tune])
fi
if [[ ! -z "$machine_arch" ]]; then
RB_VAR_PREPEND([MACHINE_FLAGS], [-march=$machine_arch])
fi
machine_tuning="${host_cpu} arch=$machine_arch tune=$machine_tune :$machine_feat"
RB_VAR_PREPEND([CXXFLAGS], [$MACHINE_FLAGS])
dnl Get the target features for this build from gcc into a readable string dnl Get the target features for this build from gcc into a readable string
AM_COND_IF([GCC], AM_COND_IF([GCC],
[ [
_cxx=$(echo "${CXX}" | cut -d' ' -f1) _cxx=$(echo "${CXX}" | cut -d' ' -f1)
_cmd="${_cxx} -Q --help=target ${TUNE_FLAGS}" _cmd="${_cxx} -Q --help=target ${MACHINE_FLAGS}"
machine_features=$(${_cmd} | grep enabled | grep -v 'mno-') machine_features=$(${_cmd} | grep enabled | grep -v 'mno-')
machine_features=$(printf "\n${machine_features}") machine_features=$(printf "\n${machine_features}")
]) ])
@ -803,7 +892,7 @@ AM_COND_IF([CLANG],
[ [
_flag=0 _flag=0
_cxx=$(echo "${CXX}" | cut -d' ' -f1) _cxx=$(echo "${CXX}" | cut -d' ' -f1)
_str=$(${_cxx} -E ${TUNE_FLAGS} - -### 2>&1) _str=$(${_cxx} -E ${MACHINE_FLAGS} - -### 2>&1)
machine_features="" machine_features=""
for i in $_str; do for i in $_str; do
if [[ $i == '"-target-feature"' ]]; then if [[ $i == '"-target-feature"' ]]; then
@ -822,22 +911,24 @@ dnl
dnl Compiler warnings dnl Compiler warnings
dnl dnl
AC_MSG_CHECKING(whether to disable warnings) AC_MSG_CHECKING(whether to enable warnings)
AC_ARG_ENABLE(warnings, AC_ARG_ENABLE(warnings,
RB_HELP_STRING([--disable-warnings], [Disable all sorts of warnings like a rockstar]), RB_HELP_STRING([--enable-warnings], [Disable all sorts of warnings like a rockstar]),
[ [
AC_MSG_RESULT([$enableval]) warnings=$enableval
warnings="$enableval"
], [ ], [
AC_MSG_RESULT([no])
warnings="yes" warnings="yes"
]) ])
AC_MSG_RESULT([$warnings])
AM_CONDITIONAL([WARNINGS], [[[[ "$warnings" = "yes" ]]]])
AC_DEFUN([RB_MAYBE_CWARN], AC_DEFUN([RB_MAYBE_CWARN],
[ [
if test x"$warnings" == x"yes"; then AM_COND_IF([WARNINGS],
[
RB_MAYBE_CXXFLAG([$1], [$2]) RB_MAYBE_CXXFLAG([$1], [$2])
fi ])
]) ])
STACK_USAGE_WARNING=16384 STACK_USAGE_WARNING=16384
@ -897,6 +988,7 @@ dnl Compiler specific
AM_COND_IF([CLANG], AM_COND_IF([CLANG],
[ [
RB_MAYBE_CWARN([-Werror=return-stack-address], charybdis_cv_c_gcc_w_error_return_stack_address) RB_MAYBE_CWARN([-Werror=return-stack-address], charybdis_cv_c_gcc_w_error_return_stack_address)
RB_MAYBE_CWARN([-Werror=abstract-final-class], charybdis_cv_c_gcc_w_error_abstract_final_class)
RB_MAYBE_CWARN([-Wundefined-reinterpret-cast], charybdis_cv_c_gcc_w_undefined_reinterpret_cast) RB_MAYBE_CWARN([-Wundefined-reinterpret-cast], charybdis_cv_c_gcc_w_undefined_reinterpret_cast)
RB_MAYBE_CWARN([-Wconditional-uninitialized], charybdis_cv_c_gcc_w_conditional_uninitialized) RB_MAYBE_CWARN([-Wconditional-uninitialized], charybdis_cv_c_gcc_w_conditional_uninitialized)
@ -956,10 +1048,13 @@ AM_COND_IF([DEBUG],
]) ])
], [ ], [
RB_MAYBE_CWARN([-Wno-unknown-pragmas], charybdis_cv_c_gcc_w_no_unknown_pragmas) RB_MAYBE_CWARN([-Wno-unknown-pragmas], charybdis_cv_c_gcc_w_no_unknown_pragmas)
RB_MAYBE_CWARN([-Wno-deprecated-declarations], charybdis_cv_c_gcc_w_no_deprecated_declarations)
RB_MAYBE_CWARN([-Wno-deprecated], charybdis_cv_c_gcc_w_no_deprecated)
AM_COND_IF([CLANG], AM_COND_IF([CLANG],
[ [
RB_MAYBE_CWARN([-Wno-unknown-attributes], charybdis_cv_c_gcc_w_no_unknown_attributes) RB_MAYBE_CWARN([-Wno-unknown-attributes], charybdis_cv_c_gcc_w_no_unknown_attributes)
RB_MAYBE_CWARN([-Wno-uninitialized], charybdis_cv_c_gcc_w_no_uninitialized)
]) ])
AM_COND_IF([GCC], AM_COND_IF([GCC],
@ -980,6 +1075,7 @@ AC_MSG_CHECKING(whether to enable warning discovery)
AC_ARG_ENABLE(every-warning, AC_ARG_ENABLE(every-warning,
RB_HELP_STRING([--enable-every-warning], [Enable warning discovery (-Weverything)]), RB_HELP_STRING([--enable-every-warning], [Enable warning discovery (-Weverything)]),
[ [
AC_MSG_RESULT([yes])
RB_MAYBE_CWARN([-Weverything], charybdis_cv_c_gcc_w_everything) RB_MAYBE_CWARN([-Weverything], charybdis_cv_c_gcc_w_everything)
dnl Ignored a priori dnl Ignored a priori
@ -1017,9 +1113,11 @@ RB_HELP_STRING([--enable-every-warning], [Enable warning discovery (-Weverything
dnl Ignored with issue high priority dnl Ignored with issue high priority
RB_MAYBE_CWARN([-Wno-reserved-identifier], charybdis_cv_c_gcc_w_no_reserved_identifier) RB_MAYBE_CWARN([-Wno-reserved-identifier], charybdis_cv_c_gcc_w_no_reserved_identifier)
RB_MAYBE_CWARN([-Wno-reserved-macro-identifier], charybdis_cv_c_gcc_w_no_reserved_macro_identifier) RB_MAYBE_CWARN([-Wno-reserved-macro-identifier], charybdis_cv_c_gcc_w_no_reserved_macro_identifier)
], [
AC_MSG_RESULT([no])
]) ])
AM_COND_IF_NOT([UNTUNED], AM_COND_IF_NOT([GENERIC],
[ [
dnl When tuning for a specific target ignore the ABI compat warnings dnl When tuning for a specific target ignore the ABI compat warnings
RB_MAYBE_CWARN([-Wno-psabi], charybdis_cv_c_gcc_w_psabi) RB_MAYBE_CWARN([-Wno-psabi], charybdis_cv_c_gcc_w_psabi)
@ -1117,7 +1215,7 @@ AC_TYPE_INTPTR_T
AC_TYPE_UINTPTR_T AC_TYPE_UINTPTR_T
dnl C standard type sizes on this platform dnl C standard type sizes on this platform
AC_CHECK_SIZEOF([char]) RB_CHECK_SIZEOF([char], 1)
AC_CHECK_SIZEOF([short]) AC_CHECK_SIZEOF([short])
AC_CHECK_SIZEOF([int]) AC_CHECK_SIZEOF([int])
AC_CHECK_SIZEOF([float]) AC_CHECK_SIZEOF([float])
@ -1343,19 +1441,20 @@ dnl
dnl ip6 dnl ip6
dnl dnl
AC_ARG_ENABLE(ipv6, RB_HELP_STRING([--disable-ipv6], [Disable IPv6 support]), AC_ARG_ENABLE(ipv6, RB_HELP_STRING([--enable-ipv6], [Enable IPv6 support]),
[ [
ipv6=$enableval ipv6=$enableval
], [ ], [
ipv6="yes" ipv6="yes"
]) ])
AM_CONDITIONAL([HAVE_IPV6], [[[[ "$ipv6" = "yes" ]]]]) AM_CONDITIONAL([IPV6], [[[[ "$ipv6" = "yes" ]]]])
AM_COND_IF(HAVE_IPV6, AM_COND_IF(IPV6,
[ [
AC_DEFINE([HAVE_IPV6], [1], [IPv6 is supported]) AC_DEFINE([HAVE_IPV6], [1], [IPv6 is supported])
],[]) AC_SUBST(IPV6, 1)
])
dnl dnl
dnl netdb / libnss_db dnl netdb / libnss_db
@ -1414,12 +1513,14 @@ dnl
AM_COND_IF(LINUX, AM_COND_IF(LINUX,
[ [
AC_ARG_ENABLE(aio, RB_HELP_STRING([--disable-aio], [Disable kernel AIO support]), AC_ARG_ENABLE(aio, RB_HELP_STRING([--enable-aio], [Enable kernel AIO support]),
[ [
aio=$enableval aio=$enableval
], [ ], [
aio=yes aio="yes"
]) ])
AC_MSG_RESULT([$aio])
]) ])
AM_CONDITIONAL([AIO], [test "x$aio" = "xyes" ]) AM_CONDITIONAL([AIO], [test "x$aio" = "xyes" ])
@ -1437,12 +1538,14 @@ dnl
AM_COND_IF(LINUX, AM_COND_IF(LINUX,
[ [
AC_ARG_ENABLE(iou, RB_HELP_STRING([--disable-iou], [Disable kernel io_uring support]), AC_ARG_ENABLE(iou, RB_HELP_STRING([--enable-iou], [Enable kernel io_uring support]),
[ [
iou=$enableval iou=$enableval
], [ ], [
iou=yes iou="yes"
]) ])
AC_MSG_RESULT([$iou])
]) ])
AM_CONDITIONAL([IOU], [test "x$iou" = "xyes" ]) AM_CONDITIONAL([IOU], [test "x$iou" = "xyes" ])
@ -2311,20 +2414,15 @@ PKG_CHECK_MODULES(jemalloc, [jemalloc],
]) ])
]) ])
AC_MSG_CHECKING(whether to disable use of any found jemalloc) AC_MSG_CHECKING(whether to enable use of any found jemalloc)
AC_ARG_ENABLE(jemalloc, RB_HELP_STRING([--disable-jemalloc], [Disable jemalloc as third-party dynamic memory manager]), AC_ARG_ENABLE(jemalloc, RB_HELP_STRING([--enable-jemalloc], [Enable jemalloc as dynamic memory manager]),
[ [
enable_jemalloc=$enableval enable_jemalloc=$enableval
if test "$enable_jemalloc" = "no"; then
AC_MSG_RESULT([yes])
else
AC_MSG_RESULT([no])
fi
], [ ], [
AC_MSG_RESULT([no])
enable_jemalloc="yes" enable_jemalloc="yes"
]) ])
AC_MSG_RESULT([$enable_jemalloc])
AM_CONDITIONAL([JEMALLOC], [test "x$have_jemalloc" = "xyes" && test "x$enable_jemalloc" = "xyes" ]) AM_CONDITIONAL([JEMALLOC], [test "x$have_jemalloc" = "xyes" && test "x$enable_jemalloc" = "xyes" ])
dnl dnl
@ -2500,16 +2598,15 @@ dnl pop
CPPFLAGS=$restore_cppflags CPPFLAGS=$restore_cppflags
LDFLAGS=$restore_ldflags LDFLAGS=$restore_ldflags
AC_MSG_CHECKING(whether to disable use of any found ROCm) AC_MSG_CHECKING(whether to enable use of any found ROCm)
AC_ARG_ENABLE(rocm, RB_HELP_STRING([--disable-rocm], [Disable ROCm support]), AC_ARG_ENABLE(rocm, RB_HELP_STRING([--enable-rocm], [Enable ROCm support]),
[ [
AC_MSG_RESULT([$enableval])
enable_rocm=$enableval enable_rocm=$enableval
], [ ], [
AC_MSG_RESULT([no])
enable_rocm="yes" enable_rocm="yes"
]) ])
AC_MSG_RESULT([$enable_rocm])
AM_CONDITIONAL([ROCM], [test "x$have_rocm" = "xyes" && test "x$enable_rocm" = "xyes" ]) AM_CONDITIONAL([ROCM], [test "x$have_rocm" = "xyes" && test "x$enable_rocm" = "xyes" ])
AM_COND_IF([ROCM], AM_COND_IF([ROCM],
@ -2566,16 +2663,15 @@ dnl pop
CPPFLAGS=$restore_cppflags CPPFLAGS=$restore_cppflags
LDFLAGS=$restore_ldflags LDFLAGS=$restore_ldflags
AC_MSG_CHECKING(whether to disable use of any found GPA) AC_MSG_CHECKING(whether to enable use of any found GPA)
AC_ARG_ENABLE(gpa, RB_HELP_STRING([--disable-gpa], [Disable GPA support]), AC_ARG_ENABLE(gpa, RB_HELP_STRING([--enable-gpa], [Enable GPA support]),
[ [
AC_MSG_RESULT([$enableval])
enable_gpa=$enableval enable_gpa=$enableval
], [ ], [
AC_MSG_RESULT([no])
enable_gpa="yes" enable_gpa="yes"
]) ])
AC_MSG_RESULT([$enable_gpa])
AM_CONDITIONAL([GPA], [test "x$have_gpa" = "xyes" && test "x$enable_gpa" = "xyes" ]) AM_CONDITIONAL([GPA], [test "x$have_gpa" = "xyes" && test "x$enable_gpa" = "xyes" ])
AM_COND_IF([GPA], AM_COND_IF([GPA],
@ -2623,22 +2719,19 @@ PKG_CHECK_MODULES(OpenCL, [OpenCL],
]) ])
]) ])
AC_MSG_CHECKING(whether to disable use of any found OpenCL) AC_MSG_CHECKING(whether to enable use of any found OpenCL)
AC_ARG_ENABLE(opencl, RB_HELP_STRING([--disable-opencl], [Disable OpenCL support]), AC_ARG_ENABLE(opencl, RB_HELP_STRING([--enable-opencl], [Enable OpenCL support]),
[ [
AC_MSG_RESULT([$enableval])
enable_opencl=$enableval enable_opencl=$enableval
], [ ], [
AC_MSG_RESULT([no])
enable_opencl="yes" enable_opencl="yes"
]) ])
AC_MSG_CHECKING(whether to enable use of any found OpenCL) AC_MSG_RESULT([$enable_opencl])
AM_CONDITIONAL([OPENCL], [test "x$have_opencl" = "xyes" && test "x$enable_opencl" = "xyes" ]) AM_CONDITIONAL([OPENCL], [test "x$have_opencl" = "xyes" && test "x$enable_opencl" = "xyes" ])
AM_COND_IF([OPENCL], AM_COND_IF([OPENCL],
[ [
AC_MSG_RESULT([yes])
IRCD_DEFINE(USE_OPENCL, [1], [OpenCL support is available and enabled]) IRCD_DEFINE(USE_OPENCL, [1], [OpenCL support is available and enabled])
RB_VAR_APPEND([OPENCL_LIBS], ["-lOpenCL"]) RB_VAR_APPEND([OPENCL_LIBS], ["-lOpenCL"])
], [ ], [
@ -2683,15 +2776,14 @@ PKG_CHECK_MODULES(armnn, [armnn],
]) ])
AC_MSG_CHECKING(whether to enable use of any found ArmNN) AC_MSG_CHECKING(whether to enable use of any found ArmNN)
AC_ARG_ENABLE(armnn, RB_HELP_STRING([--disable-armnn], [Disable Arm NN support]), AC_ARG_ENABLE(armnn, RB_HELP_STRING([--enable-armnn], [Enable Arm NN support]),
[ [
AC_MSG_RESULT([$enableval])
enable_armnn=$enableval enable_armnn=$enableval
], [ ], [
AC_MSG_RESULT([yes])
enable_armnn="yes" enable_armnn="yes"
]) ])
AC_MSG_RESULT([$enable_armnn])
AM_CONDITIONAL([ARMNN], [test "x$have_armnn" = "xyes" && test "x$enable_armnn" = "xyes" ]) AM_CONDITIONAL([ARMNN], [test "x$have_armnn" = "xyes" && test "x$enable_armnn" = "xyes" ])
AM_COND_IF([ARMNN], AM_COND_IF([ARMNN],

View file

@ -38,7 +38,6 @@ bool yes6;
bool norun; bool norun;
bool nomain; bool nomain;
bool read_only; bool read_only;
bool write_avoid;
bool slave; bool slave;
std::array<bool, 6> smoketest; std::array<bool, 6> smoketest;
bool megatest; bool megatest;
@ -51,6 +50,7 @@ const char *diagnostic;
bool nobanner; bool nobanner;
bool silentmode; bool silentmode;
bool noiouct; bool noiouct;
bool noioust;
lgetopt opts[] lgetopt opts[]
{ {
@ -76,7 +76,6 @@ lgetopt opts[]
{ "norun", &norun, lgetopt::BOOL, "[debug] Initialize but never run the event loop" }, { "norun", &norun, lgetopt::BOOL, "[debug] Initialize but never run the event loop" },
{ "nomain", &nomain, lgetopt::BOOL, "[debug] Initialize and run without entering ircd::main()" }, { "nomain", &nomain, lgetopt::BOOL, "[debug] Initialize and run without entering ircd::main()" },
{ "ro", &read_only, lgetopt::BOOL, "Read-only mode. No writes to database allowed" }, { "ro", &read_only, lgetopt::BOOL, "Read-only mode. No writes to database allowed" },
{ "wa", &write_avoid, lgetopt::BOOL, "Like read-only mode, but writes permitted if triggered" },
{ "slave", &slave, lgetopt::BOOL, "Like read-only mode; allows multiple instances of server" }, { "slave", &slave, lgetopt::BOOL, "Like read-only mode; allows multiple instances of server" },
{ "smoketest", &smoketest[0], lgetopt::BOOL, "Starts and stops the daemon to return success" }, { "smoketest", &smoketest[0], lgetopt::BOOL, "Starts and stops the daemon to return success" },
{ "megatest", &megatest, lgetopt::BOOL, "Trap execution every millionth tick for diagnostic and statistics." }, { "megatest", &megatest, lgetopt::BOOL, "Trap execution every millionth tick for diagnostic and statistics." },
@ -89,6 +88,7 @@ lgetopt opts[]
{ "nobanner", &nobanner, lgetopt::BOOL, "Terminal log enabled only in runlevel RUN" }, { "nobanner", &nobanner, lgetopt::BOOL, "Terminal log enabled only in runlevel RUN" },
{ "silent", &silentmode, lgetopt::BOOL, "Like quiet mode without console output either" }, { "silent", &silentmode, lgetopt::BOOL, "Like quiet mode without console output either" },
{ "noiouct", &noiouct, lgetopt::BOOL, "Disable experimental IORING_SETUP_COOP_TASKRUN" }, { "noiouct", &noiouct, lgetopt::BOOL, "Disable experimental IORING_SETUP_COOP_TASKRUN" },
{ "noioust", &noioust, lgetopt::BOOL, "Disable experimental IORING_SETUP_SINGLE_ISSUER" },
{ nullptr, nullptr, lgetopt::STRING, nullptr }, { nullptr, nullptr, lgetopt::STRING, nullptr },
}; };
@ -558,10 +558,11 @@ applyargs()
ircd::db::auto_deletion.set("false"); ircd::db::auto_deletion.set("false");
} }
if(single && !bootstrap) if(single)
{ {
ircd::write_avoid.set("true"); ircd::maintenance.set("true");
cmdline = !debugmode; cmdline = !debugmode;
nobackfill = true;
} }
if(bootstrap) if(bootstrap)
@ -572,21 +573,12 @@ applyargs()
if(slave) if(slave)
{ {
ircd::db::open_slave.set("true"); ircd::slave.set("true");
read_only = true; // slave implies read_only read_only = true; // slave implies read_only
} }
if(read_only) if(read_only)
{
ircd::read_only.set("true"); ircd::read_only.set("true");
write_avoid = true; // read_only implies write_avoid.
}
if(write_avoid)
{
ircd::write_avoid.set("true");
nobackfill = true;
}
if(debugmode) if(debugmode)
ircd::debugmode.set("true"); ircd::debugmode.set("true");
@ -693,17 +685,28 @@ __wrap_io_uring_queue_init(unsigned entries,
{ {
namespace info = ircd::info; namespace info = ircd::info;
#if defined(IORING_SETUP_COOP_TASKRUN)
const bool have_coop_taskrun const bool have_coop_taskrun
{ {
info::kernel_version[0] > 5 || info::kernel_version[0] > 5 ||
(info::kernel_version[0] >= 5 && info::kernel_version[1] >= 19) (info::kernel_version[0] >= 5 && info::kernel_version[1] >= 19)
}; };
#if defined(IORING_SETUP_COOP_TASKRUN)
if(have_coop_taskrun && !noiouct) if(have_coop_taskrun && !noiouct)
flags |= IORING_SETUP_COOP_TASKRUN; flags |= IORING_SETUP_COOP_TASKRUN;
#endif #endif
#if defined(IORING_SETUP_SINGLE_ISSUER)
const bool have_single_issuer
{
info::kernel_version[0] > 6 ||
(info::kernel_version[0] >= 6 && info::kernel_version[1] >= 0)
};
if(have_single_issuer && !noioust)
flags |= IORING_SETUP_SINGLE_ISSUER;
#endif
struct io_uring_params params struct io_uring_params params
{ {
.flags = flags, .flags = flags,

View file

@ -15,6 +15,9 @@
namespace fs = ircd::fs; namespace fs = ircd::fs;
using ircd::string_view; using ircd::string_view;
decltype(construct::homeserver::primary)
construct::homeserver::primary;
construct::homeserver::homeserver(struct ircd::m::homeserver::opts opts) construct::homeserver::homeserver(struct ircd::m::homeserver::opts opts)
try try
:opts :opts
@ -45,6 +48,8 @@ try
} }
} }
{ {
assert(!primary);
primary = this;
} }
catch(const std::exception &e) catch(const std::exception &e)
{ {
@ -59,4 +64,6 @@ catch(const std::exception &e)
construct::homeserver::~homeserver() construct::homeserver::~homeserver()
noexcept noexcept
{ {
assert(primary);
primary = nullptr;
} }

View file

@ -23,4 +23,6 @@ struct construct::homeserver
public: public:
homeserver(struct ircd::m::homeserver::opts); homeserver(struct ircd::m::homeserver::opts);
~homeserver() noexcept; ~homeserver() noexcept;
static homeserver *primary;
}; };

View file

@ -8,9 +8,10 @@
// copyright notice and this permission notice is present in all copies. The // copyright notice and this permission notice is present in all copies. The
// full license for this software is available in the LICENSE file. // full license for this software is available in the LICENSE file.
#include <ircd/ircd.h> // must include because asio.h.gch is fPIC #include <ircd/matrix.h>
#include <ircd/asio.h> #include <ircd/asio.h>
#include "construct.h" #include "construct.h"
#include "homeserver.h"
#include "signals.h" #include "signals.h"
#include "console.h" #include "console.h"
@ -19,6 +20,7 @@ namespace construct
namespace ph = std::placeholders; namespace ph = std::placeholders;
static void handle_cont(); static void handle_cont();
static void handle_usr2();
static void handle_usr1(); static void handle_usr1();
static void handle_quit(); static void handle_quit();
static void handle_interrupt(); static void handle_interrupt();
@ -41,6 +43,7 @@ construct::signals::signals(boost::asio::io_context &ios)
signal_set->add(SIGQUIT); signal_set->add(SIGQUIT);
signal_set->add(SIGTERM); signal_set->add(SIGTERM);
signal_set->add(SIGUSR1); signal_set->add(SIGUSR1);
signal_set->add(SIGUSR2);
signal_set->add(SIGCONT); signal_set->add(SIGCONT);
set_handle(); set_handle();
} }
@ -132,6 +135,7 @@ construct::handle_signal(const int &signum)
case SIGQUIT: return handle_quit(); case SIGQUIT: return handle_quit();
case SIGTERM: return handle_quit(); case SIGTERM: return handle_quit();
case SIGUSR1: return handle_usr1(); case SIGUSR1: return handle_usr1();
case SIGUSR2: return handle_usr2();
case SIGCONT: return handle_cont(); case SIGCONT: return handle_cont();
default: break; default: break;
} }
@ -225,24 +229,73 @@ try
return; return;
} }
if(!homeserver::primary || !homeserver::primary->module[0])
return;
// This signal handler (though not a *real* signal handler) is still // This signal handler (though not a *real* signal handler) is still
// running on the main async stack and not an ircd::ctx. The reload // running on the main async stack and not an ircd::ctx. The reload
// function does a lot of IO so it requires an ircd::ctx. // function does a lot of IO so it requires an ircd::ctx.
ircd::context{[] ircd::context{[]
{ {
ircd::mods::import<void ()> reload_conf static ircd::mods::import<void (ircd::m::homeserver *)> rehash
{ {
"s_conf", "reload_conf" homeserver::primary->module[0], "ircd::m::homeserver::rehash"
}; };
reload_conf(); assert(homeserver::primary->hs);
rehash(homeserver::primary->hs.get());
}}; }};
} }
catch(const std::exception &e) catch(const std::exception &e)
{ {
ircd::log::error ircd::log::error
{ {
"SIGUSR1 handler: %s", e.what() "SIGUSR1 handler :%s",
e.what()
};
}
void
construct::handle_usr2()
try
{
if(ircd::run::level != ircd::run::level::RUN)
{
ircd::log::warning
{
"Not synchronizing database from SIGUSR2 in runlevel %s",
reflect(ircd::run::level)
};
return;
}
if(!ircd::slave)
return;
if(!homeserver::primary || !homeserver::primary->module[0])
return;
ircd::context{[]
{
static ircd::mods::import<bool (ircd::m::homeserver *)> refresh
{
homeserver::primary->module[0], "ircd::m::homeserver::refresh"
};
assert(homeserver::primary->hs);
const bool refreshed
{
refresh(homeserver::primary->hs.get())
};
}};
}
catch(const std::exception &e)
{
ircd::log::error
{
"SIGUSR2 handler :%s",
e.what()
}; };
} }

View file

@ -1,4 +1,7 @@
FROM --platform=$TARGETPLATFORM alpine:3.16.1 ARG dist_name alpine
ARG dist_version 3.16
FROM --platform=$TARGETPLATFORM ${dist_name}:${dist_version}
ENV packages="\ ENV packages="\
boost-chrono \ boost-chrono \

View file

@ -1,17 +1,20 @@
ARG acct ARG acct
ARG repo ARG repo
ARG dist_name
ARG dist_version
ARG feature ARG feature
ARG machine
FROM ${acct}/${repo}:alpine-3.16-${feature}-${TARGETARCH} FROM ${acct}/${repo}:${dist_name}-${dist_version}-${feature}-${machine}
ARG cc ARG cc
ARG cxx ARG cxx
ARG extra_packages_dev ARG extra_packages_dev
ARG extra_packages_dev1
ARG extra_packages_dev2
ARG rocksdb_version 7.4.3 ARG rocksdb_version 7.4.3
ARG rocksdb_url ARG rocksdb_url
ARG ctor_url https://github.com/matrix-construct/construct ARG ctor_url https://github.com/matrix-construct/construct
ARG machine_spec
ARG nprocs
ENV CC ${cc} ENV CC ${cc}
ENV CXX ${cxx} ENV CXX ${cxx}
@ -20,12 +23,13 @@ ENV CONFIG_SHELL bash
ENV rocksdb_version ${rocksdb_version} ENV rocksdb_version ${rocksdb_version}
ENV rocksdb_url https://codeload.github.com/facebook/rocksdb/tar.gz/refs/tags/v${rocksdb_version} ENV rocksdb_url https://codeload.github.com/facebook/rocksdb/tar.gz/refs/tags/v${rocksdb_version}
ENV ctor_url ${ctor_url} ENV ctor_url ${ctor_url}
ENV machine_spec ${machine_spec}
ENV nprocs ${nprocs}
ENV packages_dev="\ ENV packages_dev="\
${packages_dev} \ ${packages_dev} \
autoconf \ autoconf \
autoconf-archive \ autoconf-archive \
autoconf2.13 \
automake \ automake \
bash \ bash \
binutils-gold \ binutils-gold \
@ -34,8 +38,6 @@ curl \
git \ git \
libtool \ libtool \
${extra_packages_dev} \ ${extra_packages_dev} \
${extra_packages_dev1} \
${extra_packages_dev2} \
" "
WORKDIR /usr/src WORKDIR /usr/src
@ -50,8 +52,8 @@ RUN true \
&& ln -sv /usr/src/rocksdb construct/deps/rocksdb \ && ln -sv /usr/src/rocksdb construct/deps/rocksdb \
&& cd /usr/src/construct \ && cd /usr/src/construct \
&& ./autogen.sh \ && ./autogen.sh \
&& (./configure --disable-iou || (cat config.log; exit 1)) \ && (./configure --disable-iou --enable-generic --with-machine="${machine_spec}" || (cat config.log; exit 1)) \
&& make -j `nproc` EXTRA_LDFLAGS="-Wl,--strip-all" install \ && make -j ${nprocs} EXTRA_LDFLAGS="-Wl,--strip-all" install \
&& rm -rf /usr/src/rocksdb \ && rm -rf /usr/src/rocksdb \
&& rm -rf /usr/src/construct \ && rm -rf /usr/src/construct \
&& rm -rf /usr/include/ircd \ && rm -rf /usr/include/ircd \

View file

@ -1,7 +1,10 @@
ARG acct ARG acct
ARG repo ARG repo
ARG dist_name
ARG dist_version
ARG machine
FROM ${acct}/${repo}:alpine-3.16-base-${TARGETARCH} FROM ${acct}/${repo}:${dist_name}-${dist_version}-base-${machine}
ENV packages="\ ENV packages="\
freetype \ freetype \

View file

@ -1,9 +1,12 @@
ARG acct ARG acct
ARG repo ARG repo
ARG dist_name
ARG dist_version
ARG feature ARG feature
ARG cc ARG cc
ARG machine
FROM ${acct}/${repo}:alpine-3.16-${feature}-built-${cc}-${TARGETARCH} FROM ${acct}/${repo}:${dist_name}-${dist_version}-${feature}-built-${cc}-${machine}
ENV ircd_mods_unload_check=false ENV ircd_mods_unload_check=false
WORKDIR /root WORKDIR /root

View file

@ -1,181 +1,283 @@
#!/bin/sh #!/bin/sh
BASEDIR=$(dirname "$0") # Sundry configuration
ACCT=jevolk
REPO=construct
ctor_url="https://github.com/matrix-construct/construct"
rocksdb_version=7.4.3
ARGS_="" BASEDIR=$(dirname "$0")
ARGS_="$ARGS_ --compress=true" stage=$1
ARGS_="$ARGS_ --build-arg acct=$ACCT" mode=$2
ARGS_="$ARGS_ --build-arg repo=$REPO"
ARGS_="$ARGS_ --build-arg ctor_url=$ctor_url"
ARGS_="$ARGS_ --build-arg rocksdb_version=$rocksdb_version"
export DOCKER_BUILDKIT=1 export DOCKER_BUILDKIT=1
#export BUILDKIT_PROGRESS=plain if test "$CI" = true; then
export BUILDKIT_PROGRESS="plain"
echo "plain"
fi
# The stage argument can be "base" "full" "built" "test" or "push"
default_stage="push"
stage=${stage:=$default_stage}
# The mode argument can be "real" or "test"
default_mode="real"
mode=${mode:=$default_mode}
if test "$ctor_test" = true; then
mode="test"
fi
# Account configuration environment.
#
# Override these for yourself.
default_ctor_url="https://github.com/jevolk/charybdis"
default_ctor_id="jevolk/construct"
ctor_url=${ctor_url:=$default_ctor_url}
ctor_id=${ctor_id:=$default_ctor_id}
ctor_acct=${ctor_acct:=$(echo $ctor_id | cut -d"/" -f1)}
ctor_repo=${ctor_repo:=$(echo $ctor_id | cut -d"/" -f2)}
# Job matrix configuration environment
#
# All combinations of these arrays will be run by this script. Overriding
# these with one element for each variable allows this script to do one thing
# at a time.
default_ctor_features="base full"
default_ctor_distros="ubuntu-22.04 ubuntu-22.10 alpine-3.17"
default_ctor_machines="amd64 amd64-avx amd64-avx512"
default_ctor_toolchains="gcc-10 gcc-11 gcc-12 clang-14"
ctor_features=${ctor_features:=$default_ctor_features}
ctor_distros=${ctor_distros:=$default_ctor_distros}
ctor_machines=${ctor_machines:=$default_ctor_machines}
ctor_toolchains=${ctor_toolchains:=$default_ctor_toolchains}
############################################################################### ###############################################################################
#
# Alpine 3.16
#
# matrix()
# L0 - Base featured image {
# for ctor_feature in $ctor_features; do
for ctor_distro in $ctor_distros; do
for ctor_machine in $ctor_machines; do
for ctor_toolchain in $ctor_toolchains; do
build $ctor_feature $ctor_distro $ctor_machine $ctor_toolchain
if test $? -ne 0; then return 1; fi
done
done
done
done
return 0
}
ARGS="$ARGS_" build()
ARGS="$ARGS --platform linux/amd64" {
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-base-amd64 $BASEDIR/alpine/3.16/base feature=$1
distro=$2
machine=$3
toolchain=$4
dist_name=$(echo $distro | cut -d"-" -f1)
dist_version=$(echo $distro | cut -d"-" -f2)
runner_name=$(echo $RUNNER_NAME | cut -d"." -f1)
runner_num=$(echo $RUNNER_NAME | cut -d"." -f2)
# args="$ctor_docker_build_args"
# L1 - Fully featured image args="$args --compress=true"
#
ARGS="$ARGS_" if test ! -z "$runner_num"; then
ARGS="$ARGS --platform linux/amd64" cpu_num=$(expr $runner_num % $(nproc))
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-full-amd64 $BASEDIR/alpine/3.16/full args="$args --cpuset-cpus=${cpu_num}"
args="$args --build-arg nprocs=1"
# https://github.com/moby/buildkit/issues/1276
else
nprocs=$(nproc)
args="$args --build-arg nprocs=${nprocs}"
fi
# args="$args --build-arg acct=${ctor_acct}"
# L2/L3 - Built/Test images args="$args --build-arg repo=${ctor_repo}"
# args="$args --build-arg ctor_url=${ctor_url}"
args="$args --build-arg dist_name=${dist_name}"
args="$args --build-arg dist_version=${dist_version}"
args="$args --build-arg machine=${machine}"
args="$args --build-arg feature=${feature}"
ARGS="$ARGS_" args_dist $dist_name $dist_version
ARGS="$ARGS --platform linux/amd64" if test $? -ne 0; then return 1; fi
ARGS="$ARGS --build-arg feature=base"
ARGS="$ARGS --build-arg extra_packages_dev=gcc"
ARGS="$ARGS --build-arg extra_packages_dev1=g++"
ARGS="$ARGS --build-arg cc=gcc --build-arg cxx=g++"
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-base-built-gcc-amd64 $BASEDIR/alpine/3.16/built
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-base-test-gcc-amd64 $BASEDIR/alpine/3.16/test
ARGS="$ARGS_" args_machine $machine
ARGS="$ARGS --platform linux/amd64" if test $? -ne 0; then return 1; fi
ARGS="$ARGS --build-arg feature=base"
ARGS="$ARGS --build-arg extra_packages_dev=clang"
ARGS="$ARGS --build-arg extra_packages_dev1=llvm"
ARGS="$ARGS --build-arg extra_packages_dev2=llvm-dev"
ARGS="$ARGS --build-arg cc=clang --build-arg cxx=clang++"
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-base-built-clang-amd64 $BASEDIR/alpine/3.16/built
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-base-test-clang-amd64 $BASEDIR/alpine/3.16/test
ARGS="$ARGS_" args_platform $machine
ARGS="$ARGS --platform linux/amd64" if test $? -ne 0; then return 1; fi
ARGS="$ARGS --build-arg feature=full"
ARGS="$ARGS --build-arg extra_packages_dev=gcc"
ARGS="$ARGS --build-arg extra_packages_dev1=g++"
ARGS="$ARGS --build-arg cc=gcc --build-arg cxx=g++"
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-full-built-gcc-amd64 $BASEDIR/alpine/3.16/built
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-full-test-gcc-amd64 $BASEDIR/alpine/3.16/test
ARGS="$ARGS_" if test $toolchain != "false"; then
ARGS="$ARGS --platform linux/amd64" args_toolchain $toolchain $dist_name $dist_version
ARGS="$ARGS --build-arg feature=full" if test $? -ne 0; then return 1; fi
ARGS="$ARGS --build-arg extra_packages_dev=clang" fi
ARGS="$ARGS --build-arg extra_packages_dev1=llvm"
ARGS="$ARGS --build-arg extra_packages_dev2=llvm-dev"
ARGS="$ARGS --build-arg cc=clang --build-arg cxx=clang++"
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-full-built-clang-amd64 $BASEDIR/alpine/3.16/built
docker build $ARGS -t $ACCT/$REPO:alpine-3.16-full-test-clang-amd64 $BASEDIR/alpine/3.16/test
# if test $mode = "test"; then
# Pushed images cmd=$(which echo)
# else
cmd=$(which docker)
fi
docker push $ACCT/$REPO:alpine-3.16-base-built-gcc-amd64 # Intermediate stage build; usually cached from prior iteration.
docker push $ACCT/$REPO:alpine-3.16-base-built-clang-amd64 tag="$ctor_acct/$ctor_repo:${distro}-${feature}-${machine}"
arg="$args -t $tag $BASEDIR/${dist_name}/${feature}"
eval "$cmd build $arg"
if test $? -ne 0; then return 1; fi
if test $stage = $feature; then return 0; fi
if test $toolchain = "false"; then return 0; fi
docker push $ACCT/$REPO:alpine-3.16-full-built-gcc-amd64 # Leaf build; unique to each iteration.
docker push $ACCT/$REPO:alpine-3.16-full-built-clang-amd64 tag="$ctor_acct/$ctor_repo:${distro}-${feature}-built-${toolchain}-${machine}"
arg="$args --no-cache -t $tag $BASEDIR/${dist_name}/built"
eval "$cmd build $arg"
if test $? -ne 0; then return 1; fi
if test $stage = "built"; then return 0; fi
############################################################################### # Test built;
# arg="$args $BASEDIR/${dist_name}/test"
# Ubuntu 22.04 eval "$cmd build $arg"
# if test $? -ne 0; then return 1; fi
if test $stage = "test"; then return 0; fi
# # Push built
# L0 - Base featured image eval "$cmd push $tag"
# if test $? -ne 0; then return 1; fi
ARGS="$ARGS_" return 0
ARGS="$ARGS --platform linux/amd64" }
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-amd64 $BASEDIR/ubuntu/22.04/base
# args_dist()
# L1 - Fully featured image {
# case $1 in
alpine)
case $2 in
3.16)
args="$args --build-arg rocksdb_version=7.2.2"
return 0
;;
3.17)
args="$args --build-arg rocksdb_version=7.7.3"
return 0
;;
esac
;;
ubuntu)
case $2 in
22.04)
args="$args --build-arg rocksdb_version=7.4.3"
args="$args --build-arg boost_version=1.74"
args="$args --build-arg icu_version=70"
return 0
;;
22.10)
args="$args --build-arg rocksdb_version=7.10.2"
args="$args --build-arg boost_version=1.74"
args="$args --build-arg icu_version=71"
return 0
;;
esac
;;
esac
return 1
}
ARGS="$ARGS_" args_toolchain()
ARGS="$ARGS --platform linux/amd64" {
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-full-amd64 $BASEDIR/ubuntu/22.04/full _name=$(echo $1 | cut -d"-" -f1)
_epoch=$(echo $1 | cut -d"-" -f2)
# case $2 in
# L2/L3 - Build/Built/Test alpine)
# toolchain=$_name
case $1 in
gcc*)
extra_dev="gcc"
extra_dev="${extra_dev} g++"
args="$args --build-arg extra_packages_dev=\"${extra_dev}\""
args="$args --build-arg cc=gcc --build-arg cxx=g++"
return 0
;;
clang*)
extra_dev="clang"
extra_dev="${extra_dev} llvm"
extra_dev="${extra_dev} llvm-dev"
args="$args --build-arg extra_packages_dev=\"${extra_dev}\""
args="$args --build-arg cc=clang --build-arg cxx=clang++"
test $3 != "3.16"
return $?
;;
esac
;;
ubuntu)
case $1 in
gcc*)
extra_dev="gcc-${_epoch}"
extra_dev="${extra_dev} g++-${_epoch}"
args="$args --build-arg extra_packages_dev=\"${extra_dev}\""
args="$args --build-arg cc=gcc-${_epoch} --build-arg cxx=g++-${_epoch}"
return 0
;;
clang*)
if test "$_epoch" -ge 15; then
extra="mesa-opencl-icd"
extra="${extra} ocl-icd-opencl-dev"
extra="${extra} libclc-${_epoch}-dev"
args="$args --build-arg extra_packages=\"${extra}\""
fi
extra_dev="clang-${_epoch}"
extra_dev="${extra_dev} llvm-${_epoch}"
extra_dev="${extra_dev} llvm-spirv-${_epoch}"
extra_dev="${extra_dev} libclc-${_epoch}-dev"
args="$args --build-arg extra_packages_dev=\"${extra_dev}\""
args="$args --build-arg cc=clang-${_epoch} --build-arg cxx=clang++-${_epoch}"
return 0
;;
esac
;;
esac
return 1
}
ARGS="$ARGS_" args_machine()
ARGS="$ARGS --platform linux/amd64" {
ARGS="$ARGS --build-arg feature=base" case $1 in
ARGS="$ARGS --build-arg extra_packages_dev=gcc-9" amd64)
ARGS="$ARGS --build-arg extra_packages_dev1=g++-9" args="$args --build-arg machine_spec=arch=amdfam10"
ARGS="$ARGS --build-arg cc=gcc-9 --build-arg cxx=g++-9" ;;
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-build-gcc-9-amd64 $BASEDIR/ubuntu/22.04/build amd64-avx)
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-built-gcc-9-amd64 $BASEDIR/ubuntu/22.04/built args="$args --build-arg machine_spec=arch=sandybridge"
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-test-gcc-9-amd64 $BASEDIR/ubuntu/22.04/test args="$args --build-arg rocksdb_avx=1"
;;
amd64-avx2)
args="$args --build-arg machine_spec=arch=haswell"
args="$args --build-arg rocksdb_avx2=1"
;;
amd64-avx512)
args="$args --build-arg machine_spec=arch=skylake-avx512"
args="$args --build-arg rocksdb_avx2=1"
;;
esac
return 0
}
ARGS="$ARGS_" args_platform()
ARGS="$ARGS --platform linux/amd64" {
ARGS="$ARGS --build-arg feature=base" case $1 in
ARGS="$ARGS --build-arg extra_packages_dev=gcc-12" amd64*)
ARGS="$ARGS --build-arg extra_packages_dev1=g++-12" args="$args --platform linux/amd64"
ARGS="$ARGS --build-arg cc=gcc-12 --build-arg cxx=g++-12" test $(uname -m) = "x86_64"
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-build-gcc-12-amd64 $BASEDIR/ubuntu/22.04/build return $?
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-built-gcc-12-amd64 $BASEDIR/ubuntu/22.04/built ;;
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-base-test-gcc-12-amd64 $BASEDIR/ubuntu/22.04/test arm64*)
args="$args --platform linux/arm64"
test $(uname -m) = "aarch64"
return $?
;;
esac
return 1
}
ARGS="$ARGS_" matrix
ARGS="$ARGS --platform linux/amd64"
ARGS="$ARGS --build-arg feature=full"
ARGS="$ARGS --build-arg extra_packages_dev=clang-15"
ARGS="$ARGS --build-arg extra_packages_dev1=llvm-15-dev"
ARGS="$ARGS --build-arg extra_packages_dev2=llvm-spirv-15"
ARGS="$ARGS --build-arg cc=clang-15 --build-arg cxx=clang++-15"
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-full-build-clang-15-amd64 $BASEDIR/ubuntu/22.04/build
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-full-built-clang-15-amd64 $BASEDIR/ubuntu/22.04/built
docker build $ARGS -t $ACCT/$REPO:ubuntu-22.04-full-test-clang-15-amd64 $BASEDIR/ubuntu/22.04/test
#
# Pushed images
#
docker push $ACCT/$REPO:ubuntu-22.04-base-build-gcc-9-amd64
docker push $ACCT/$REPO:ubuntu-22.04-base-built-gcc-9-amd64
docker push $ACCT/$REPO:ubuntu-22.04-base-build-gcc-12-amd64
docker push $ACCT/$REPO:ubuntu-22.04-base-built-gcc-12-amd64
docker push $ACCT/$REPO:ubuntu-22.04-full-build-clang-15-amd64
docker push $ACCT/$REPO:ubuntu-22.04-full-built-clang-15-amd64
###############################################################################
#
# Ubuntu 20.04
#
#docker build -t $ACCT/$REPO:ubuntu-20.04 $BASEDIR/ubuntu/20.04/base
#docker build -t $ACCT/$REPO:ubuntu-20.04-gcc-8 $BASEDIR/ubuntu/20.04/gcc-8
#docker build -t $ACCT/$REPO:ubuntu-20.04-gcc-9 $BASEDIR/ubuntu/20.04/gcc-9
#docker build -t $ACCT/$REPO:ubuntu-20.04-gcc-10 $BASEDIR/ubuntu/20.04/gcc-10
#docker build -t $ACCT/$REPO:ubuntu-20.04-clang-9 $BASEDIR/ubuntu/20.04/clang-9
#docker build -t $ACCT/$REPO:ubuntu-20.04-clang-10 $BASEDIR/ubuntu/20.04/clang-10
#docker build -t $ACCT/$REPO:ubuntu-20.04-clang-11 $BASEDIR/ubuntu/20.04/clang-11
#docker build -t $ACCT/$REPO:ubuntu-20.04-clang-12 $BASEDIR/ubuntu/20.04/clang-12
#docker push $ACCT/$REPO:ubuntu-20.04-gcc-8
#docker push $ACCT/$REPO:ubuntu-20.04-gcc-9
#docker push $ACCT/$REPO:ubuntu-20.04-gcc-10
#docker push $ACCT/$REPO:ubuntu-20.04-clang-9
#docker push $ACCT/$REPO:ubuntu-20.04-clang-10
#docker push $ACCT/$REPO:ubuntu-20.04-clang-11
#docker push $ACCT/$REPO:ubuntu-20.04-clang-12

View file

@ -1,60 +0,0 @@
FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN \
apt-get update && \
apt-get install --no-install-recommends -y software-properties-common && \
apt-get update && \
apt-get install --no-install-recommends -y \
autoconf \
autoconf-archive \
autoconf2.13 \
automake \
autotools-dev \
boost1.71 \
build-essential \
cmake \
curl \
git \
libbz2-dev \
liblz4-dev \
libgraphicsmagick1-dev \
libicu-dev \
libjemalloc-dev \
libmagic-dev \
libnss-db \
libsodium-dev \
libssl-dev \
libtool \
libzstd-dev \
shtool \
xz-utils && \
apt-get purge -y software-properties-common && \
apt-get clean && \
apt-get autoremove --purge -y && \
rm -rf /var/lib/apt/lists/*
ENV ROCKSDB_VERSION=6.6.4
RUN \
cd /usr/src && \
curl -sL https://github.com/facebook/rocksdb/archive/v${ROCKSDB_VERSION}.tar.gz -o rocksdb-${ROCKSDB_VERSION}.tar.gz && \
tar xfvz rocksdb-${ROCKSDB_VERSION}.tar.gz && \
rm rocksdb-${ROCKSDB_VERSION}.tar.gz && \
ln -s /usr/src/rocksdb-${ROCKSDB_VERSION} /usr/src/rocksdb && \
cd /usr/src/rocksdb-${ROCKSDB_VERSION} && \
CFLAGS="-g0" \
cmake -H. -Bbuild \
-DCMAKE_BUILD_TYPE=Release \
-DWITH_TESTS=0 \
-DWITH_TOOLS=0 \
-DUSE_RTTI=1 \
-DWITH_LZ4=1 \
-DWITH_GFLAGS=0 \
-DBUILD_SHARED_LIBS=1 && \
cmake --build build --target install && \
rm -rf build
RUN mkdir /build
WORKDIR /build

View file

@ -1,10 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CC clang-10
ENV CXX clang++-10
RUN apt-get update \
&& apt-get install --no-install-recommends -y clang-10 llvm-10-dev \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,10 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CC clang-11
ENV CXX clang++-11
RUN apt-get update \
&& apt-get install --no-install-recommends -y clang-11 llvm-11-dev \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,10 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CC clang-12
ENV CXX clang++-12
RUN apt-get update \
&& apt-get install --no-install-recommends -y clang-12 llvm-12-dev \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,10 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CC clang-9
ENV CXX clang++-9
RUN apt-get update \
&& apt-get install --no-install-recommends -y clang-9 llvm-9-dev \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,11 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CXX g++-10
ENV CC gcc-10
RUN apt-get update \
&& apt-get install --no-install-recommends -y software-properties-common \
&& apt-get update \
&& apt-get install --no-install-recommends -y g++-10 gcc-10 \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,9 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CXX g++-8
ENV CC gcc-8
RUN apt-get update \
&& apt-get install --no-install-recommends -y g++-8 gcc-8 \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,11 +0,0 @@
FROM jevolk/construct:ubuntu-20.04
ENV CXX g++-9
ENV CC gcc-9
RUN apt-get update \
&& apt-get install --no-install-recommends -y software-properties-common \
&& apt-get update \
&& apt-get install --no-install-recommends -y g++-9 gcc-9 \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*

View file

@ -1,35 +0,0 @@
ARG acct
ARG repo
ARG feature
FROM ${acct}/${repo}:ubuntu-22.04-${feature}-${TARGETARCH}
ARG cc
ARG cxx
ARG extra_packages_dev
ARG extra_packages_dev1
ARG extra_packages_dev2
ARG ctor_url https://github.com/matrix-construct/construct
ENV CC ${cc}
ENV CXX ${cxx}
ENV ctor_url ${ctor_url}
ENV packages_dev="\
${packages_dev} \
autoconf \
autoconf-archive \
autoconf2.13 \
automake \
autotools-dev \
libtool \
shtool \
${extra_packages_dev} \
${extra_packages_dev1} \
${extra_packages_dev2} \
"
RUN true \
&& eval ${do_install} ${packages_dev} \
&& eval ${do_clean} \
&& true

View file

@ -1,11 +0,0 @@
ARG acct
ARG repo
ARG feature
ARG cc
FROM ${acct}/${repo}:ubuntu-22.04-${feature}-built-${cc}-${TARGETARCH}
WORKDIR /root
RUN true \
&& construct -debug -smoketest localhost \
&& true

View file

@ -1,22 +1,35 @@
FROM ubuntu:22.04 ARG dist_name ubuntu
ARG dist_version 22.04
FROM --platform=$TARGETPLATFORM ${dist_name}:${dist_version}
ARG skiprocks ARG skiprocks
ARG rocksdb_version 7.4.5 ARG rocksdb_version 7.4.5
ARG rocksdb_url ARG rocksdb_url
ARG rocksdb_avx 0
ARG rocksdb_avx2 0
ARG boost_version 1.74
ARG icu_version 70
ARG nprocs 1
ENV rocksdb_version ${rocksdb_version} ENV rocksdb_version ${rocksdb_version}
ENV rocksdb_url https://codeload.github.com/facebook/rocksdb/tar.gz/refs/tags/v${rocksdb_version} ENV rocksdb_url https://codeload.github.com/facebook/rocksdb/tar.gz/refs/tags/v${rocksdb_version}
ENV rocksdb_avx ${rocksdb_avx}
ENV rocksdb_avx2 ${rocksdb_avx2}
ENV boost_version ${boost_version}
ENV icu_version ${icu_version}
ENV nprocs ${nprocs}
ENV packages="\ ENV packages="\
ca-certificates \ ca-certificates \
libatomic1 \ libatomic1 \
libjemalloc2 \ libjemalloc2 \
libboost-chrono1.74 \ libboost-chrono${boost_version}.0 \
libboost-context1.74 \ libboost-context${boost_version}.0 \
libboost-coroutine1.74 \ libboost-coroutine${boost_version}.0 \
libboost-system1.74 \ libboost-system${boost_version}.0 \
libboost-thread1.74 \ libboost-thread${boost_version}.0 \
libicu70 \ libicu${icu_version} \
libnss-db \ libnss-db \
libsodium23 \ libsodium23 \
libssl3 \ libssl3 \
@ -28,8 +41,6 @@ libzstd1 \
ENV packages_rocksdb_dev="\ ENV packages_rocksdb_dev="\
build-essential \ build-essential \
cmake \ cmake \
curl \
git \
libjemalloc-dev \ libjemalloc-dev \
liblz4-dev \ liblz4-dev \
libzstd-dev \ libzstd-dev \
@ -38,11 +49,11 @@ xz-utils \
ENV packages_dev="\ ENV packages_dev="\
${packages_rocksdb_dev} \ ${packages_rocksdb_dev} \
libboost-chrono1.74-dev \ libboost-chrono${boost_version}-dev \
libboost-context1.74-dev \ libboost-context${boost_version}-dev \
libboost-coroutine1.74-dev \ libboost-coroutine${boost_version}-dev \
libboost-system1.74-dev \ libboost-system${boost_version}-dev \
libboost-thread1.74-dev \ libboost-thread${boost_version}-dev \
libicu-dev \ libicu-dev \
libmagic-dev \ libmagic-dev \
libsodium-dev \ libsodium-dev \
@ -52,19 +63,24 @@ libssl-dev \
ENV rocksdb_cmake="\ ENV rocksdb_cmake="\
-DCMAKE_RULE_MESSAGES:BOOL=OFF \ -DCMAKE_RULE_MESSAGES:BOOL=OFF \
-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \ -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=1 \
-DFAIL_ON_WARNINGS=0 \
-DUSE_RTTI=1 \
-DPORTABLE=1 \
-DFORCE_AVX=${rocksdb_avx} \
-DFORCE_AVX2=${rocksdb_avx2} \
-DWITH_JNI=0 \ -DWITH_JNI=0 \
-DWITH_TESTS=0 \ -DWITH_TESTS=0 \
-DWITH_BENCHMARK_TOOLS=0 \ -DWITH_BENCHMARK_TOOLS=0 \
-DWITH_TRACE_TOOLS=0 \
-DWITH_CORE_TOOLS=0 \ -DWITH_CORE_TOOLS=0 \
-DFAIL_ON_WARNINGS=0 \ -DWITH_TOOLS=0 \
-DCMAKE_BUILD_TYPE=Release \
-DWITH_GFLAGS=0 \ -DWITH_GFLAGS=0 \
-DWITH_LIBURING=0 \ -DWITH_LIBURING=0 \
-DWITH_JEMALLOC=1 \ -DWITH_JEMALLOC=1 \
-DWITH_LZ4=1 \ -DWITH_LZ4=1 \
-DWITH_ZSTD=1 \ -DWITH_ZSTD=1 \
-DUSE_RTTI=1 \
-DBUILD_SHARED_LIBS=1 \
" "
ENV do_install true \ ENV do_install true \
@ -83,6 +99,17 @@ ENV do_clean true \
&& rm -rf /var/lib/apt/lists/* \ && rm -rf /var/lib/apt/lists/* \
&& true && true
ENV do_fetch_rocksdb true \
&& eval ${do_install} curl \
&& cd /usr/src \
&& curl -sL ${rocksdb_url} -o rocksdb-${rocksdb_version}.tar.gz \
&& tar xfz rocksdb-${rocksdb_version}.tar.gz \
&& rm rocksdb-${rocksdb_version}.tar.gz \
&& mv /usr/src/rocksdb-${rocksdb_version} /usr/src/rocksdb \
&& cd - \
&& eval ${do_purge} curl \
&& true
RUN true \ RUN true \
&& eval ${do_install} ${packages} \ && eval ${do_install} ${packages} \
&& update-ca-certificates \ && update-ca-certificates \
@ -96,18 +123,15 @@ RUN true \
exit 0; \ exit 0; \
fi \ fi \
&& eval ${do_install} ${packages_rocksdb_dev} \ && eval ${do_install} ${packages_rocksdb_dev} \
&& cd /usr/src \ && eval ${do_fetch_rocksdb} \
&& curl -sL ${rocksdb_url} -o rocksdb-${rocksdb_version}.tar.gz \ && cd /usr/src/rocksdb \
&& tar xfvz rocksdb-${rocksdb_version}.tar.gz \
&& rm rocksdb-${rocksdb_version}.tar.gz \
&& ln -s /usr/src/rocksdb-${rocksdb_version} /usr/src/rocksdb \
&& cd /usr/src/rocksdb-${rocksdb_version} \
&& \ && \
CFLAGS="-g0 -ftls-model=initial-exec" \ CFLAGS="-g0 -ftls-model=initial-exec" \
LDFLAGS="-Wl,--strip-all" \ LDFLAGS="-Wl,--strip-all" \
cmake -H. -Bbuild ${rocksdb_cmake} \ cmake -H. -Bbuild ${rocksdb_cmake} \
&& cmake --build build --target install --parallel `nproc` \ && cmake --build build --target install --parallel ${nprocs} \
&& rm -rf build \ && rm -rf /usr/lib/$(uname -m)-linux-gnu/librocksdb.a \
&& rm -rf /usr/src/rocksdb \
&& eval ${do_purge} ${packages_rocksdb_dev} \ && eval ${do_purge} ${packages_rocksdb_dev} \
&& eval ${do_clean} \ && eval ${do_clean} \
&& true && true

View file

@ -1,19 +1,24 @@
ARG acct ARG acct
ARG repo ARG repo
ARG dist_name
ARG dist_version
ARG feature ARG feature
ARG machine
FROM ${acct}/${repo}:ubuntu-22.04-${feature}-${TARGETARCH} FROM ${acct}/${repo}:${dist_name}-${dist_version}-${feature}-${machine}
ARG cc ARG cc
ARG cxx ARG cxx
ARG extra_packages_dev ARG extra_packages_dev
ARG extra_packages_dev1
ARG extra_packages_dev2
ARG ctor_url https://github.com/matrix-construct/construct ARG ctor_url https://github.com/matrix-construct/construct
ARG machine_spec
ARG nprocs
ENV CC ${cc} ENV CC ${cc}
ENV CXX ${cxx} ENV CXX ${cxx}
ENV ctor_url ${ctor_url} ENV ctor_url ${ctor_url}
ENV machine_spec ${machine_spec}
ENV nprocs ${nprocs}
ENV packages_dev="\ ENV packages_dev="\
${packages_dev} \ ${packages_dev} \
@ -22,26 +27,26 @@ autoconf-archive \
autoconf2.13 \ autoconf2.13 \
automake \ automake \
autotools-dev \ autotools-dev \
git \
libtool \ libtool \
shtool \ shtool \
${extra_packages_dev} \ ${extra_packages_dev} \
${extra_packages_dev1} \
${extra_packages_dev2} \
" "
RUN true \ RUN true \
&& eval ${do_install} ${packages_dev} \ && eval ${do_install} ${packages_dev} \
&& eval ${do_fetch_rocksdb} \
&& git clone ${ctor_url} construct \ && git clone ${ctor_url} construct \
&& cd construct \ && cd construct \
&& rmdir -v deps/rocksdb \ && rmdir -v deps/rocksdb \
&& ln -sv /usr/src/rocksdb deps \ && ln -sv /usr/src/rocksdb deps \
&& ./autogen.sh \ && ./autogen.sh \
&& ./configure \ && (./configure --enable-generic --with-machine="${machine_spec}" || (cat config.log; exit 1)) \
&& make -j `nproc` \ && make -j ${nprocs} \
&& make install \ && make install \
&& cd .. \ && cd .. \
&& rm -rf construct \ && rm -rf construct \
&& rm -rf /usr/src/rocksdb* \ && rm -rf /usr/src/rocksdb \
&& eval ${do_purge} ${packages_dev} \ && eval ${do_purge} ${packages_dev} \
&& eval ${do_clean} \ && eval ${do_clean} \
&& true && true

View file

@ -1,13 +1,17 @@
ARG acct ARG acct
ARG repo ARG repo
ARG dist_name
ARG dist_version
ARG machine
FROM ${acct}/${repo}:ubuntu-22.04-base-${TARGETARCH} FROM ${acct}/${repo}:${dist_name}-${dist_version}-base-${machine}
ARG extra_packages
ENV packages="\ ENV packages="\
libgraphicsmagick-q16-3 \ libgraphicsmagick-q16-3 \
libpng16-16 \ libpng16-16 \
mesa-opencl-icd \ ${extra_packages} \
ocl-icd-opencl-dev \
" "
ENV packages_dev="\ ENV packages_dev="\

View file

@ -0,0 +1,14 @@
ARG acct
ARG repo
ARG dist_name
ARG dist_version
ARG feature
ARG cc
ARG machine
FROM ${acct}/${repo}:${dist_name}-${dist_version}-${feature}-built-${cc}-${machine}
WORKDIR /root
RUN true \
&& construct -debug -smoketest localhost \
&& true

View file

@ -74,7 +74,7 @@ namespace boost
// In boost 1.79+ asio implements some filesystem operations we can use. While // In boost 1.79+ asio implements some filesystem operations we can use. While
// these are available in 1.78 they were buggy for our purposes until 1.79. // these are available in 1.78 they were buggy for our purposes until 1.79.
#if BOOST_VERSION >= 107900 #if IRCD_USE_ASIO_IO_URING && BOOST_VERSION >= 107900
#define IRCD_USE_ASIO_READ 1 #define IRCD_USE_ASIO_READ 1
#define IRCD_USE_ASIO_WRITE 1 #define IRCD_USE_ASIO_WRITE 1
#else #else

View file

@ -44,8 +44,8 @@ namespace ircd::b64
size_t encode_unpadded_size(const const_buffer &in) noexcept; size_t encode_unpadded_size(const const_buffer &in) noexcept;
const_buffer decode(const mutable_buffer out, const string_view in); const_buffer decode(const mutable_buffer out, const string_view in);
string_view encode_unpadded(const mutable_buffer out, const const_buffer in, const dictionary & = dict_rfc1421) noexcept; string_view encode_unpadded(const mutable_buffer out, const const_buffer in, const dictionary = dict_rfc1421) noexcept;
string_view encode(const mutable_buffer out, const const_buffer in, const dictionary & = dict_rfc1421) noexcept; string_view encode(const mutable_buffer out, const const_buffer in, const dictionary = dict_rfc1421) noexcept;
} }
inline size_t inline size_t

View file

@ -32,6 +32,7 @@ struct ircd::buffer::unique_buffer
unique_buffer() = default; unique_buffer() = default;
unique_buffer(const size_t &size, const size_t &align = 0); unique_buffer(const size_t &size, const size_t &align = 0);
explicit unique_buffer(const const_buffer &); explicit unique_buffer(const const_buffer &);
template<class T> unique_buffer(unique_buffer<T> &&) noexcept;
unique_buffer(unique_buffer &&) noexcept; unique_buffer(unique_buffer &&) noexcept;
unique_buffer(const unique_buffer &) = delete; unique_buffer(const unique_buffer &) = delete;
unique_buffer &operator=(unique_buffer &&) & noexcept; unique_buffer &operator=(unique_buffer &&) & noexcept;
@ -71,6 +72,19 @@ ircd::buffer::unique_buffer<buffer_type>::unique_buffer(const size_t &size,
} }
{} {}
template<class buffer_type>
template<class other_type>
inline
ircd::buffer::unique_buffer<buffer_type>::unique_buffer(unique_buffer<other_type> &&other)
noexcept
:buffer_type
{
other.release()
}
{
assert(std::get<0>(other) == nullptr);
}
template<class buffer_type> template<class buffer_type>
inline inline
ircd::buffer::unique_buffer<buffer_type>::unique_buffer(unique_buffer &&other) ircd::buffer::unique_buffer<buffer_type>::unique_buffer(unique_buffer &&other)

View file

@ -56,7 +56,9 @@ struct ircd::client
size_t write_all(const net::const_buffers &); size_t write_all(const net::const_buffers &);
size_t write_all(const const_buffer &); size_t write_all(const const_buffer &);
void close(const net::close_opts &, net::close_callback); void close(const net::close_opts &, net::close_callback);
ctx::future<void> close(const net::close_opts & = {}); void close(const net::dc, net::close_callback);
ctx::future<void> close(const net::close_opts &);
ctx::future<void> close(const net::dc);
private: private:
void discard_unconsumed(const http::request::head &); void discard_unconsumed(const http::request::head &);

View file

@ -46,6 +46,8 @@ struct ircd::ctx::future
bool valid() const { return !is(state(), future_state::INVALID); } bool valid() const { return !is(state(), future_state::INVALID); }
bool operator!() const { return !valid(); } bool operator!() const { return !valid(); }
std::exception_ptr eptr() const { return state().eptr; }
string_view what() const { return util::what(eptr()); }
explicit operator T() { return get(); } explicit operator T() { return get(); }
template<class U, class time_point> friend bool wait_until(const future<U> &, const time_point &, std::nothrow_t); template<class U, class time_point> friend bool wait_until(const future<U> &, const time_point &, std::nothrow_t);
@ -75,6 +77,8 @@ struct ircd::ctx::future<void>
bool valid() const { return !is(state(), future_state::INVALID); } bool valid() const { return !is(state(), future_state::INVALID); }
bool operator!() const { return !valid(); } bool operator!() const { return !valid(); }
std::exception_ptr eptr() const { return state().eptr; }
string_view what() const { return util::what(eptr()); }
template<class U, class time_point> friend bool wait_until(const future<U> &, const time_point &, std::nothrow_t); template<class U, class time_point> friend bool wait_until(const future<U> &, const time_point &, std::nothrow_t);
template<class U, class time_point> friend void wait_until(const future<U> &, const time_point &); template<class U, class time_point> friend void wait_until(const future<U> &, const time_point &);

View file

@ -38,9 +38,9 @@ namespace ircd::db
template<> prop_map property(const column &, const string_view &name); template<> prop_map property(const column &, const string_view &name);
// Access to the column's caches (see cache.h interface) // Access to the column's caches (see cache.h interface)
const rocksdb::Cache *cache_compressed(const column &); [[gnu::deprecated]] const rocksdb::Cache *cache_compressed(const column &);
const rocksdb::Cache *cache(const column &); const rocksdb::Cache *cache(const column &);
rocksdb::Cache *cache_compressed(column &); [[gnu::deprecated]] rocksdb::Cache *cache_compressed(column &);
rocksdb::Cache *cache(column &); rocksdb::Cache *cache(column &);
// [GET] Tests if key exists // [GET] Tests if key exists

View file

@ -18,9 +18,16 @@ namespace ircd::db
// Broad conf items // Broad conf items
extern conf::item<std::string> open_recover; extern conf::item<std::string> open_recover;
extern conf::item<bool> open_repair; extern conf::item<bool> open_repair;
extern conf::item<bool> open_slave;
extern conf::item<bool> auto_compact; extern conf::item<bool> auto_compact;
extern conf::item<bool> auto_deletion; extern conf::item<bool> auto_deletion;
extern conf::item<bool> open_stats;
extern conf::item<bool> paranoid;
extern conf::item<bool> paranoid_checks;
extern conf::item<bool> paranoid_size;
extern conf::item<bool> paranoid_uuid;
extern conf::item<bool> paranoid_wal;
extern conf::item<bool> paranoid_sst;
extern conf::item<bool> paranoid_lsm;
// General information // General information
const std::string &name(const database &); const std::string &name(const database &);

View file

@ -102,3 +102,11 @@ const
{ {
return &this->operator*(); return &this->operator*();
} }
inline bool
ircd::db::seek(domain::const_iterator_base &it,
const string_view &p)
{
it.opts.prefix = true;
return seek(static_cast<column::const_iterator_base &>(it), p);
}

View file

@ -14,9 +14,9 @@
namespace ircd::fs::dev namespace ircd::fs::dev
{ {
struct blk; struct blk;
struct stats;
using major_minor = std::pair<ulong, ulong>; using major_minor = std::pair<ulong, ulong>;
using blk_closure = std::function<bool (const ulong &id, const blk &)>;
// Convert device ID's with the major(3) / minor(3) / makedev(3) // Convert device ID's with the major(3) / minor(3) / makedev(3)
ulong id(const major_minor &); ulong id(const major_minor &);
@ -36,13 +36,12 @@ namespace ircd::fs::dev
sysfs(const ulong &id, sysfs(const ulong &id,
const string_view &path, const string_view &path,
const R &def = 0); const R &def = 0);
bool for_each(const string_view &devtype, const blk_closure &);
bool for_each(const blk_closure &);
} }
struct ircd::fs::dev::blk struct ircd::fs::dev::blk
{ {
using closure = util::function_bool<const ulong &, const blk &>;
static const size_t SECTOR_SIZE; static const size_t SECTOR_SIZE;
static const string_view BASE_PATH; static const string_view BASE_PATH;
@ -66,6 +65,47 @@ struct ircd::fs::dev::blk
blk(const ulong &id); blk(const ulong &id);
blk() = default; blk() = default;
static bool for_each(const string_view &devtype, const closure &);
static bool for_each(const closure &);
};
struct ircd::fs::dev::stats
{
using closure = util::function_bool<const stats &>;
char name[32] {0};
major_minor id {0, 0};
uint64_t read {0};
uint64_t read_merged {0};
uint64_t read_sectors {0};
milliseconds read_time {0ms};
uint64_t write {0};
uint64_t write_merged {0};
uint64_t write_sectors {0};
milliseconds write_time {0ms};
uint64_t io_current {0};
milliseconds io_time {0ms};
milliseconds io_weighted_time {0ms};
// 4.18+
uint64_t discard {0};
uint64_t discard_merged {0};
uint64_t discard_sectors {0};
milliseconds discard_time {0ms};
// 5.5+
uint64_t flush {0};
milliseconds flush_time {0ms};
stats(const string_view &line);
stats() = default;
static bool for_each(const closure &);
static stats get(const major_minor &id);
}; };
/// Return a lex_cast'able (an integer) from a sysfs target. /// Return a lex_cast'able (an integer) from a sysfs target.

View file

@ -124,7 +124,7 @@ namespace ircd::info::hardware::x86
extern const string_view vendor; extern const string_view vendor;
extern const bool sse, sse2, sse3, ssse3, sse4a, sse4_1, sse4_2; extern const bool sse, sse2, sse3, ssse3, sse4a, sse4_1, sse4_2;
extern const bool avx, avx2, avx512f; extern const bool avx, avx2, avx512f;
extern const bool tsc, tsc_constant; extern const bool tsc, tsc_constant, tsc_nonstop;
} }
namespace ircd::info::hardware::arm namespace ircd::info::hardware::arm

View file

@ -29,14 +29,14 @@ struct ircd::ios::descriptor
static uint64_t ids; static uint64_t ids;
static void *default_allocator(handler &, const size_t &); static void *default_allocator(handler &, const size_t);
static void default_deallocator(handler &, void *const &, const size_t &) noexcept; static void default_deallocator(handler &, void *, const size_t) noexcept;
string_view name; string_view name;
uint64_t id {++ids}; uint64_t id {++ids};
std::unique_ptr<struct stats> stats; std::unique_ptr<struct stats> stats;
void *(*allocator)(handler &, const size_t &); void *(*allocator)(handler &, const size_t);
void (*deallocator)(handler &, void *const &, const size_t &); void (*deallocator)(handler &, void *, const size_t);
std::vector<std::array<uint64_t, 2>> history; // epoch, cycles std::vector<std::array<uint64_t, 2>> history; // epoch, cycles
uint8_t history_pos {0}; uint8_t history_pos {0};
bool continuation {false}; bool continuation {false};
@ -72,10 +72,10 @@ struct ircd::ios::descriptor::stats
item alloc_bytes; item alloc_bytes;
item frees; item frees;
item free_bytes; item free_bytes;
item slice_total;
item slice_last; item slice_last;
item latency_total; item slice_total;
item latency_last; item latency_last;
item latency_total;
stats(descriptor &); stats(descriptor &);
stats() = delete; stats() = delete;
@ -87,8 +87,8 @@ struct ircd::ios::descriptor::stats
[[gnu::hot]] [[gnu::hot]]
inline void inline void
ircd::ios::descriptor::default_deallocator(handler &handler, ircd::ios::descriptor::default_deallocator(handler &handler,
void *const &ptr, void *const ptr,
const size_t &size) const size_t size)
noexcept noexcept
{ {
#ifdef __clang__ #ifdef __clang__
@ -101,7 +101,7 @@ noexcept
[[gnu::hot]] [[gnu::hot]]
inline void * inline void *
ircd::ios::descriptor::default_allocator(handler &handler, ircd::ios::descriptor::default_allocator(handler &handler,
const size_t &size) const size_t size)
{ {
return ::operator new(size); return ::operator new(size);
} }

View file

@ -28,13 +28,13 @@ struct ircd::ios::handler
static thread_local handler *current; static thread_local handler *current;
static thread_local uint64_t epoch; static thread_local uint64_t epoch;
static void enqueue(handler *const &) noexcept; static void enqueue(handler *) noexcept;
static void *allocate(handler *const &, const size_t &); static void *allocate(handler *, const size_t);
static void deallocate(handler *const &, void *const &, const size_t &) noexcept; static void deallocate(handler *, void *, const size_t) noexcept;
static bool continuation(handler *const &) noexcept; static bool continuation(handler *) noexcept;
static void enter(handler *const &) noexcept; static void enter(handler *) noexcept;
static void leave(handler *const &) noexcept; static void leave(handler *) noexcept;
static bool fault(handler *const &) noexcept; static bool fault(handler *) noexcept;
ios::descriptor *descriptor {nullptr}; ios::descriptor *descriptor {nullptr};
uint64_t ts {0}; // last tsc sample; for profiling each phase uint64_t ts {0}; // last tsc sample; for profiling each phase
@ -91,9 +91,9 @@ const
[[gnu::hot]] [[gnu::hot]]
inline void inline void
ircd::ios::handler::deallocate(handler *const &handler, ircd::ios::handler::deallocate(handler *const handler,
void *const &ptr, void *const ptr,
const size_t &size) const size_t size)
noexcept noexcept
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
@ -110,8 +110,8 @@ noexcept
[[gnu::hot]] [[gnu::hot]]
inline void * inline void *
ircd::ios::handler::allocate(handler *const &handler, ircd::ios::handler::allocate(handler *const handler,
const size_t &size) const size_t size)
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
auto &descriptor(*handler->descriptor); auto &descriptor(*handler->descriptor);
@ -127,7 +127,7 @@ ircd::ios::handler::allocate(handler *const &handler,
[[gnu::hot]] [[gnu::hot]]
inline void inline void
ircd::ios::handler::enqueue(handler *const &handler) ircd::ios::handler::enqueue(handler *const handler)
noexcept noexcept
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
@ -152,7 +152,7 @@ noexcept
[[gnu::hot]] [[gnu::hot]]
inline bool inline bool
ircd::ios::handler::continuation(handler *const &handler) ircd::ios::handler::continuation(handler *const handler)
noexcept noexcept
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);

View file

@ -30,6 +30,7 @@ namespace ircd::ios
extern asio::executor user, main; extern asio::executor user, main;
extern std::thread::id main_thread_id; extern std::thread::id main_thread_id;
extern thread_local bool is_main_thread; extern thread_local bool is_main_thread;
extern bool user_available, main_available;
bool available() noexcept; bool available() noexcept;
const uint64_t &epoch() noexcept; const uint64_t &epoch() noexcept;
@ -64,3 +65,11 @@ noexcept
{ {
return handler::epoch; return handler::epoch;
} }
inline bool
__attribute__((always_inline))
ircd::ios::available()
noexcept
{
return main_available;
}

View file

@ -107,6 +107,7 @@
#include "mods/mods.h" #include "mods/mods.h"
#include "net/net.h" #include "net/net.h"
#include "server/server.h" #include "server/server.h"
#include "rest.h"
#include "png.h" #include "png.h"
#include "beep.h" #include "beep.h"
#include "magick.h" #include "magick.h"
@ -129,10 +130,10 @@ namespace ircd
// Operating Mode Selectors // Operating Mode Selectors
extern conf::item<bool> debugmode; extern conf::item<bool> debugmode;
extern conf::item<bool> maintenance;
extern conf::item<bool> soft_assert; extern conf::item<bool> soft_assert;
extern conf::item<bool> write_avoid; // implies maintenance extern conf::item<bool> maintenance;
extern conf::item<bool> read_only; // implies write_avoid extern conf::item<bool> slave;
extern conf::item<bool> read_only;
extern conf::item<bool> defaults; extern conf::item<bool> defaults;
} }

View file

@ -39,20 +39,7 @@ namespace ircd::json
#include "strung.h" #include "strung.h"
#include "tuple/tuple.h" #include "tuple/tuple.h"
#include "stack/stack.h" #include "stack/stack.h"
#include "tool.h"
// Convenience toolset for higher level operations.
namespace ircd::json
{
strung append(const array &, const string_view &val);
strung prepend(const array &, const string_view &val);
void merge(stack::object &out, const vector &);
strung remove(const object &, const string_view &key);
strung remove(const object &, const size_t &index);
strung insert(const object &, const member &);
strung replace(const object &, const member &);
strung replace(const object &, const members &);
}
// Exports to ircd:: // Exports to ircd::
namespace ircd namespace ircd

View file

@ -41,6 +41,10 @@ struct ircd::json::stack::member
member(stack &s, const string_view &name, const json::value &); member(stack &s, const string_view &name, const json::value &);
template<class... T> member(object &po, const string_view &name, const json::tuple<T...> &t); template<class... T> member(object &po, const string_view &name, const json::tuple<T...> &t);
template<class... T> member(stack &s, const string_view &name, const json::tuple<T...> &t); template<class... T> member(stack &s, const string_view &name, const json::tuple<T...> &t);
explicit member(object &, const json::object::member &);
explicit member(stack &, const json::object::member &);
explicit member(object &, const json::member &);
explicit member(stack &, const json::member &);
member() = default; member() = default;
member(const member &) = delete; member(const member &) = delete;
member(member &&) noexcept; member(member &&) noexcept;

View file

@ -105,28 +105,28 @@ template<>
inline ircd::json::stack::array & inline ircd::json::stack::array &
ircd::json::stack::stack::top<ircd::json::stack::array>(stack &s) ircd::json::stack::stack::top<ircd::json::stack::array>(stack &s)
{ {
return array::top(s); return array::top(s);
} }
template<> template<>
inline const ircd::json::stack::array & inline const ircd::json::stack::array &
ircd::json::stack::stack::top<ircd::json::stack::array>(const stack &s) ircd::json::stack::stack::top<ircd::json::stack::array>(const stack &s)
{ {
return array::top(s); return array::top(s);
} }
template<> template<>
inline ircd::json::stack::object & inline ircd::json::stack::object &
ircd::json::stack::stack::top<ircd::json::stack::object>(stack &s) ircd::json::stack::stack::top<ircd::json::stack::object>(stack &s)
{ {
return object::top(s); return object::top(s);
} }
template<> template<>
inline const ircd::json::stack::object & inline const ircd::json::stack::object &
ircd::json::stack::stack::top<ircd::json::stack::object>(const stack &s) ircd::json::stack::stack::top<ircd::json::stack::object>(const stack &s)
{ {
return object::top(s); return object::top(s);
} }
template<> template<>

44
include/ircd/json/tool.h Normal file
View file

@ -0,0 +1,44 @@
// Matrix Construct
//
// Copyright (C) Matrix Construct Developers, Authors & Contributors
// Copyright (C) 2016-2023 Jason Volk <jason@zemos.net>
//
// Permission to use, copy, modify, and/or distribute this software for any
// purpose with or without fee is hereby granted, provided that the above
// copyright notice and this permission notice is present in all copies. The
// full license for this software is available in the LICENSE file.
#pragma once
#define HAVE_IRCD_JSON_TOOL_H
// Convenience toolset for higher level operations.
namespace ircd::json
{
strung append(const array &, const string_view &val);
strung prepend(const array &, const string_view &val);
void merge(stack::object &out, const vector &);
strung remove(const object &, const string_view &key);
strung remove(const object &, const size_t &index);
strung insert(const object &, const members &);
strung insert(const object &, const member &);
strung replace(const object &, const members &);
strung replace(const object &, const member &);
}
inline ircd::json::strung
ircd::json::replace(const object &s,
const json::member &m)
{
return replace(s, json::members{m});
}
inline ircd::json::strung
ircd::json::insert(const object &s,
const json::member &m)
{
return insert(s, json::members{m});
}

View file

@ -60,17 +60,18 @@ template<class T>
struct ircd::json::keys<T>::include struct ircd::json::keys<T>::include
:selection :selection
{ {
constexpr include(const vector_view<const string_view> list) include(const vector_view<const string_view> &keys)
:selection{0} :selection{0}
{ {
for(const auto key : list) for(const auto &key : keys)
selection::set(key, true); selection::set(key, true);
} }
constexpr include(const std::initializer_list<const string_view> list) template<class... list>
consteval include(list&&... keys)
:selection{0} :selection{0}
{ {
for(const auto key : list) for(auto&& key : {keys...})
selection::set(key, true); selection::set(key, true);
} }
@ -86,17 +87,19 @@ template<class T>
struct ircd::json::keys<T>::exclude struct ircd::json::keys<T>::exclude
:selection :selection
{ {
constexpr exclude(const vector_view<const string_view> list) exclude(const vector_view<const string_view> &keys)
:selection{} :selection{}
{ {
for(const auto key : list) for(const auto &key : keys)
selection::set(key, false); selection::set(key, false);
} }
constexpr exclude(const std::initializer_list<const string_view> list)
template<class... list>
consteval exclude(list&&... keys)
:selection{} :selection{}
{ {
for(const auto key : list) for(auto&& key : {keys...})
selection::set(key, false); selection::set(key, false);
} }

View file

@ -29,26 +29,18 @@ namespace ircd::json
/// not use the strict overload. /// not use the strict overload.
IRCD_OVERLOAD(strict) IRCD_OVERLOAD(strict)
// Determine the type
enum type type(const string_view &);
enum type type(const string_view &, std::nothrow_t);
enum type type(const string_view &, strict_t);
enum type type(const string_view &, strict_t, std::nothrow_t);
// Query if type
bool type(const string_view &, const enum type &, strict_t);
bool type(const string_view &, const enum type &);
// Utils // Utils
string_view reflect(const enum type &); [[gnu::pure]] string_view reflect(const enum type) noexcept;
extern const string_view literal_null; // Determine the type w/ strict correctness (full scan)
extern const string_view literal_true; [[gnu::pure]] bool type(const string_view &, const enum type, strict_t) noexcept;
extern const string_view literal_false; [[gnu::pure]] enum type type(const string_view &, strict_t, std::nothrow_t) noexcept;
extern const string_view empty_string; enum type type(const string_view &, strict_t);
extern const string_view empty_object;
extern const string_view empty_array; // Determine the type quickly
extern const int64_t undefined_number; [[gnu::pure]] bool type(const string_view &, const enum type) noexcept;
[[gnu::pure]] enum type type(const string_view &, std::nothrow_t) noexcept;
enum type type(const string_view &);
} }
enum ircd::json::type enum ircd::json::type

View file

@ -36,7 +36,15 @@ namespace ircd::json
void valid(const string_view &); void valid(const string_view &);
std::string why(const string_view &); std::string why(const string_view &);
struct stats extern stats; extern const string_view literal_null;
extern const string_view literal_true;
extern const string_view literal_false;
extern const string_view empty_string;
extern const string_view empty_object;
extern const string_view empty_array;
extern const int64_t undefined_number;
extern struct stats stats;
} }
/// Statistics counter access; unfortunately these cannot participate as /// Statistics counter access; unfortunately these cannot participate as

View file

@ -53,11 +53,14 @@ struct ircd::m::signing_key_update
/// Required. The user ID whose cross-signing keys have changed. /// Required. The user ID whose cross-signing keys have changed.
json::property<name::user_id, json::string>, json::property<name::user_id, json::string>,
/// Cross signing key /// Master signing key
json::property<name::master_key, json::object>, json::property<name::master_key, json::object>,
/// Cross signing key /// Self signing key
json::property<name::self_signing_key, json::object> json::property<name::self_signing_key, json::object>,
/// User signing key (local only)
json::property<name::user_signing_key, json::object>
> >
{ {
using super_type::tuple; using super_type::tuple;

View file

@ -29,6 +29,29 @@ struct ircd::m::event::append
{ {
struct opts; struct opts;
private:
static const event::keys::exclude exclude_keys;
static const event::keys default_keys;
static conf::item<std::string> exclude_types;
static conf::item<bool> info;
static log::log log;
bool is_ignored(const event &, const opts &) const;
bool is_redacted(const event &, const opts &) const;
bool is_invisible(const event &, const opts &) const;
bool is_excluded(const event &, const opts &) const;
bool bundle_replace(json::stack::object &, const event &, const opts &);
void _relations(json::stack::object &, const event &, const opts &);
void _age(json::stack::object &, const event &, const opts &);
void _txnid(json::stack::object &, const event &, const opts &);
void _prev_state(json::stack::object &, const event &, const opts &);
void _unsigned(json::stack::object &, const event &, const opts &);
bool members(json::stack::object &, const event &, const opts &);
bool object(json::stack::array &, const event &, const opts &);
public:
append(json::stack::object &, const event &, const opts &); append(json::stack::object &, const event &, const opts &);
append(json::stack::object &, const event &); append(json::stack::object &, const event &);
append(json::stack::array &, const event &, const opts &); append(json::stack::array &, const event &, const opts &);
@ -39,11 +62,11 @@ struct ircd::m::event::append
/// can provide the best result. /// can provide the best result.
struct ircd::m::event::append::opts struct ircd::m::event::append::opts
{ {
const event::idx *event_idx {nullptr}; event::idx event_idx {0};
const string_view *client_txnid {nullptr}; string_view client_txnid;
const id::user *user_id {nullptr}; id::user user_id;
const room *user_room {nullptr}; id::room user_room_id;
const int64_t *room_depth {nullptr}; int64_t room_depth {-1L};
const event::keys *keys {nullptr}; const event::keys *keys {nullptr};
const m::event_filter *event_filter {nullptr}; const m::event_filter *event_filter {nullptr};
long age {std::numeric_limits<long>::min()}; long age {std::numeric_limits<long>::min()};
@ -51,6 +74,8 @@ struct ircd::m::event::append::opts
bool query_prev_state {true}; bool query_prev_state {true};
bool query_redacted {true}; bool query_redacted {true};
bool query_visible {false}; bool query_visible {false};
bool bundle_all {false};
bool bundle_replace {false};
}; };
inline inline
@ -64,3 +89,23 @@ ircd::m::event::append::append(json::stack::object &o,
const event &e) const event &e)
:append{o, e, {}} :append{o, e, {}}
{} {}
inline
ircd::m::event::append::append(json::stack::array &array,
const event &event,
const opts &opts)
:returns<bool>{[this, &array, &event, &opts]
{
return object(array, event, opts);
}}
{}
inline
ircd::m::event::append::append(json::stack::object &object,
const event &event,
const opts &opts)
:returns<bool>{[this, &object, &event, &opts]
{
return members(object, event, opts);
}}
{}

View file

@ -41,11 +41,12 @@ namespace ircd::m
size_t degree(const event &); size_t degree(const event &);
bool before(const event &a, const event &b); bool before(const event &a, const event &b);
json::object hashes(const mutable_buffer &, const event &);
event signatures(const mutable_buffer &, const m::event &, const string_view &origin);
event signatures(const mutable_buffer &, const m::event &);
event essential(event, const mutable_buffer &content, const bool &sigs = false); event essential(event, const mutable_buffer &content, const bool &sigs = false);
event signatures(const mutable_buffer &, const event &, const string_view &origin, const ed25519::sk &, const string_view &pkid);
event signatures(const mutable_buffer &, const event &, const string_view &origin);
event signatures(const mutable_buffer &, const event &);
json::object hashes(const mutable_buffer &, const event &);
bool verify_hash(const event &, const sha256::buf &); bool verify_hash(const event &, const sha256::buf &);
bool verify_hash(const event &); bool verify_hash(const event &);

View file

@ -83,6 +83,8 @@ struct ircd::m::homeserver
/// Factory to create homeserver with single procedure for shlib purposes. /// Factory to create homeserver with single procedure for shlib purposes.
static homeserver *init(const struct opts *); static homeserver *init(const struct opts *);
static void fini(homeserver *) noexcept; static void fini(homeserver *) noexcept;
static bool rehash(homeserver *);
static bool refresh(homeserver *);
}; };
struct ircd::m::homeserver::key struct ircd::m::homeserver::key

View file

@ -63,7 +63,8 @@ struct ircd::m::keys
static bool get(const queries &, const closure_bool &); static bool get(const queries &, const closure_bool &);
static bool get(const string_view &server_name, const closure &); static bool get(const string_view &server_name, const closure &);
static bool get(const string_view &server_name, const string_view &key_id, const closure &); static bool get(const string_view &server_name, const string_view &key_id, const closure &);
static bool query(const string_view &query_server, const queries &, const closure_bool &); static bool query(const string_view &remote, const queries &, const closure_bool &, const mutable_buffer &, const bool dynamic = false);
static bool query(const string_view &remote, const queries &, const closure_bool &);
static size_t fetch(const queries &); static size_t fetch(const queries &);
static size_t fetch(const pdus &); static size_t fetch(const pdus &);

View file

@ -89,6 +89,7 @@ namespace ircd::m
#include "direct_to_device.h" #include "direct_to_device.h"
#include "visible.h" #include "visible.h"
#include "redacted.h" #include "redacted.h"
#include "replaced.h"
#include "feds.h" #include "feds.h"
#include "app.h" #include "app.h"
#include "bridge.h" #include "bridge.h"

View file

@ -33,7 +33,7 @@ namespace ircd::m::media::file
room::id::buf room_id(const mxc &); room::id::buf room_id(const mxc &);
size_t read(const room &, const closure &); size_t read(const room &, const closure &);
size_t write(const room &, const user::id &, const const_buffer &content, const string_view &content_type); size_t write(const room &, const user::id &, const const_buffer &content, const string_view &content_type, const string_view &name = {});
room::id::buf room::id::buf
download(const mxc &, download(const mxc &,

View file

@ -221,4 +221,5 @@ struct ircd::m::name
static constexpr const char *const usage {"usage"}; static constexpr const char *const usage {"usage"};
static constexpr const char *const master_key {"master_key"}; static constexpr const char *const master_key {"master_key"};
static constexpr const char *const self_signing_key {"self_signing_key"}; static constexpr const char *const self_signing_key {"self_signing_key"};
static constexpr const char *const user_signing_key {"user_signing_key"};
}; };

View file

@ -44,9 +44,11 @@ struct ircd::m::relates
const event::idx &, const json::object &, const m::relates_to & const event::idx &, const json::object &, const m::relates_to &
>; >;
static conf::item<std::string> latest_column;
event::refs refs; event::refs refs;
bool match_sender {false}; bool match_sender {false};
bool prefetch_depth {false}; bool prefetch_latest {false};
bool prefetch_sender {false}; bool prefetch_sender {false};
private: private:

80
include/ircd/m/replaced.h Normal file
View file

@ -0,0 +1,80 @@
// Matrix Construct
//
// Copyright (C) Matrix Construct Developers, Authors & Contributors
// Copyright (C) 2016-2023 Jason Volk <jason@zemos.net>
//
// Permission to use, copy, modify, and/or distribute this software for any
// purpose with or without fee is hereby granted, provided that the above
// copyright notice and this permission notice is present in all copies. The
// full license for this software is available in the LICENSE file.
#pragma once
#define HAVE_IRCD_M_REPLACED_H
namespace ircd::m
{
struct replaced;
}
class ircd::m::replaced
:public returns<event::idx>
{
m::relates relates;
public:
IRCD_OVERLOAD(latest);
replaced(const event::idx &, latest_t);
replaced(const event::idx &);
replaced(const event::id &, latest_t);
replaced(const event::id &);
explicit replaced(const event &, latest_t);
explicit replaced(const event &);
};
inline
ircd::m::replaced::replaced(const event &event)
:replaced{event.event_id}
{}
inline
ircd::m::replaced::replaced(const event &event,
latest_t)
:replaced{event.event_id, latest}
{}
inline
ircd::m::replaced::replaced(const event::id &event_id)
:replaced{index(std::nothrow, event_id)}
{}
inline
ircd::m::replaced::replaced(const event::id &event_id,
latest_t)
:replaced{index(std::nothrow, event_id), latest}
{}
inline
ircd::m::replaced::replaced(const event::idx &event_idx)
:relates
{
.refs = event_idx,
.match_sender = true,
}
{
this->returns::ret = relates.has("m.replace")? -1UL: 0UL;
}
inline
ircd::m::replaced::replaced(const event::idx &event_idx,
latest_t)
:relates
{
.refs = event_idx,
.match_sender = true,
}
{
this->returns::ret = relates.latest("m.replace");
}

View file

@ -31,6 +31,7 @@ namespace ircd::m
bool exists(const id::room_alias &, const bool &remote = false); bool exists(const id::room_alias &, const bool &remote = false);
bool internal(const id::room &); bool internal(const id::room &);
bool federated(const id::room &); bool federated(const id::room &);
bool type(const id::room &, const string_view &);
bool creator(const id::room &, const id::user &); bool creator(const id::room &, const id::user &);
bool contains(const id::room &, const event::idx &); bool contains(const id::room &, const event::idx &);
bool membership(const room &, const id::user &, const string_view & = "join"); bool membership(const room &, const id::user &, const string_view & = "join");
@ -186,9 +187,11 @@ struct ircd::m::room
const vm::copts *const &copts, const vm::copts *const &copts,
const event::fetch::opts *const &fopts = nullptr) noexcept; const event::fetch::opts *const &fopts = nullptr) noexcept;
room(const id &room_id = {}, room(const id &room_id,
const event::fetch::opts *const &fopts = nullptr) noexcept; const event::fetch::opts *const &fopts = nullptr) noexcept;
room() = default;
// Index of create event // Index of create event
static event::idx index(const id &, std::nothrow_t); static event::idx index(const id &, std::nothrow_t);
static event::idx index(const id &); static event::idx index(const id &);
@ -228,7 +231,9 @@ ircd::m::room::room(const id &room_id,
,event_id{event_id? event::id{event_id} : event::id{}} ,event_id{event_id? event::id{event_id} : event::id{}}
,copts{copts} ,copts{copts}
,fopts{fopts} ,fopts{fopts}
{} {
assert(room_id);
}
inline inline
ircd::m::room::room(const id &room_id, ircd::m::room::room(const id &room_id,
@ -238,7 +243,9 @@ noexcept
:room_id{room_id} :room_id{room_id}
,copts{copts} ,copts{copts}
,fopts{fopts} ,fopts{fopts}
{} {
assert(room_id);
}
inline inline
ircd::m::room::room(const id &room_id, ircd::m::room::room(const id &room_id,
@ -246,7 +253,9 @@ ircd::m::room::room(const id &room_id,
noexcept noexcept
:room_id{room_id} :room_id{room_id}
,fopts{fopts} ,fopts{fopts}
{} {
assert(room_id);
}
inline ircd::m::room::operator inline ircd::m::room::operator
const ircd::m::room::id &() const ircd::m::room::id &()

View file

@ -48,6 +48,9 @@ struct ircd::m::rooms::opts
/// Room alias prefix search /// Room alias prefix search
string_view room_alias; string_view room_alias;
/// Room type search
string_view room_type;
/// user::rooms convenience /// user::rooms convenience
id::user user_id; id::user user_id;

View file

@ -13,8 +13,10 @@
struct ircd::m::user::devices struct ircd::m::user::devices
{ {
struct send;
using closure = std::function<void (const event::idx &, const string_view &)>; using closure = std::function<void (const event::idx &, const string_view &)>;
using closure_bool = std::function<bool (const event::idx &, const string_view &)>; using closure_bool = util::function_bool<const event::idx &, const string_view &>;
m::user user; m::user user;
@ -36,9 +38,17 @@ struct ircd::m::user::devices
///TODO: XXX junk ///TODO: XXX junk
static std::map<std::string, long> count_one_time_keys(const m::user &, const string_view &); static std::map<std::string, long> count_one_time_keys(const m::user &, const string_view &);
static bool update(const device_list_update &); static bool update(const device_list_update &);
static bool send(json::iov &content);
devices(const m::user &user) devices(const m::user &user)
:user{user} :user{user}
{} {}
}; };
/// Broadcast m.device_list_update.
///
struct ircd::m::user::devices::send
{
send(const m::user::devices &,
const m::id::device &,
const string_view = {});
};

128
include/ircd/m/user/keys.h Normal file
View file

@ -0,0 +1,128 @@
// The Construct
//
// Copyright (C) The Construct Developers, Authors & Contributors
// Copyright (C) 2016-2023 Jason Volk <jason@zemos.net>
//
// Permission to use, copy, modify, and/or distribute this software for any
// purpose with or without fee is hereby granted, provided that the above
// copyright notice and this permission notice is present in all copies. The
// full license for this software is available in the LICENSE file.
#pragma once
#define HAVE_IRCD_M_USER_KEYS_H
struct ircd::m::user::keys
{
struct send;
static string_view make_sigs_state_key(const mutable_buffer &, const string_view &tgt, const string_view &src);
static std::tuple<string_view, string_view> unmake_sigs_state_key(const string_view &) noexcept;
m::user::room user_room;
void attach_sigs(json::stack::object &, const json::object &, const user::id &) const;
bool attach_sigs(json::stack::object &, const event::idx &, const user::id &) const;
void append_sigs(json::stack::object &, const json::object &, const user::id &) const;
void append_keys(json::stack::object &, const json::object &, const user::id &) const;
bool append_keys(json::stack::object &, const event::idx &, const user::id &) const;
public:
bool has_device(const string_view &) const;
bool has_cross(const string_view &type) const;
bool has_cross_master() const;
bool has_cross_self() const;
bool has_cross_user() const;
void device(json::stack::object &, const string_view &device_id) const;
void cross(json::stack::object &, const string_view &type) const;
void cross_master(json::stack::object &) const;
void cross_self(json::stack::object &) const;
void cross_user(json::stack::object &) const;
bool claim(json::stack::object &, const string_view &device_id, const string_view &algo) const;
void update(const m::signing_key_update &) const;
keys(const m::user &user)
:user_room{user}
{}
};
struct ircd::m::user::keys::send
{
send(const m::user::keys &,
const string_view = {});
};
inline void
ircd::m::user::keys::cross_user(json::stack::object &out)
const
{
return cross(out, "ircd.cross_signing.user");
}
inline void
ircd::m::user::keys::cross_self(json::stack::object &out)
const
{
return cross(out, "ircd.cross_signing.self");
}
inline void
ircd::m::user::keys::cross_master(json::stack::object &out)
const
{
return cross(out, "ircd.cross_signing.master");
}
inline void
ircd::m::user::keys::cross(json::stack::object &out,
const string_view &type)
const
{
const auto event_idx
{
user_room.get(std::nothrow, type, "")
};
append_keys(out, event_idx, user_room.user.user_id);
}
inline bool
ircd::m::user::keys::has_cross_user()
const
{
return has_cross("ircd.cross_signing.user");
}
inline bool
ircd::m::user::keys::has_cross_self()
const
{
return has_cross("ircd.cross_signing.self");
}
inline bool
ircd::m::user::keys::has_cross_master()
const
{
return has_cross("ircd.cross_signing.master");
}
inline bool
ircd::m::user::keys::has_cross(const string_view &type)
const
{
return user_room.has(type, "");
}
inline bool
ircd::m::user::keys::has_device(const string_view &device_id)
const
{
const m::user::devices devices
{
user_room.user
};
return devices.has(device_id, "keys");
}

View file

@ -52,6 +52,7 @@ struct ircd::m::user
struct tokens; struct tokens;
struct devices; struct devices;
struct reading; struct reading;
struct keys;
using id = m::id::user; using id = m::id::user;
using closure = std::function<void (const user &)>; using closure = std::function<void (const user &)>;
@ -73,13 +74,17 @@ struct ircd::m::user
event::id::buf deoper(); event::id::buf deoper();
event::id::buf oper(); event::id::buf oper();
user(const id &user_id) user(const id &user_id);
:user_id{user_id}
{}
user() = default; user() = default;
}; };
inline
ircd::m::user::user(const id &user_id)
:user_id{user_id}
{
assert(user_id);
}
inline ircd::m::user::operator inline ircd::m::user::operator
const ircd::m::user::id &() const ircd::m::user::id &()
const const
@ -105,3 +110,4 @@ const
#include "tokens.h" #include "tokens.h"
#include "devices.h" #include "devices.h"
#include "reading.h" #include "reading.h"
#include "keys.h"

View file

@ -13,9 +13,12 @@
namespace ircd::m::vm::notify namespace ircd::m::vm::notify
{ {
using value_type = std::pair<const event::id, ctx::ctx *>; struct future;
using value_type = std::pair<const event::id, ctx::promise<> *>;
using alloc_type = allocator::node<value_type>; using alloc_type = allocator::node<value_type>;
using map_type = std::multimap<event::id, ctx::ctx *, std::less<>, alloc_type::allocator>; using map_type = std::multimap<const event::id, ctx::promise<> *, std::less<>, alloc_type::allocator>;
using node_type = std::pair<map_type::node_type, value_type>;
extern map_type map; extern map_type map;
@ -23,6 +26,19 @@ namespace ircd::m::vm::notify
bool wait(const event::id &, const milliseconds); bool wait(const event::id &, const milliseconds);
} }
class ircd::m::vm::notify::future
:public ctx::future<>
{
node_type node;
ctx::promise<> promise;
unique_iterator<map_type> it;
public:
future(const event::id &);
future(future &&) = delete;
future(const future &) = delete;
};
/// Yields ctx until event was successfully evaluated. Returns false if /// Yields ctx until event was successfully evaluated. Returns false if
/// timeout occurred. /// timeout occurred.
inline bool inline bool

View file

@ -16,6 +16,7 @@ namespace ircd::m::vm
enum phase :uint; enum phase :uint;
string_view reflect(const phase &); string_view reflect(const phase &);
phase phase_reflect(const string_view &) noexcept; // default NONE
} }
/// Evaluation phases /// Evaluation phases

View file

@ -13,6 +13,8 @@
namespace ircd::m::vm::sequence namespace ircd::m::vm::sequence
{ {
struct refresh;
extern ctx::dock dock; extern ctx::dock dock;
extern uint64_t retired; // already written; always monotonic extern uint64_t retired; // already written; always monotonic
extern uint64_t committed; // pending write; usually monotonic extern uint64_t committed; // pending write; usually monotonic
@ -25,3 +27,12 @@ namespace ircd::m::vm::sequence
uint64_t max(); uint64_t max();
uint64_t min(); uint64_t min();
} }
struct ircd::m::vm::sequence::refresh
{
uint64_t database[2] {0, 0};
uint64_t retired[2] {0, 0};
m::event::id::buf event_id;
refresh();
};

View file

@ -61,17 +61,22 @@ ircd::net::acceptor
ip::tcp::acceptor a; ip::tcp::acceptor a;
size_t accepting {0}; size_t accepting {0};
sockets handshaking; sockets handshaking;
bool secure {false};
bool interrupting {false}; bool interrupting {false};
ctx::dock joining; ctx::dock joining;
// Internal configuration // Internal configuration
void configure_dh(const json::object &); void configure_dh(const json::object &);
void configure_certs(const json::object &); bool configure_certs(const json::object &);
void configure_curves(const json::object &); void configure_curves(const json::object &);
void configure_ciphers(const json::object &); void configure_ciphers(const json::object &);
void configure_flags(const json::object &); void configure_flags(const json::object &);
void configure_password(const json::object &); void configure_password(const json::object &);
void configure(const json::object &opts); void configure_sni(const json::object &);
bool configure(const json::object &opts);
// Completion stack
void accepted(const std::shared_ptr<socket> &);
// Handshake stack // Handshake stack
bool handle_sni(socket &, int &ad); bool handle_sni(socket &, int &ad);

View file

@ -17,11 +17,15 @@ namespace ircd::net
struct close_opts extern const close_opts_default; struct close_opts extern const close_opts_default;
using close_callback = std::function<void (std::exception_ptr)>; using close_callback = std::function<void (std::exception_ptr)>;
string_view reflect(const dc) noexcept;
// Callback-based closer. // Callback-based closer.
void close(socket &, const close_opts &, close_callback); void close(socket &, const close_opts &, close_callback);
void close(socket &, const dc &, close_callback);
// Future-based closer. // Future-based closer.
ctx::future<void> close(socket &, const close_opts & = close_opts_default); ctx::future<void> close(socket &, const close_opts & = close_opts_default);
ctx::future<void> close(socket &, const dc &);
// Fire-and-forget helper callback for close(). // Fire-and-forget helper callback for close().
extern const close_callback close_ignore; extern const close_callback close_ignore;
@ -44,9 +48,6 @@ struct ircd::net::close_opts
{ {
static conf::item<milliseconds> default_timeout; static conf::item<milliseconds> default_timeout;
close_opts() = default;
close_opts(const net::dc &);
/// The type of close() to be conducted is specified here. /// The type of close() to be conducted is specified here.
net::dc type { dc::SSL_NOTIFY }; net::dc type { dc::SSL_NOTIFY };
@ -56,11 +57,10 @@ struct ircd::net::close_opts
/// If specified, these socket options will be applied when conducting /// If specified, these socket options will be applied when conducting
/// the disconnect (useful for adding an SO_LINGER time etc). /// the disconnect (useful for adding an SO_LINGER time etc).
const sock_opts *sopts { nullptr }; const sock_opts *sopts { nullptr };
};
/// Allows for implicit construction of close_opts in arguments to close() /// For portable clean disconnection shutdown(2) might be called prior to
/// without requiring brackets for the close_opts& argument. /// close(2). Setting this option to dc::RST skips the shutdown(2) when
inline /// the caller deems it unnecessary. At this time it only affects non-SSL
ircd::net::close_opts::close_opts(const net::dc &type) /// sockets and in the future we will have io_uring(7) fuse these calls.
:type{type} net::dc shutdown { dc::FIN };
{} };

View file

@ -37,6 +37,7 @@ ircd::net::dns::resolver
static conf::item<milliseconds> send_rate; static conf::item<milliseconds> send_rate;
static conf::item<size_t> send_burst; static conf::item<size_t> send_burst;
static conf::item<size_t> retry_max; static conf::item<size_t> retry_max;
static conf::item<bool> retry_serv_fail;
answers_callback callback; answers_callback callback;
std::vector<ip::udp::endpoint> server; // The list of active servers std::vector<ip::udp::endpoint> server; // The list of active servers
@ -131,11 +132,11 @@ ircd::net::dns::tag::tag(const hostport &hp,
{ {
this->hp.host = this->hp.host =
{ {
hostbuf, copy(hostbuf, hp.host) tolower(hostbuf, hp.host)
}; };
this->hp.service = this->hp.service =
{ {
servicebuf, copy(servicebuf, hp.service) tolower(servicebuf, hp.service)
}; };
} }

View file

@ -65,6 +65,9 @@ struct ircd::net::open_opts
/// if given. Defaults to null; no application is made. /// if given. Defaults to null; no application is made.
const sock_opts *sopts { nullptr }; const sock_opts *sopts { nullptr };
/// Option to disable SSL. Use false for plaintext socket.
bool secure { true };
/// Option to toggle whether to perform the SSL handshake; you want true. /// Option to toggle whether to perform the SSL handshake; you want true.
bool handshake { true }; bool handshake { true };

View file

@ -16,8 +16,9 @@ namespace ircd::net
using mutable_buffers = vector_view<const mutable_buffer>; using mutable_buffers = vector_view<const mutable_buffer>;
// Observers // Observers
size_t readable(const socket &); size_t readable(const socket &); // don't use w/ ssl
size_t available(const socket &) noexcept; size_t available(const socket &) noexcept; // don't use w/ ssl
size_t pending(const socket &) noexcept; // use with either.
// Non-blocking; read into buffers in a single syscall // Non-blocking; read into buffers in a single syscall
size_t read_one(socket &, const mutable_buffers &); size_t read_one(socket &, const mutable_buffers &);

View file

@ -14,10 +14,13 @@
namespace ircd::net namespace ircd::net
{ {
struct sock_opts; struct sock_opts;
IRCD_OVERLOAD(system);
bool v6only(const socket &); bool v6only(const socket &);
bool blocking(const socket &, system_t);
bool blocking(const socket &); bool blocking(const socket &);
bool nopush(const socket &); bool nopush(const socket &);
bool nodelay(const socket &, system_t);
bool nodelay(const socket &); bool nodelay(const socket &);
bool quickack(const socket &); bool quickack(const socket &);
bool keepalive(const socket &); bool keepalive(const socket &);
@ -29,21 +32,23 @@ namespace ircd::net
int attach(const socket &); int attach(const socket &);
// returns true if supported, false if unsupported; failures will throw. // returns true if supported, false if unsupported; failures will throw.
bool v6only(socket &, const bool &); bool v6only(socket &, const bool);
bool blocking(socket &, const bool &); bool blocking(socket &, const bool, system_t);
bool nopush(socket &, const bool &); bool blocking(socket &, const bool);
bool nodelay(socket &, const bool &); bool nopush(socket &, const bool);
bool quickack(socket &, const bool &); bool nodelay(socket &, const bool, system_t);
bool keepalive(socket &, const bool &); bool nodelay(socket &, const bool);
bool linger(socket &, const time_t &); // -1 is OFF; >= 0 is ON bool quickack(socket &, const bool);
bool read_bufsz(socket &, const size_t &bytes); bool keepalive(socket &, const bool);
bool write_bufsz(socket &, const size_t &bytes); bool linger(socket &, const time_t); // -1 is OFF; >= 0 is ON
bool read_lowat(socket &, const size_t &bytes); bool read_bufsz(socket &, const size_t bytes);
bool write_lowat(socket &, const size_t &bytes); bool write_bufsz(socket &, const size_t bytes);
bool attach(const int &sd, const int &fd); bool read_lowat(socket &, const size_t bytes);
bool attach(socket &, const int &fd); bool write_lowat(socket &, const size_t bytes);
bool detach(const int &sd, const int &fd); bool attach(const int sd, const int fd);
bool detach(socket &, const int &fd); bool attach(socket &, const int fd);
bool detach(const int sd, const int fd);
bool detach(socket &, const int fd);
void set(socket &, const sock_opts &); void set(socket &, const sock_opts &);
} }

View file

@ -28,7 +28,7 @@ namespace ircd::net
} }
/// Internal socket interface /// Internal socket interface
/// /// Socket cannot be copied or moved; must be constructed as shared ptr.
struct [[gnu::visibility("protected")]] struct [[gnu::visibility("protected")]]
ircd::net::socket ircd::net::socket
:std::enable_shared_from_this<ircd::net::socket> :std::enable_shared_from_this<ircd::net::socket>
@ -40,6 +40,7 @@ ircd::net::socket
using endpoint = ip::tcp::endpoint; using endpoint = ip::tcp::endpoint;
using wait_type = ip::tcp::socket::wait_type; using wait_type = ip::tcp::socket::wait_type;
using message_flags = asio::socket_base::message_flags; using message_flags = asio::socket_base::message_flags;
using ssl_stream = asio::ssl::stream<ip::tcp::socket &>;
using handshake_type = asio::ssl::stream<ip::tcp::socket>::handshake_type; using handshake_type = asio::ssl::stream<ip::tcp::socket>::handshake_type;
using ec_handler = std::function<void (const error_code &)>; using ec_handler = std::function<void (const error_code &)>;
using eptr_handler = std::function<void (std::exception_ptr)>; using eptr_handler = std::function<void (std::exception_ptr)>;
@ -64,7 +65,7 @@ ircd::net::socket
uint64_t id {++count}; uint64_t id {++count};
ip::tcp::socket sd; ip::tcp::socket sd;
asio::ssl::stream<ip::tcp::socket &> ssl; std::optional<ssl_stream> ssl;
endpoint local, remote; endpoint local, remote;
stat in, out; stat in, out;
deadline_timer timer; deadline_timer timer;
@ -73,6 +74,7 @@ ircd::net::socket
bool timer_set {false}; // boolean lockout bool timer_set {false}; // boolean lockout
bool timedout {false}; bool timedout {false};
bool fini {false}; bool fini {false};
mutable bool _nodelay {false}; // userspace tracking only
void call_user(const eptr_handler &, const error_code &) noexcept; void call_user(const eptr_handler &, const error_code &) noexcept;
void call_user(const ec_handler &, const error_code &) noexcept; void call_user(const ec_handler &, const error_code &) noexcept;
@ -120,9 +122,8 @@ ircd::net::socket
void connect(const endpoint &, const open_opts &, eptr_handler); void connect(const endpoint &, const open_opts &, eptr_handler);
bool cancel() noexcept; bool cancel() noexcept;
socket(asio::ssl::context & = sslv23_client); socket(asio::ssl::context &);
socket();
// Socket cannot be copied or moved; must be constructed as shared ptr
socket(socket &&) = delete; socket(socket &&) = delete;
socket(const socket &) = delete; socket(const socket &) = delete;
socket &operator=(socket &&) = delete; socket &operator=(socket &&) = delete;

View file

@ -39,7 +39,7 @@ struct ircd::resource::response
static const size_t HEAD_BUF_SZ; static const size_t HEAD_BUF_SZ;
static conf::item<std::string> access_control_allow_origin; static conf::item<std::string> access_control_allow_origin;
response(client &, const http::code &, const string_view &content_type, const size_t &content_length, const string_view &headers = {}); response(client &, const http::code &, const string_view &content_type, const size_t &content_length, const string_view &headers = {}, const string_view &content = {});
response(client &, const string_view &str, const string_view &content_type, const http::code &, const vector_view<const http::header> &); response(client &, const string_view &str, const string_view &content_type, const http::code &, const vector_view<const http::header> &);
response(client &, const string_view &str, const string_view &content_type, const http::code & = http::OK, const string_view &headers = {}); response(client &, const string_view &str, const string_view &content_type, const http::code & = http::OK, const string_view &headers = {});
response(client &, const json::object &str, const http::code & = http::OK); response(client &, const json::object &str, const http::code & = http::OK);

211
include/ircd/rest.h Normal file
View file

@ -0,0 +1,211 @@
// The Construct
//
// Copyright (C) The Construct Developers, Authors & Contributors
// Copyright (C) 2016-2023 Jason Volk <jason@zemos.net>
//
// Permission to use, copy, modify, and/or distribute this software for any
// purpose with or without fee is hereby granted, provided that the above
// copyright notice and this permission notice is present in all copies. The
// full license for this software is available in the LICENSE file.
#pragma once
#define HAVE_IRCD_REST_H
/// Simple highlevel interface for web/http requests.
///
/// Prior to this it was too difficult to orchestrate all the objects and
/// buffers and low-level non-ergonomic procedures split between ircd::http
/// and ircd::server. This should instead have some familiarity to the
/// browser-js environment which developers can easily commit to their memory.
namespace ircd::rest
{
struct opts;
struct request;
struct get;
struct put;
struct post;
}
struct ircd::rest::request
:returns<string_view>
{
unique_const_buffer out;
request(const mutable_buffer &out, const rfc3986::uri &, opts);
request(const rfc3986::uri &, opts);
};
struct ircd::rest::get
:request
{
get(const mutable_buffer &out, const rfc3986::uri &, opts);
get(const mutable_buffer &out, const rfc3986::uri &);
get(const rfc3986::uri &, opts);
};
struct ircd::rest::put
:request
{
put(const mutable_buffer &out, const rfc3986::uri &, const string_view &content, opts);
put(const rfc3986::uri &, const string_view &content, opts);
};
struct ircd::rest::post
:request
{
post(const mutable_buffer &out, const rfc3986::uri &, const string_view &content, opts);
post(const mutable_buffer &out, const rfc3986::uri &, opts);
post(const rfc3986::uri &, const string_view &content, opts);
post(const rfc3986::uri &, opts);
};
struct ircd::rest::opts
{
/// The HTTP method to use. This is overridden and should not be set unless
/// using the generic rest::request() call where it must be set.
string_view method;
/// The HTTP request body. This is overridden and should not be set unless
/// using the generic rest::request() call where it's set as needed.
string_view content;
/// The HTTP request body content-type. It is a good idea to set this when
/// there is request body content.
string_view content_type;
/// Additional request headers to send. These are pairs of string_views.
vector_view<const http::header> headers;
/// This is set automatically from the URI argument's domain and scheme
/// (service) by default. Setting it here will override.
net::hostport remote;
/// Managed internally by default and passed to server::request. Setting
/// things here will override.
server::out sout;
/// Managed internally by default and passed to server::request. Setting
/// things here will override.
server::in sin;
/// Passed to server::request. The http_exceptions option is useful here
/// to prevent this suite from throwing on non-2xx codes.
server::request::opts sopts;
/// Allows the HTTP response code to be returned to the caller. This may
/// not be initialized if the call throws any error first.
http::code *code {nullptr};
/// Allows the user to override the request::out with their own for
/// receiving dynamic content. Supply an empty unique_buffer instance.
unique_const_buffer *out {nullptr};
/// Optionally supply the temporary buffer for headers in/out in lieu of
/// any internally allocated.
mutable_buffer buf;
/// Timeout for the yielding/synchronous calls of this interface.
seconds timeout {20s};
/// Internal use
opts &&set(const string_view &method, const string_view &content = {});
};
inline
ircd::rest::post::post(const rfc3986::uri &uri,
opts opts)
:request
{
uri, opts.set("POST")
}
{}
inline
ircd::rest::post::post(const rfc3986::uri &uri,
const string_view &content,
opts opts)
:request
{
uri, opts.set("POST", content)
}
{}
inline
ircd::rest::post::post(const mutable_buffer &out,
const rfc3986::uri &uri,
opts opts)
:request
{
out, uri, opts.set("POST")
}
{}
inline
ircd::rest::post::post(const mutable_buffer &out,
const rfc3986::uri &uri,
const string_view &content,
opts opts)
:request
{
out, uri, opts.set("POST", content)
}
{}
inline
ircd::rest::put::put(const rfc3986::uri &uri,
const string_view &content,
opts opts)
:request
{
uri, opts.set("PUT", content)
}
{}
inline
ircd::rest::put::put(const mutable_buffer &out,
const rfc3986::uri &uri,
const string_view &content,
opts opts)
:request
{
out, uri, opts.set("PUT", content)
}
{}
inline
ircd::rest::get::get(const rfc3986::uri &uri,
opts opts)
:request
{
uri, opts.set("GET")
}
{}
inline
ircd::rest::get::get(const mutable_buffer &out,
const rfc3986::uri &uri)
:get
{
out, uri, opts{}
}
{}
inline
ircd::rest::get::get(const mutable_buffer &out,
const rfc3986::uri &uri,
opts opts)
:request
{
out, uri, opts.set("GET")
}
{}
inline ircd::rest::opts &&
ircd::rest::opts::set(const string_view &method,
const string_view &content)
{
this->method = method;
this->content = content?: this->content;
return std::move(*this);
}

View file

@ -70,6 +70,7 @@ struct ircd::rfc3986::uri
string_view path; string_view path;
string_view query; string_view query;
string_view fragment; string_view fragment;
string_view resource() const; // path and query string as one
uri(const string_view &); uri(const string_view &);
uri() = default; uri() = default;
@ -179,3 +180,12 @@ namespace ircd::rfc3986
void valid_remote(const string_view &); void valid_remote(const string_view &);
bool valid_remote(std::nothrow_t, const string_view &); bool valid_remote(std::nothrow_t, const string_view &);
} }
inline ircd::string_view
ircd::rfc3986::uri::resource()
const
{
return query?
string_view(begin(path), end(query)):
path;
}

View file

@ -93,7 +93,8 @@ struct ircd::server::link
void submit(request &); void submit(request &);
// control panel // control panel
bool close(const net::close_opts & = net::close_opts_default); bool close(const net::close_opts &);
bool close(const net::dc = net::dc::SSL_NOTIFY);
bool open(const net::open_opts &); bool open(const net::open_opts &);
link(server::peer &); link(server::peer &);

View file

@ -24,23 +24,24 @@ namespace ircd::simd
/// T = inner aligned type /// T = inner aligned type
template<class T> template<class T>
struct struct
__attribute__((packed)) [[using clang: internal_linkage]]
__attribute__((aligned(1))) [[using gnu: packed, aligned(1), visibility("internal")]]
__attribute__((visibility("internal")))
ircd::simd::unaligned ircd::simd::unaligned
{ {
using value_type = T; using value_type = T;
T val; T val;
operator T() const [[gnu::always_inline]]
operator T() const noexcept
{ {
return val; return val;
} }
template<class U> template<class U>
unaligned(U&& val) [[gnu::always_inline]]
:val(std::forward<U>(val)) unaligned(const U val) noexcept
:val(val)
{} {}
}; };

View file

@ -16,13 +16,13 @@ namespace ircd
struct string_view; struct string_view;
constexpr size_t _constexpr_strlen(const char *) noexcept; constexpr size_t _constexpr_strlen(const char *) noexcept;
template<size_t N> constexpr size_t _constexpr_strlen(const char (&)[N]) noexcept; template<size_t N> consteval size_t _constexpr_strlen(const char (&)[N]) noexcept;
constexpr bool _constexpr_equal(const char *a, const char *b) noexcept; constexpr bool _constexpr_equal(const char *a, const char *b) noexcept;
constexpr bool _constexpr_equal(const char *a, const char *b, size_t) noexcept; constexpr bool _constexpr_equal(const char *a, const char *b, size_t) noexcept;
constexpr bool _constexpr_equal(const char *a, const size_t, const char *b, const size_t) noexcept; constexpr bool _constexpr_equal(const char *a, const size_t, const char *b, const size_t) noexcept;
constexpr bool _constexpr_equal(const string_view &, const string_view &) noexcept; constexpr bool _constexpr_equal(const string_view &, const string_view &) noexcept;
template<size_t N0, size_t N1> constexpr bool _constexpr_equal(const char (&)[N0], const char (&)[N1]) noexcept; template<size_t N0, size_t N1> consteval bool _constexpr_equal(const char (&)[N0], const char (&)[N1]) noexcept;
constexpr const char *data(const string_view &) noexcept; constexpr const char *data(const string_view &) noexcept;
constexpr size_t size(const string_view &) noexcept; constexpr size_t size(const string_view &) noexcept;
@ -327,7 +327,7 @@ noexcept
template<size_t N0, template<size_t N0,
size_t N1> size_t N1>
constexpr bool consteval bool
ircd::_constexpr_equal(const char (&a)[N0], ircd::_constexpr_equal(const char (&a)[N0],
const char (&b)[N1]) const char (&b)[N1])
noexcept noexcept
@ -347,7 +347,10 @@ ircd::_constexpr_equal(const string_view &a,
const string_view &b) const string_view &b)
noexcept noexcept
{ {
return _constexpr_equal(data(a), size(a), data(b), size(b)); if constexpr(__cplusplus >= 202002L)
return a == b;
else
return _constexpr_equal(data(a), size(a), data(b), size(b));
} }
constexpr bool constexpr bool
@ -378,7 +381,7 @@ noexcept
} }
template<size_t N> template<size_t N>
constexpr size_t consteval size_t
ircd::_constexpr_strlen(const char (&a)[N]) ircd::_constexpr_strlen(const char (&a)[N])
noexcept noexcept
{ {

View file

@ -107,6 +107,7 @@ constexpr void
ircd::util::bitset<N>::set(const size_t pos, ircd::util::bitset<N>::set(const size_t pos,
const bool val) const bool val)
{ {
reset(pos);
buf[byte(pos)] |= word(val) << bit(pos); buf[byte(pos)] |= word(val) << bit(pos);
} }
@ -152,7 +153,9 @@ const
{ {
constexpr auto max(words - 1); constexpr auto max(words - 1);
const auto off(pos / 8); const auto off(pos / 8);
assert(off <= max); if(!__builtin_is_constant_evaluated())
assert(off <= max);
return std::min(off, max); return std::min(off, max);
} }

View file

@ -140,6 +140,34 @@ struct is_specialization_of<T<args...>, T>
:std::true_type :std::true_type
{}; {};
//
// Convenience signedness cast template
//
template<class T>
using enable_if_s2u = std::enable_if<std::is_signed<T>::value, typename std::make_unsigned<T>::type>;
template<class T>
[[using gnu: always_inline, gnu_inline, artificial]]
extern inline typename enable_if_s2u<T>::type *
sign_cast(T *const t)
{
using type = typename std::make_unsigned<T>::type;
return reinterpret_cast<type *>(t);
}
template<class T>
using enable_if_u2s = std::enable_if<std::is_unsigned<T>::value, typename std::make_signed<T>::type>;
template<class T>
[[using gnu: always_inline, gnu_inline, artificial]]
extern inline typename enable_if_u2s<T>::type *
sign_cast(T *const t)
{
using type = typename std::make_signed<T>::type;
return reinterpret_cast<type *>(t);
}
// //
// Convenience const_cast deduction template // Convenience const_cast deduction template
// //

View file

@ -32,41 +32,24 @@ template<class container,
class iterator> class iterator>
struct ircd::util::unique_iterator struct ircd::util::unique_iterator
{ {
container *c; container *c {nullptr};
iterator it; iterator it;
unique_iterator(container &c, iterator it) operator const iterator &() const;
:c{&c} decltype(auto) operator->() const;
,it{std::move(it)} decltype(auto) operator*() const;
{}
unique_iterator() operator iterator &();
:c{nullptr} decltype(auto) operator->();
{} decltype(auto) operator*();
unique_iterator(container &c, iterator it);
unique_iterator() = default;
unique_iterator(unique_iterator &&o) noexcept;
unique_iterator(const unique_iterator &) = delete; unique_iterator(const unique_iterator &) = delete;
unique_iterator(unique_iterator &&o) noexcept unique_iterator &operator=(unique_iterator &&o) noexcept;
:c{std::move(o.c)}
,it{std::move(o.it)}
{
o.c = nullptr;
}
unique_iterator &operator=(const unique_iterator &) = delete; unique_iterator &operator=(const unique_iterator &) = delete;
unique_iterator &operator=(unique_iterator &&o) noexcept ~unique_iterator() noexcept;
{
this->~unique_iterator();
c = std::move(o.c);
it = std::move(o.it);
o.c = nullptr;
return *this;
}
~unique_iterator() noexcept
{
if(c)
c->erase(it);
}
}; };
template<class container> template<class container>
@ -76,3 +59,89 @@ struct ircd::util::unique_const_iterator
using iterator_type = typename container::const_iterator; using iterator_type = typename container::const_iterator;
using unique_iterator<container, iterator_type>::unique_iterator; using unique_iterator<container, iterator_type>::unique_iterator;
}; };
template<class container,
class iterator>
inline
ircd::util::unique_iterator<container, iterator>::unique_iterator(container &c,
iterator it)
:c{&c}
,it{std::move(it)}
{}
template<class container,
class iterator>
inline
ircd::util::unique_iterator<container, iterator>::unique_iterator(unique_iterator &&o)
noexcept
:c{std::move(o.c)}
,it{std::move(o.it)}
{
o.c = nullptr;
}
template<class container,
class iterator>
inline ircd::util::unique_iterator<container, iterator> &
ircd::util::unique_iterator<container, iterator>::operator=(unique_iterator &&o)
noexcept
{
this->~unique_iterator();
c = std::move(o.c);
it = std::move(o.it);
o.c = nullptr;
return *this;
}
template<class container,
class iterator>
inline
ircd::util::unique_iterator<container, iterator>::~unique_iterator()
noexcept
{
if(c)
c->erase(it);
}
template<class container,
class iterator>
inline decltype(auto)
ircd::util::unique_iterator<container, iterator>::operator*()
{
return it.operator*();
}
template<class container,
class iterator>
inline decltype(auto)
ircd::util::unique_iterator<container, iterator>::operator->()
{
return it.operator->();
}
template<class container,
class iterator>
inline decltype(auto)
ircd::util::unique_iterator<container, iterator>::operator*()
const
{
return it.operator*();
}
template<class container,
class iterator>
inline decltype(auto)
ircd::util::unique_iterator<container, iterator>::operator->()
const
{
return it.operator->();
}
template<class container,
class iterator>
inline ircd::util::unique_iterator<container, iterator>::operator
const iterator &()
const
{
return it;
}

View file

@ -73,16 +73,9 @@ struct ircd::versions
~versions() noexcept; ~versions() noexcept;
}; };
namespace ircd template<>
{ decltype(ircd::versions::list)
template<> ircd::instance_list<ircd::versions>::list;
decltype(versions::allocator)
instance_list<versions>::allocator;
template<>
decltype(versions::list)
instance_list<versions>::list;
}
inline ircd::versions::operator inline ircd::versions::operator
const long &() const long &()

View file

@ -237,6 +237,7 @@ endif
libircd_la_SOURCES += server.cc libircd_la_SOURCES += server.cc
libircd_la_SOURCES += client.cc libircd_la_SOURCES += client.cc
libircd_la_SOURCES += resource.cc libircd_la_SOURCES += resource.cc
libircd_la_SOURCES += rest.cc
if JS if JS
libircd_la_SOURCES += js.cc libircd_la_SOURCES += js.cc
endif endif
@ -435,6 +436,8 @@ gpt_gpu.spv.cc: gpt_gpu.spv
# GCN-HSA # GCN-HSA
# #
if AMD64
GCN_HSA_TARGET = amdgcn--amdhsa GCN_HSA_TARGET = amdgcn--amdhsa
GCN_HSA_CPPFLAGS = $(GPU_CPPFLAGS) GCN_HSA_CPPFLAGS = $(GPU_CPPFLAGS)
@ -469,10 +472,14 @@ libircd_la_SOURCES += gpt_gpu.gcn_hsa.bc.cc
gpt_gpu.gcn_hsa.bc.cc: gpt_gpu.gcn_hsa.bc gpt_gpu.gcn_hsa.bc.cc: gpt_gpu.gcn_hsa.bc
xxd -i $^ $@ xxd -i $^ $@
endif # AMD64
# #
# R600 # R600
# #
if AMD64
R600_TARGET = r600-- R600_TARGET = r600--
# #
@ -510,6 +517,8 @@ libircd_la_SOURCES += gpt_gpu.r600_barts.bc.cc
gpt_gpu.r600_barts.bc.cc: gpt_gpu.r600_barts.bc gpt_gpu.r600_barts.bc.cc: gpt_gpu.r600_barts.bc
xxd -i $^ $@ xxd -i $^ $@
endif # AMD64
# #
# #
# #

View file

@ -29,7 +29,7 @@ namespace ircd::b64
static u8x64 decode_block(const u8x64 block, i64x8 &__restrict__ err) noexcept; static u8x64 decode_block(const u8x64 block, i64x8 &__restrict__ err) noexcept;
[[IRCD_CLONES(IRCD_B64_TARGETS)]] [[IRCD_CLONES(IRCD_B64_TARGETS)]]
static u8x64 encode_block(const u8x64 block, const dictionary &) noexcept; static u8x64 encode_block(const u8x64 block, const dictionary) noexcept;
} }
#pragma GCC visibility pop #pragma GCC visibility pop
@ -157,7 +157,7 @@ alignas(64)
ircd::string_view ircd::string_view
ircd::b64::encode(const mutable_buffer out, ircd::b64::encode(const mutable_buffer out,
const const_buffer in, const const_buffer in,
const dictionary &dict) const dictionary dict)
noexcept noexcept
{ {
const auto pads const auto pads
@ -193,19 +193,9 @@ noexcept
ircd::string_view ircd::string_view
ircd::b64::encode_unpadded(const mutable_buffer out, ircd::b64::encode_unpadded(const mutable_buffer out,
const const_buffer in, const const_buffer in,
const dictionary &dict) const dictionary dict)
noexcept noexcept
{ {
char *const __restrict__ dst
{
data(out)
};
const char *const __restrict__ src
{
data(in)
};
const size_t res_len const size_t res_len
{ {
encode_unpadded_size(in) encode_unpadded_size(in)
@ -216,38 +206,49 @@ noexcept
std::min(res_len, size(out)) std::min(res_len, size(out))
}; };
u8x64 block {0}; uint i;
size_t i(0), j(0); for(i = 0; i < size(in) / 48 && i < out_len / 64; ++i)
for(; i < size(in) / 48 && i < out_len / 64; ++i)
{ {
// Destination is indexed at 64 byte stride // Destination is indexed at 64 byte stride
const auto di u512x1_u *const __restrict__ dx
{ {
reinterpret_cast<u512x1_u *__restrict__>(dst + (i * 64)) reinterpret_cast<u512x1_u *>(data(out))
}; };
// Source is indexed at 48 byte stride // Source is indexed at 48 byte stride
const auto si const auto *const __restrict__ si
{ {
reinterpret_cast<const u512x1_u *__restrict__>(src + (i * 48)) data(in) + i * 48
}; };
block = *si; u8x64 block {0};
#pragma clang loop vectorize(enable) unroll(full)
for(uint j(0); j < 48; ++j)
block[j] = si[j];
block = encode_block(block, dict); block = encode_block(block, dict);
*di = block; dx[i] = block;
} }
for(; i * 48 < size(in) && i * 64 < out_len; ++i) for(; i * 48 < size(in) && i * 64 < out_len; ++i)
{ {
#if !defined(__AVX__) auto *const __restrict__ di
#pragma clang loop unroll_count(2) {
#endif data(out) + i * 64
for(j = 0; j < 48 && i * 48 + j < size(in); ++j) };
block[j] = src[i * 48 + j];
const auto *const __restrict__ si
{
data(in) + i * 48
};
u8x64 block {0};
for(uint j(0); j < 48 && i * 48 + j < size(in); ++j)
block[j] = si[j];
block = encode_block(block, dict); block = encode_block(block, dict);
for(j = 0; j < 64 && i * 64 + j < out_len; ++j) for(uint j(0); j < 64 && i * 64 + j < out_len; ++j)
dst[i * 64 + j] = block[j]; di[j] = block[j];
} }
return string_view return string_view
@ -270,7 +271,7 @@ noexcept
[[IRCD_CLONES(IRCD_B64_TARGETS)]] [[IRCD_CLONES(IRCD_B64_TARGETS)]]
ircd::u8x64 ircd::u8x64
ircd::b64::encode_block(const u8x64 in, ircd::b64::encode_block(const u8x64 in,
const dictionary &dict) const dictionary dict)
noexcept noexcept
{ {
size_t i, j, k; size_t i, j, k;
@ -319,16 +320,6 @@ ircd::const_buffer
ircd::b64::decode(const mutable_buffer out, ircd::b64::decode(const mutable_buffer out,
const string_view in) const string_view in)
{ {
char *const __restrict__ dst
{
data(out)
};
const char *const __restrict__ src
{
data(in)
};
const size_t pads const size_t pads
{ {
endswith_count(in, '=') endswith_count(in, '=')
@ -344,39 +335,51 @@ ircd::b64::decode(const mutable_buffer out,
std::min(decode_size(in_len), size(out)) std::min(decode_size(in_len), size(out))
}; };
uint i;
i64x8 err {0}; i64x8 err {0};
u8x64 block {0}; for(i = 0; i < in_len / 64 && i < out_len / 48; ++i)
size_t i(0), j(0);
for(; i < in_len / 64 && i < out_len / 48; ++i)
{ {
// Destination is indexed at 48 byte stride // Destination is indexed at 48 byte stride
const auto di auto *const __restrict__ di
{ {
reinterpret_cast<u512x1_u *__restrict__>(dst + (i * 48)) data(out) + i * 48
}; };
// Source is indexed at 64 byte stride // Source is indexed at 64 byte stride
const auto si const u512x1_u *const __restrict__ sx
{ {
reinterpret_cast<const u512x1_u *__restrict__>(src + (i * 64)) reinterpret_cast<const u512x1_u *>(data(in))
}; };
block = *si; u8x64 block;
block = sx[i];
block = decode_block(block, err); block = decode_block(block, err);
*di = block; #pragma clang loop vectorize(enable) unroll(full)
for(uint j(0); j < 48; ++j)
di[j] = block[j];
} }
for(; i * 64 < in_len && i * 48 < out_len; ++i) for(; i * 64 < in_len && i * 48 < out_len; ++i)
{ {
u8x64 mask {0}; auto *const __restrict__ di
for(j = 0; j < 64 && i * 64 + j < in_len; ++j) {
block[j] = src[i * 64 + j], data(out) + i * 48
};
const auto *const __restrict__ si
{
data(in) + i * 64
};
u8x64 block {0}, mask {0};
for(uint j(0); j < 64 && i * 64 + j < in_len; ++j)
block[j] = si[j],
mask[j] = 0xff; mask[j] = 0xff;
i64x8 _err {0}; i64x8 _err {0};
block = decode_block(block, _err); block = decode_block(block, _err);
for(j = 0; j < 48 && i * 48 + j < out_len; ++j) for(uint j(0); j < 48 && i * 48 + j < out_len; ++j)
dst[i * 48 + j] = block[j]; di[j] = block[j];
err |= _err & i64x8(mask); err |= _err & i64x8(mask);
} }
@ -438,7 +441,7 @@ noexcept
u8x64 c(b), ret; u8x64 c(b), ret;
#pragma clang loop vectorize(enable) unroll(full) #pragma clang loop vectorize(enable) unroll(full)
for(i = 0; i < 64; ++i) for(i = 0; i < 48; ++i)
ret[i] = c[decode_permute_tab_le[i]]; ret[i] = c[decode_permute_tab_le[i]];
err |= i64x8(_err); err |= i64x8(_err);

View file

@ -192,8 +192,8 @@ ircd::cl::envs
{ "default", "true" }, { "default", "true" },
}, },
{ {
{ "name", "MESA_GLSL_CACHE_DISABLE" }, { "name", "MESA_SHADER_CACHE_DISABLE" },
{ "default", "true" }, { "default", "true" },
}, },
{ {
{ "name", "AMD_DEBUG" }, { "name", "AMD_DEBUG" },
@ -355,6 +355,7 @@ ircd::cl::init::fini_libs()
size_t size_t
ircd::cl::init::init_platforms() ircd::cl::init::init_platforms()
try
{ {
// OpenCL sez platform=null is implementation defined. // OpenCL sez platform=null is implementation defined.
constexpr auto ignore(CL_INVALID_PLATFORM); constexpr auto ignore(CL_INVALID_PLATFORM);
@ -365,9 +366,21 @@ ircd::cl::init::init_platforms()
return platforms; return platforms;
} }
catch(const std::exception &e)
{
log::logf
{
log, log::level::DERROR,
"OpenCL platforms initialization :%s",
e.what(),
};
return 0;
}
size_t size_t
ircd::cl::init::init_devices() ircd::cl::init::init_devices()
try
{ {
// Get the devices. // Get the devices.
size_t devices_total(0); size_t devices_total(0);
@ -414,6 +427,16 @@ ircd::cl::init::init_devices()
return devices_total; return devices_total;
} }
catch(const std::exception &e)
{
log::error
{
log, "OpenCL devices initialization :%s",
e.what(),
};
return 0;
}
size_t size_t
ircd::cl::init::init_ctxs() ircd::cl::init::init_ctxs()

View file

@ -587,7 +587,13 @@ try
loghead(client), loghead(client),
}; };
client.close(net::dc::SSL_NOTIFY, net::close_ignore); const net::close_opts opts
{
.type = net::dc::SSL_NOTIFY,
.shutdown = net::dc::RST,
};
client.close(opts, net::close_ignore);
return false; return false;
} }
catch(const std::exception &e) catch(const std::exception &e)
@ -1048,6 +1054,15 @@ ircd::client::discard_unconsumed(const http::request::head &head)
assert(content_consumed == head.content_length); assert(content_consumed == head.content_length);
} }
ircd::ctx::future<void>
ircd::client::close(const net::dc type)
{
return close(net::close_opts
{
.type = type,
});
}
ircd::ctx::future<void> ircd::ctx::future<void>
ircd::client::close(const net::close_opts &opts) ircd::client::close(const net::close_opts &opts)
{ {
@ -1056,6 +1071,18 @@ ircd::client::close(const net::close_opts &opts)
ctx::already; ctx::already;
} }
void
ircd::client::close(const net::dc type,
net::close_callback callback)
{
const net::close_opts opts
{
.type = type,
};
close(opts, std::move(callback));
}
void void
ircd::client::close(const net::close_opts &opts, ircd::client::close(const net::close_opts &opts,
net::close_callback callback) net::close_callback callback)

View file

@ -179,10 +179,10 @@ try
fs::base::db fs::base::db
}; };
if(!fs::is_dir(dbdir) && (ircd::read_only || ircd::write_avoid)) if(!fs::is_dir(dbdir) && (ircd::read_only || ircd::maintenance))
log::warning log::warning
{ {
log, "Not creating database directory `%s' in read-only/write-avoid mode.", dbdir log, "Not creating database directory `%s' in read-only/maintenance mode.", dbdir
}; };
else if(fs::mkdir(dbdir)) else if(fs::mkdir(dbdir))
log::notice log::notice
@ -2358,15 +2358,20 @@ ircd::db::seek(domain::const_iterator_base &it,
{ {
switch(p) switch(p)
{ {
// This is inefficient as per RocksDB's prefix impl.
case pos::BACK: case pos::BACK:
{ {
// This is inefficient as per RocksDB's prefix impl. unknown why char buf[512];
// a seek to NEXT is still needed after walking back one. string_view key;
assert(bool(it)); do
{
assert(size(it.it->key()) <= sizeof(buf));
key = string_view(buf, copy(buf, slice(it.it->key())));
}
while(seek(it, pos::NEXT)); while(seek(it, pos::NEXT));
if(seek(it, pos::PREV))
seek(it, pos::NEXT);
return bool(it); assert(key);
return seek(it, key);
} }
default: default:
@ -2377,14 +2382,6 @@ ircd::db::seek(domain::const_iterator_base &it,
return seek(static_cast<column::const_iterator_base &>(it), p); return seek(static_cast<column::const_iterator_base &>(it), p);
} }
bool
ircd::db::seek(domain::const_iterator_base &it,
const string_view &p)
{
it.opts.prefix = true;
return seek(static_cast<column::const_iterator_base &>(it), p);
}
ircd::db::domain::const_iterator ircd::db::domain::const_iterator
ircd::db::domain::begin(const string_view &key, ircd::db::domain::begin(const string_view &key,
gopts opts) gopts opts)
@ -3204,7 +3201,7 @@ rocksdb::Cache *
ircd::db::cache_compressed(column &column) ircd::db::cache_compressed(column &column)
{ {
database::column &c(column); database::column &c(column);
return c.table_opts.block_cache_compressed.get(); return nullptr; // c.table_opts.block_cache_compressed.get();
} }
[[gnu::hot]] [[gnu::hot]]
@ -3219,7 +3216,7 @@ const rocksdb::Cache *
ircd::db::cache_compressed(const column &column) ircd::db::cache_compressed(const column &column)
{ {
const database::column &c(column); const database::column &c(column);
return c.table_opts.block_cache_compressed.get(); return nullptr; // c.table_opts.block_cache_compressed.get();
} }
template<> template<>
@ -3852,9 +3849,16 @@ ircd::db::options::operator rocksdb::PlainTableOptions()
const const
{ {
rocksdb::PlainTableOptions ret; rocksdb::PlainTableOptions ret;
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
#endif
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetPlainTableOptionsFromString(opts, ret, *this, &ret)
#else
rocksdb::GetPlainTableOptionsFromString(ret, *this, &ret) rocksdb::GetPlainTableOptionsFromString(ret, *this, &ret)
#endif
}; };
return ret; return ret;
@ -3864,9 +3868,16 @@ ircd::db::options::operator rocksdb::BlockBasedTableOptions()
const const
{ {
rocksdb::BlockBasedTableOptions ret; rocksdb::BlockBasedTableOptions ret;
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
#endif
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetBlockBasedTableOptionsFromString(opts, ret, *this, &ret)
#else
rocksdb::GetBlockBasedTableOptionsFromString(ret, *this, &ret) rocksdb::GetBlockBasedTableOptionsFromString(ret, *this, &ret)
#endif
}; };
return ret; return ret;
@ -3876,9 +3887,16 @@ ircd::db::options::operator rocksdb::ColumnFamilyOptions()
const const
{ {
rocksdb::ColumnFamilyOptions ret; rocksdb::ColumnFamilyOptions ret;
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
#endif
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetColumnFamilyOptionsFromString(opts, ret, *this, &ret)
#else
rocksdb::GetColumnFamilyOptionsFromString(ret, *this, &ret) rocksdb::GetColumnFamilyOptionsFromString(ret, *this, &ret)
#endif
}; };
return ret; return ret;
@ -3888,9 +3906,16 @@ ircd::db::options::operator rocksdb::DBOptions()
const const
{ {
rocksdb::DBOptions ret; rocksdb::DBOptions ret;
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
#endif
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetDBOptionsFromString(opts, ret, *this, &ret)
#else
rocksdb::GetDBOptionsFromString(ret, *this, &ret) rocksdb::GetDBOptionsFromString(ret, *this, &ret)
#endif
}; };
return ret; return ret;
@ -3900,9 +3925,16 @@ ircd::db::options::operator rocksdb::Options()
const const
{ {
rocksdb::Options ret; rocksdb::Options ret;
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
#endif
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetOptionsFromString(opts, ret, *this, &ret)
#else
rocksdb::GetOptionsFromString(ret, *this, &ret) rocksdb::GetOptionsFromString(ret, *this, &ret)
#endif
}; };
return ret; return ret;
@ -3949,52 +3981,88 @@ const
} }
rocksdb::BlockBasedTableOptions rocksdb::BlockBasedTableOptions
ircd::db::options::map::merge(const rocksdb::BlockBasedTableOptions &opts) ircd::db::options::map::merge(const rocksdb::BlockBasedTableOptions &in)
const const
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
opts.ignore_unknown_options = true;
#endif
rocksdb::BlockBasedTableOptions ret; rocksdb::BlockBasedTableOptions ret;
throw_on_error throw_on_error
{ {
rocksdb::GetBlockBasedTableOptionsFromMap(opts, *this, &ret, true, true) #ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetBlockBasedTableOptionsFromMap(opts, in, *this, &ret)
#else
rocksdb::GetBlockBasedTableOptionsFromMap(in, *this, &ret, true, true)
#endif
}; };
return ret; return ret;
} }
rocksdb::PlainTableOptions rocksdb::PlainTableOptions
ircd::db::options::map::merge(const rocksdb::PlainTableOptions &opts) ircd::db::options::map::merge(const rocksdb::PlainTableOptions &in)
const const
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
opts.ignore_unknown_options = true;
#endif
rocksdb::PlainTableOptions ret; rocksdb::PlainTableOptions ret;
throw_on_error throw_on_error
{ {
rocksdb::GetPlainTableOptionsFromMap(opts, *this, &ret, true, true) #ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetPlainTableOptionsFromMap(opts, in, *this, &ret)
#else
rocksdb::GetPlainTableOptionsFromMap(in, *this, &ret, true, true)
#endif
}; };
return ret; return ret;
} }
rocksdb::ColumnFamilyOptions rocksdb::ColumnFamilyOptions
ircd::db::options::map::merge(const rocksdb::ColumnFamilyOptions &opts) ircd::db::options::map::merge(const rocksdb::ColumnFamilyOptions &in)
const const
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
opts.ignore_unknown_options = true;
#endif
rocksdb::ColumnFamilyOptions ret; rocksdb::ColumnFamilyOptions ret;
throw_on_error throw_on_error
{ {
rocksdb::GetColumnFamilyOptionsFromMap(opts, *this, &ret, true, true) #ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetColumnFamilyOptionsFromMap(opts, in, *this, &ret)
#else
rocksdb::GetColumnFamilyOptionsFromMap(in, *this, &ret, true, true)
#endif
}; };
return ret; return ret;
} }
rocksdb::DBOptions rocksdb::DBOptions
ircd::db::options::map::merge(const rocksdb::DBOptions &opts) ircd::db::options::map::merge(const rocksdb::DBOptions &in)
const const
{ {
#ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::ConfigOptions opts;
opts.ignore_unknown_options = true;
#endif
rocksdb::DBOptions ret; rocksdb::DBOptions ret;
throw_on_error throw_on_error
{ {
rocksdb::GetDBOptionsFromMap(opts, *this, &ret, true, true) #ifdef IRCD_DB_HAS_CONFIG_OPTIONS
rocksdb::GetDBOptionsFromMap(opts, in, *this, &ret)
#else
rocksdb::GetDBOptionsFromMap(in, *this, &ret, true, true)
#endif
}; };
return ret; return ret;
@ -4053,11 +4121,19 @@ ircd::db::insert(rocksdb::Cache &cache,
// the argument execution doesn't throw after release() // the argument execution doesn't throw after release()
throw_on_error throw_on_error
{ {
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
cache.Insert(slice(key),
mutable_cast(data(value.release())),
cache.GetCacheItemHelper(nullptr), // ???
value_size,
nullptr)
#else
cache.Insert(slice(key), cache.Insert(slice(key),
mutable_cast(data(value.release())), mutable_cast(data(value.release())),
value_size, value_size,
deleter, deleter,
nullptr) nullptr)
#endif
}; };
return true; return true;
@ -4066,6 +4142,24 @@ ircd::db::insert(rocksdb::Cache &cache,
void void
ircd::db::for_each(const rocksdb::Cache &cache, ircd::db::for_each(const rocksdb::Cache &cache,
const cache_closure &closure) const cache_closure &closure)
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
{
const auto _closure{[&closure]
(const auto &slice, void *const value, size_t size, const auto *const helper)
noexcept
{
const const_buffer buf
{
reinterpret_cast<const char *>(value), size
};
closure(buf);
}};
rocksdb::Cache::ApplyToAllEntriesOptions opts;
mutable_cast(cache).ApplyToAllEntries(_closure, opts);
}
#else
{ {
// Due to the use of the global variables which are required when using a // Due to the use of the global variables which are required when using a
// C-style callback for RocksDB, we have to make use of this function // C-style callback for RocksDB, we have to make use of this function
@ -4094,6 +4188,7 @@ ircd::db::for_each(const rocksdb::Cache &cache,
}, },
true); true);
} }
#endif
#ifdef IRCD_DB_HAS_CACHE_GETCHARGE #ifdef IRCD_DB_HAS_CACHE_GETCHARGE
size_t size_t
@ -4794,7 +4889,7 @@ ircd::db::_read(const vector_view<_read_op> &op,
const bool parallelize const bool parallelize
{ {
#ifdef IRCD_DB_HAS_MULTIGET_DIRECT #if defined(IRCD_DB_HAS_MULTIGET_DIRECT) && defined(IRCD_DB_HAS_MULTIREAD_FIX)
true && num > 1 true && num > 1
#else #else
false false
@ -4913,24 +5008,23 @@ ircd::db::_seek(const vector_view<_read_op> &op,
namespace ircd::db namespace ircd::db
{ {
static rocksdb::Iterator &_seek_(rocksdb::Iterator &, const pos &); static rocksdb::Iterator &_seek_(rocksdb::Iterator &, const pos &);
static rocksdb::Iterator &_seek_(rocksdb::Iterator &, const string_view &); static rocksdb::Iterator &_seek_(rocksdb::Iterator &, const string_view &, const bool lte);
static rocksdb::Iterator &_seek_lower_(rocksdb::Iterator &, const string_view &); static bool _seek(database::column &, const pos &, const rocksdb::ReadOptions &, rocksdb::Iterator &it, const bool lte);
static rocksdb::Iterator &_seek_upper_(rocksdb::Iterator &, const string_view &); static bool _seek(database::column &, const string_view &, const rocksdb::ReadOptions &, rocksdb::Iterator &it, const bool lte);
static bool _seek(database::column &, const pos &, const rocksdb::ReadOptions &, rocksdb::Iterator &it);
static bool _seek(database::column &, const string_view &, const rocksdb::ReadOptions &, rocksdb::Iterator &it);
} }
std::unique_ptr<rocksdb::Iterator> std::unique_ptr<rocksdb::Iterator>
ircd::db::seek(column &column, ircd::db::seek(column &column,
const string_view &key, const string_view &key,
const gopts &opts) const gopts &opts,
const bool lte)
{ {
database &d(column); database &d(column);
database::column &c(column); database::column &c(column);
std::unique_ptr<rocksdb::Iterator> ret; std::unique_ptr<rocksdb::Iterator> ret;
const auto ropts(make_opts(opts)); const auto ropts(make_opts(opts));
seek(c, key, ropts, ret); seek(c, key, ropts, ret, lte);
return ret; return ret;
} }
@ -4939,7 +5033,8 @@ bool
ircd::db::seek(database::column &c, ircd::db::seek(database::column &c,
const pos &p, const pos &p,
const rocksdb::ReadOptions &opts, const rocksdb::ReadOptions &opts,
std::unique_ptr<rocksdb::Iterator> &it) std::unique_ptr<rocksdb::Iterator> &it,
const bool lte)
{ {
const ctx::uninterruptible ui; const ctx::uninterruptible ui;
const ctx::stack_usage_assertion sua; const ctx::stack_usage_assertion sua;
@ -4951,31 +5046,33 @@ ircd::db::seek(database::column &c,
it.reset(d.d->NewIterator(opts, cf)); it.reset(d.d->NewIterator(opts, cf));
} }
return _seek(c, p, opts, *it); return _seek(c, p, opts, *it, lte);
} }
bool bool
ircd::db::_seek(database::column &c, ircd::db::_seek(database::column &c,
const string_view &p, const string_view &p,
const rocksdb::ReadOptions &opts, const rocksdb::ReadOptions &opts,
rocksdb::Iterator &it) rocksdb::Iterator &it,
const bool lte)
try try
{ {
util::timer timer{util::timer::nostart}; util::timer timer{util::timer::nostart};
if constexpr(RB_DEBUG_DB_SEEK) if constexpr(RB_DEBUG_DB_SEEK)
timer = util::timer{}; timer = util::timer{};
_seek_(it, p); _seek_(it, p, lte);
database &d(*c.d); database &d(*c.d);
if constexpr(RB_DEBUG_DB_SEEK) if constexpr(RB_DEBUG_DB_SEEK)
log::debug log::debug
{ {
log, "[%s] %lu:%lu SEEK %s %s in %ld$us '%s'", log, "[%s] %lu:%lu SEEK[%s] %s %s in %ld$us '%s'",
name(d), name(d),
sequence(d), sequence(d),
sequence(opts.snapshot), sequence(opts.snapshot),
valid(it)? "VALID" : "INVALID", lte? "LTE"_sv: "GTE"_sv,
valid(it)? "VALID"_sv: "INVALID"_sv,
it.status().ok()? "OK"s: it.status().ToString(), it.status().ok()? "OK"s: it.status().ToString(),
timer.at<microseconds>().count(), timer.at<microseconds>().count(),
name(c) name(c)
@ -4988,11 +5085,12 @@ catch(const error &e)
const database &d(*c.d); const database &d(*c.d);
log::critical log::critical
{ {
log, "[%s][%s] %lu:%lu SEEK key :%s", log, "[%s][%s] %lu:%lu SEEK[%s] key :%s",
name(d), name(d),
name(c), name(c),
sequence(d), sequence(d),
sequence(opts.snapshot), sequence(opts.snapshot),
lte? "LTE"_sv: "GTE"_sv,
e.what(), e.what(),
}; };
@ -5003,7 +5101,8 @@ bool
ircd::db::_seek(database::column &c, ircd::db::_seek(database::column &c,
const pos &p, const pos &p,
const rocksdb::ReadOptions &opts, const rocksdb::ReadOptions &opts,
rocksdb::Iterator &it) rocksdb::Iterator &it,
const bool)
try try
{ {
bool valid_it; bool valid_it;
@ -5025,7 +5124,7 @@ try
sequence(d), sequence(d),
sequence(opts.snapshot), sequence(opts.snapshot),
reflect(p), reflect(p),
valid_it? "VALID" : "INVALID", valid_it? "VALID"_sv: "INVALID"_sv,
it.status().ok()? "OK"s: it.status().ToString(), it.status().ok()? "OK"s: it.status().ToString(),
timer.at<microseconds>().count(), timer.at<microseconds>().count(),
name(c) name(c)
@ -5044,41 +5143,27 @@ catch(const error &e)
sequence(d), sequence(d),
sequence(opts.snapshot), sequence(opts.snapshot),
reflect(p), reflect(p),
it.Valid()? "VALID" : "INVALID", it.Valid()? "VALID"_sv: "INVALID"_sv,
e.what(), e.what(),
}; };
throw; throw;
} }
/// Seek to entry NOT GREATER THAN key. That is, equal to or less than key
rocksdb::Iterator &
ircd::db::_seek_lower_(rocksdb::Iterator &it,
const string_view &sv)
{
assert(!ctx::interruptible());
it.SeekForPrev(slice(sv));
return it;
}
/// Seek to entry NOT LESS THAN key. That is, equal to or greater than key
rocksdb::Iterator &
ircd::db::_seek_upper_(rocksdb::Iterator &it,
const string_view &sv)
{
assert(!ctx::interruptible());
it.Seek(slice(sv));
return it;
}
/// Defaults to _seek_upper_ because it has better support from RocksDB. /// Defaults to _seek_upper_ because it has better support from RocksDB.
rocksdb::Iterator & rocksdb::Iterator &
ircd::db::_seek_(rocksdb::Iterator &it, ircd::db::_seek_(rocksdb::Iterator &it,
const string_view &sv) const string_view &sv,
const bool lte)
{ {
return _seek_upper_(it, sv); assert(!ctx::interruptible());
if(lte)
it.SeekForPrev(slice(sv));
else
it.Seek(slice(sv));
return it;
} }
rocksdb::Iterator & rocksdb::Iterator &
@ -5089,11 +5174,24 @@ ircd::db::_seek_(rocksdb::Iterator &it,
switch(p) switch(p)
{ {
case pos::NEXT: it.Next(); break; case pos::NEXT:
case pos::PREV: it.Prev(); break; assert(valid(it));
case pos::FRONT: it.SeekToFirst(); break; it.Next();
case pos::BACK: it.SeekToLast(); break; break;
default:
case pos::PREV:
assert(valid(it));
it.Prev();
break;
case pos::FRONT:
it.SeekToFirst();
break;
case pos::BACK:
it.SeekToLast();
break;
case pos::END: case pos::END:
{ {
it.SeekToLast(); it.SeekToLast();
@ -5102,6 +5200,10 @@ ircd::db::_seek_(rocksdb::Iterator &it,
break; break;
} }
default:
assert(false);
break;
} }
return it; return it;
@ -5825,6 +5927,9 @@ ircd::db::reflect(const rocksdb::CompactionReason &r)
#ifdef IRCD_DB_HAS_ROUND_ROBIN_TTL #ifdef IRCD_DB_HAS_ROUND_ROBIN_TTL
case Reason::kRoundRobinTtl: return "kRoundRobinTtl"; case Reason::kRoundRobinTtl: return "kRoundRobinTtl";
#endif #endif
#ifdef IRCD_DB_HAS_REFIT_LEVEL
case Reason::kRefitLevel: return "RefitLevel";
#endif
case Reason::kNumOfReasons: case Reason::kNumOfReasons:
break; break;

View file

@ -61,6 +61,9 @@
#include <rocksdb/compaction_filter.h> #include <rocksdb/compaction_filter.h>
#include <rocksdb/wal_filter.h> #include <rocksdb/wal_filter.h>
#include <rocksdb/rate_limiter.h> #include <rocksdb/rate_limiter.h>
#if __has_include(<rocksdb/advanced_cache.h>)
#include <rocksdb/advanced_cache.h>
#endif
#pragma clang attribute pop #pragma clang attribute pop
#include "db_has.h" #include "db_has.h"
@ -131,10 +134,8 @@ namespace ircd::db
static void valid_eq_or_throw(const rocksdb::Iterator &, const string_view &); static void valid_eq_or_throw(const rocksdb::Iterator &, const string_view &);
// [GET] iterator seek suite // [GET] iterator seek suite
template<class pos> static bool seek(database::column &, const pos &, const rocksdb::ReadOptions &, std::unique_ptr<rocksdb::Iterator> &it); template<class pos> static bool seek(database::column &, const pos &, const rocksdb::ReadOptions &, std::unique_ptr<rocksdb::Iterator> &it, const bool lte = false);
static std::unique_ptr<rocksdb::Iterator> seek(column &, const gopts &); static std::unique_ptr<rocksdb::Iterator> seek(column &, const string_view &key, const gopts &, const bool lte = false);
static std::unique_ptr<rocksdb::Iterator> seek(column &, const string_view &key, const gopts &);
static std::vector<row::value_type> seek(database &, const gopts &);
static std::pair<string_view, string_view> operator*(const rocksdb::Iterator &); static std::pair<string_view, string_view> operator*(const rocksdb::Iterator &);
// [GET] read suite // [GET] read suite
@ -218,8 +219,18 @@ ircd::db::database::cache final
std::shared_ptr<rocksdb::Cache> c; std::shared_ptr<rocksdb::Cache> c;
const char *Name() const noexcept override; const char *Name() const noexcept override;
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
Status Insert(const Slice &key, ObjectPtr, const CacheItemHelper *, size_t charge, Handle **, Priority) noexcept override;
#else
Status Insert(const Slice &key, void *value, size_t charge, deleter, Handle **, Priority) noexcept override; Status Insert(const Slice &key, void *value, size_t charge, deleter, Handle **, Priority) noexcept override;
#endif
#if defined(IRCD_DB_HAS_CACHE_ASYNC)
Handle *Lookup(const Slice &key, const CacheItemHelper *, CreateContext *, Priority, Statistics *) noexcept override;
#elif defined(IRCD_DB_HAS_CACHE_ITEMHELPER)
Handle *Lookup(const Slice &key, const CacheItemHelper *, CreateContext *, Priority, bool, Statistics *) noexcept override;
#else
Handle *Lookup(const Slice &key, Statistics *) noexcept override; Handle *Lookup(const Slice &key, Statistics *) noexcept override;
#endif
bool Ref(Handle *) noexcept override; bool Ref(Handle *) noexcept override;
bool Release(Handle *, bool force_erase) noexcept override; bool Release(Handle *, bool force_erase) noexcept override;
void *Value(Handle *) noexcept override; void *Value(Handle *) noexcept override;
@ -233,19 +244,30 @@ ircd::db::database::cache final
size_t GetUsage(Handle *) const noexcept override; size_t GetUsage(Handle *) const noexcept override;
size_t GetPinnedUsage() const noexcept override; size_t GetPinnedUsage() const noexcept override;
void DisownData() noexcept override; void DisownData() noexcept override;
#ifndef IRCD_DB_HAS_CACHE_ITEMHELPER
void ApplyToAllCacheEntries(callback, bool thread_safe) noexcept override; void ApplyToAllCacheEntries(callback, bool thread_safe) noexcept override;
#endif
void EraseUnRefEntries() noexcept override; void EraseUnRefEntries() noexcept override;
std::string GetPrintableOptions() const noexcept override; std::string GetPrintableOptions() const noexcept override;
#ifdef IRCD_DB_HAS_CACHE_GETCHARGE #ifdef IRCD_DB_HAS_CACHE_GETCHARGE
size_t GetCharge(Handle *) const noexcept override; size_t GetCharge(Handle *) const noexcept override;
#endif #endif
#ifdef IRCD_DB_HAS_CACHE_GETDELETER #if defined(IRCD_DB_HAS_CACHE_GETDELETER) && !defined(IRCD_DB_HAS_CACHE_ITEMHELPER)
DeleterFn GetDeleter(Handle *) const noexcept override; DeleterFn GetDeleter(Handle *) const noexcept override;
#endif #endif
#ifdef IRCD_DB_HAS_CACHE_APPLYTOALL #ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
using callbackstd = std::function<void (const Slice &, ObjectPtr, size_t, const CacheItemHelper *)>;
void ApplyToAllEntries(const callbackstd &, const ApplyToAllEntriesOptions &) noexcept override;
#elif defined(IRCD_DB_HAS_CACHE_APPLYTOALL)
using callbackstd = std::function<void (const Slice &, void *, size_t, DeleterFn)>; using callbackstd = std::function<void (const Slice &, void *, size_t, DeleterFn)>;
void ApplyToAllEntries(const callbackstd &, const ApplyToAllEntriesOptions &) noexcept override; void ApplyToAllEntries(const callbackstd &, const ApplyToAllEntriesOptions &) noexcept override;
#endif #endif
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
const CacheItemHelper *GetCacheItemHelper(Handle *) const noexcept override;
#endif
#ifdef IRCD_DB_HAS_CACHE_ASYNC
Handle *CreateStandalone(const Slice &, ObjectPtr, const CacheItemHelper *, size_t, bool) noexcept override;
#endif
cache(database *const &, cache(database *const &,
std::shared_ptr<struct database::stats>, std::shared_ptr<struct database::stats>,

View file

@ -81,17 +81,87 @@ ircd::db::auto_deletion
{ "persist", false }, { "persist", false },
}; };
/// Conf item dictates whether databases will be opened in slave mode; this /// Gather statistics about files on open to inform the compaction algorithm.
/// is a recent feature of RocksDB which may not be available. It allows two /// This can be disabled to prevent touching a lot of files on open, but it's
/// instances of a database, so long as only one is not opened as a slave. /// unclear when/if that information will be gathered to ever inform compactor.
decltype(ircd::db::open_slave) decltype(ircd::db::open_stats)
ircd::db::open_slave ircd::db::open_stats
{ {
{ "name", "ircd.db.open.slave" }, { "name", "ircd.db.open.stats" },
{ "default", false }, { "default", true },
{ "persist", false }, { "persist", false },
}; };
/// Paranoid suite toggle. This allows coarse control over the rest of the
/// configuration from here. If this is set to false, all other paranoid confs
/// will default to false; note that each conf can still be explicitly set.
decltype(ircd::db::paranoid)
ircd::db::paranoid
{
{ "name", "ircd.db.paranoid.enable" },
{ "default", true },
{ "persist", false },
};
/// General paranoid checks; enabled by rocksdb by default. Disabling this
/// might implicitly override some of the other paranoid features configurable
/// here and disable them within rocksdb.
decltype(ircd::db::paranoid_checks)
ircd::db::paranoid_checks
{
{ "name", "ircd.db.paranoid.checks" },
{ "default", bool(paranoid) },
{ "persist", false },
};
/// Check file sizes on open. This is expensive because it opens every file
/// with only mediocre value for finding corruption in practice.
decltype(ircd::db::paranoid_size)
ircd::db::paranoid_size
{
{ "name", "ircd.db.paranoid.size" },
{ "default", false },
{ "persist", false },
};
/// Check UUIDs of SST files against the manifest.
decltype(ircd::db::paranoid_uuid)
ircd::db::paranoid_uuid
{
{ "name", "ircd.db.paranoid.uuid" },
{ "default", bool(paranoid) },
{ "persist", false },
};
/// Check WAL against the manifest for corruption.
decltype(ircd::db::paranoid_wal)
ircd::db::paranoid_wal
{
{ "name", "ircd.db.paranoid.wal" },
{ "default", bool(paranoid) },
{ "persist", false },
};
/// Check every SST after writing it. This is expensive and rocksdb doesn't
/// enable it in their own defaults.
decltype(ircd::db::paranoid_sst)
ircd::db::paranoid_sst
{
{ "name", "ircd.db.paranoid.sst" },
{ "default", false },
{ "persist", false },
};
/// Check LSM after changing it. This is claimed to be cheap by rocksdb and
/// they enable it by default.
decltype(ircd::db::paranoid_lsm)
ircd::db::paranoid_lsm
{
{ "name", "ircd.db.paranoid.lsm" },
{ "default", bool(paranoid) },
{ "persist", false },
};
void void
ircd::db::sync(database &d) ircd::db::sync(database &d)
{ {
@ -859,7 +929,7 @@ try
} }
,slave ,slave
{ {
db::open_slave ircd::slave
} }
,read_only ,read_only
{ {
@ -968,8 +1038,9 @@ try
opts->bytes_per_sync = 0; opts->bytes_per_sync = 0;
opts->wal_bytes_per_sync = 0; opts->wal_bytes_per_sync = 0;
// This prevents the creation of additional SST files and lots of I/O on // We need flush during shutdown or data which bypasses the WAL might get
// either DB open and close. // desynchronized from data which doesn't (or simply lost). Flush during
// recovery seems to happen anyway but setting to avoid is probably better.
opts->avoid_flush_during_recovery = true; opts->avoid_flush_during_recovery = true;
opts->avoid_flush_during_shutdown = false; opts->avoid_flush_during_shutdown = false;
@ -994,6 +1065,19 @@ try
// Use the determined direct io value for writes as well. // Use the determined direct io value for writes as well.
//opts->use_direct_io_for_flush_and_compaction = opts->use_direct_reads; //opts->use_direct_io_for_flush_and_compaction = opts->use_direct_reads;
// Additional optimizations at the cost of trading some added risk.
opts->skip_stats_update_on_db_open = !open_stats;
opts->paranoid_checks = bool(paranoid_checks);
#ifdef IRCD_DB_HAS_SKIP_CHECKSIZE
opts->skip_checking_sst_file_sizes_on_db_open = !paranoid_size;
#endif
#ifdef IRCD_DB_HAS_MANIFEST_WALS
opts->track_and_verify_wals_in_manifest = bool(paranoid_wal);
#endif
#ifdef IRCD_DB_HAS_MANIFEST_UUIDS
opts->verify_sst_unique_id_in_manifest = bool(paranoid_uuid);
#endif
// Default corruption tolerance is zero-tolerance; db fails to open with // Default corruption tolerance is zero-tolerance; db fails to open with
// error by default to inform the user. The rest of the options are // error by default to inform the user. The rest of the options are
// various relaxations for how to proceed. // various relaxations for how to proceed.
@ -1202,7 +1286,7 @@ try
// If the directory does not exist, though rocksdb will create it, we can // If the directory does not exist, though rocksdb will create it, we can
// avoid scaring the user with an error log message if we just do that.. // avoid scaring the user with an error log message if we just do that..
if(opts->create_if_missing && !fs::is_dir(path) && !ircd::write_avoid) if(opts->create_if_missing && !fs::is_dir(path) && !ircd::maintenance && !ircd::read_only)
fs::mkdir(path); fs::mkdir(path);
// Announce attempt before usual point where exceptions are thrown // Announce attempt before usual point where exceptions are thrown
@ -1745,6 +1829,11 @@ ircd::db::database::column::column(database &d,
} }
,options_preconfiguration{[this] ,options_preconfiguration{[this]
{ {
// Setup sundry
this->options.report_bg_io_stats = true;
this->options.paranoid_file_checks = bool(paranoid_sst);
this->options.force_consistency_checks = bool(paranoid_lsm);
// If possible, deduce comparator based on type given in descriptor // If possible, deduce comparator based on type given in descriptor
if(!this->descriptor->cmp.less) if(!this->descriptor->cmp.less)
{ {
@ -1780,11 +1869,6 @@ ircd::db::database::column::column(database &d,
// Set the compaction filter // Set the compaction filter
this->options.compaction_filter = &this->cfilter; this->options.compaction_filter = &this->cfilter;
//this->options.paranoid_file_checks = true;
// More stats reported by the rocksdb.stats property.
this->options.report_bg_io_stats = true;
// Set filter reductions for this column. This means we expect a key to exist. // Set filter reductions for this column. This means we expect a key to exist.
this->options.optimize_filters_for_hits = this->descriptor->expect_queries_hit; this->options.optimize_filters_for_hits = this->descriptor->expect_queries_hit;
@ -1985,7 +2069,7 @@ ircd::db::database::column::column(database &d,
// Setup the cache for compressed assets. // Setup the cache for compressed assets.
const auto &cache_size_comp(this->descriptor->cache_size_comp); const auto &cache_size_comp(this->descriptor->cache_size_comp);
if(cache_size_comp != 0) if(cache_size_comp != 0)
table_opts.block_cache_compressed = std::make_shared<database::cache>(this->d, this->stats, this->allocator, this->name, cache_size_comp); ; //table_opts.block_cache_compressed = std::make_shared<database::cache>(this->d, this->stats, this->allocator, this->name, cache_size_comp);
// Setup the bloom filter. // Setup the bloom filter.
const auto &bloom_bits(this->descriptor->bloom_bits); const auto &bloom_bits(this->descriptor->bloom_bits);
@ -2059,7 +2143,7 @@ ircd::db::database::column::column(database &d,
this->cmp.Name(), this->cmp.Name(),
this->options.prefix_extractor? this->prefix.Name() : "none", this->options.prefix_extractor? this->prefix.Name() : "none",
table_opts.block_cache? "YES": "NO", table_opts.block_cache? "YES": "NO",
table_opts.block_cache_compressed? "YES": "NO", "NO", //table_opts.block_cache_compressed? "YES": "NO",
this->descriptor->bloom_bits, this->descriptor->bloom_bits,
int(this->options.compression), int(this->options.compression),
this->descriptor->name this->descriptor->name
@ -3182,6 +3266,15 @@ const noexcept
c->Name(); c->Name();
} }
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
rocksdb::Status
ircd::db::database::cache::Insert(const Slice &key,
ObjectPtr value,
const CacheItemHelper *const helper,
size_t charge,
Handle **const handle,
Priority priority)
#else
rocksdb::Status rocksdb::Status
ircd::db::database::cache::Insert(const Slice &key, ircd::db::database::cache::Insert(const Slice &key,
void *const value, void *const value,
@ -3189,6 +3282,7 @@ ircd::db::database::cache::Insert(const Slice &key,
deleter del, deleter del,
Handle **const handle, Handle **const handle,
Priority priority) Priority priority)
#endif
noexcept noexcept
{ {
using rocksdb::Tickers; using rocksdb::Tickers;
@ -3198,7 +3292,11 @@ noexcept
const rocksdb::Status &ret const rocksdb::Status &ret
{ {
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
c->Insert(key, value, helper, charge, handle, priority)
#else
c->Insert(key, value, charge, del, handle, priority) c->Insert(key, value, charge, del, handle, priority)
#endif
}; };
stats->recordTick(Tickers::BLOCK_CACHE_ADD, ret.ok()); stats->recordTick(Tickers::BLOCK_CACHE_ADD, ret.ok());
@ -3223,9 +3321,26 @@ noexcept
return ret; return ret;
} }
#if defined(IRCD_DB_HAS_CACHE_ASYNC)
rocksdb::Cache::Handle *
ircd::db::database::cache::Lookup(const Slice &key,
const CacheItemHelper *const helper,
CreateContext *const cc,
Priority pri,
Statistics *const statistics)
#elif defined(IRCD_DB_HAS_CACHE_ITEMHELPER)
rocksdb::Cache::Handle *
ircd::db::database::cache::Lookup(const Slice &key,
const CacheItemHelper *const helper,
CreateContext *const cc,
Priority pri,
bool wait,
Statistics *const statistics)
#else
rocksdb::Cache::Handle * rocksdb::Cache::Handle *
ircd::db::database::cache::Lookup(const Slice &key, ircd::db::database::cache::Lookup(const Slice &key,
Statistics *const statistics) Statistics *const statistics)
#endif
noexcept noexcept
{ {
using rocksdb::Tickers; using rocksdb::Tickers;
@ -3247,7 +3362,13 @@ noexcept
auto *const &ret auto *const &ret
{ {
#if defined(IRCD_DB_HAS_CACHE_ASYNC)
c->Lookup(key, helper, cc, pri, statistics)
#elif defined(IRCD_DB_HAS_CACHE_ITEMHELPER)
c->Lookup(key, helper, cc, pri, wait, statistics)
#else
c->Lookup(key, s) c->Lookup(key, s)
#endif
}; };
// Rocksdb's LRUCache stats are broke. The statistics ptr is null and // Rocksdb's LRUCache stats are broke. The statistics ptr is null and
@ -3379,6 +3500,7 @@ noexcept
return c->DisownData(); return c->DisownData();
} }
#ifndef IRCD_DB_HAS_CACHE_ITEMHELPER
void void
ircd::db::database::cache::ApplyToAllCacheEntries(callback cb, ircd::db::database::cache::ApplyToAllCacheEntries(callback cb,
bool thread_safe) bool thread_safe)
@ -3387,6 +3509,7 @@ noexcept
assert(bool(c)); assert(bool(c));
return c->ApplyToAllCacheEntries(cb, thread_safe); return c->ApplyToAllCacheEntries(cb, thread_safe);
} }
#endif
void void
ircd::db::database::cache::EraseUnRefEntries() ircd::db::database::cache::EraseUnRefEntries()
@ -3414,7 +3537,7 @@ const noexcept
} }
#endif #endif
#ifdef IRCD_DB_HAS_CACHE_GETDELETER #if defined(IRCD_DB_HAS_CACHE_GETDELETER) && !defined(IRCD_DB_HAS_CACHE_ITEMHELPER)
rocksdb::Cache::DeleterFn rocksdb::Cache::DeleterFn
ircd::db::database::cache::GetDeleter(Handle *const h) ircd::db::database::cache::GetDeleter(Handle *const h)
const noexcept const noexcept
@ -3435,6 +3558,30 @@ noexcept
} }
#endif #endif
#ifdef IRCD_DB_HAS_CACHE_ITEMHELPER
const rocksdb::Cache::CacheItemHelper *
ircd::db::database::cache::GetCacheItemHelper(Handle *const h)
const noexcept
{
assert(bool(c));
return c->GetCacheItemHelper(h);
}
#endif
#if defined(IRCD_DB_HAS_CACHE_ASYNC)
rocksdb::Cache::Handle *
ircd::db::database::cache::CreateStandalone(const Slice &key,
ObjectPtr ptr,
const CacheItemHelper *const helper,
size_t charge,
bool allow_uncharged)
noexcept
{
assert(bool(c));
return c->CreateStandalone(key, ptr, helper, charge, allow_uncharged);
}
#endif
/////////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////////
// //
// database::compaction_filter // database::compaction_filter

View file

@ -3265,7 +3265,11 @@ catch(const std::exception &e)
return error_to_status{e}; return error_to_status{e};
} }
#ifdef IRCD_DB_HAS_ENV_MULTIREAD #if defined(IRCD_DB_HAS_ENV_MULTIREAD) && !defined(IRCD_DB_HAS_MULTIREAD_FIX)
#warning "RocksDB MultiRead is buggy in this version. Please upgrade to 6.12.6+"
#endif
#if defined(IRCD_DB_HAS_ENV_MULTIREAD)
rocksdb::Status rocksdb::Status
ircd::db::database::env::random_access_file::MultiRead(rocksdb::ReadRequest *const req, ircd::db::database::env::random_access_file::MultiRead(rocksdb::ReadRequest *const req,
size_t num) size_t num)

View file

@ -73,12 +73,30 @@
#define IRCD_DB_HAS_ENV_FILESYSTEM #define IRCD_DB_HAS_ENV_FILESYSTEM
#endif #endif
#if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 8) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 8 && ROCKSDB_PATCH >= 1)
#define IRCD_DB_HAS_SKIP_CHECKSIZE
#endif
#if ROCKSDB_MAJOR > 6 \ #if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 10) \ || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 10) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 10 && ROCKSDB_PATCH >= 0) || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 10 && ROCKSDB_PATCH >= 0)
#define IRCD_DB_HAS_MULTIGET_DIRECT #define IRCD_DB_HAS_MULTIGET_DIRECT
#endif #endif
#if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 10) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 10 && ROCKSDB_PATCH >= 1)
#define IRCD_DB_HAS_CONFIG_OPTIONS
#endif
#if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 12) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 12 && ROCKSDB_PATCH >= 6)
#define IRCD_DB_HAS_MULTIREAD_FIX
#endif
#if ROCKSDB_MAJOR > 6 \ #if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 12) \ || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 12) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 12 && ROCKSDB_PATCH >= 6) || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 12 && ROCKSDB_PATCH >= 6)
@ -92,9 +110,9 @@
#endif #endif
#if ROCKSDB_MAJOR > 6 \ #if ROCKSDB_MAJOR > 6 \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 16) \ || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR > 15) \
|| (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 16 && ROCKSDB_PATCH >= 3) || (ROCKSDB_MAJOR == 6 && ROCKSDB_MINOR == 15 && ROCKSDB_PATCH >= 2)
#define IRCD_DB_HAS_MANIFEST_WRITE_NOWAL #define IRCD_DB_HAS_MANIFEST_WALS
#endif #endif
#if ROCKSDB_MAJOR > 6 \ #if ROCKSDB_MAJOR > 6 \
@ -140,8 +158,44 @@
#define IRCD_DB_HAS_FORCED_BLOBGC #define IRCD_DB_HAS_FORCED_BLOBGC
#endif #endif
#if ROCKSDB_MAJOR > 7 \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 2) \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 2 && ROCKSDB_PATCH >= 0)
#define IRCD_DB_HAS_SECONDARY_CACHE
#endif
#if ROCKSDB_MAJOR > 7 \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 7) \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 7 && ROCKSDB_PATCH >= 2)
#define IRCD_DB_HAS_MANIFEST_UUIDS
#endif
#if ROCKSDB_MAJOR > 7 \ #if ROCKSDB_MAJOR > 7 \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 8) \ || (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 8) \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 8 && ROCKSDB_PATCH >= 3) || (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 8 && ROCKSDB_PATCH >= 3)
#define IRCD_DB_HAS_ROUND_ROBIN_TTL #define IRCD_DB_HAS_ROUND_ROBIN_TTL
#endif #endif
#if ROCKSDB_MAJOR > 7 \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 10) \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 10 && ROCKSDB_PATCH >= 0)
#define IRCD_DB_HAS_CACHE_ITEMHELPER
#endif
#if ROCKSDB_MAJOR > 7 \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR > 10) \
|| (ROCKSDB_MAJOR == 7 && ROCKSDB_MINOR == 10 && ROCKSDB_PATCH >= 2)
#define IRCD_DB_HAS_REFIT_LEVEL
#endif
#if ROCKSDB_MAJOR > 8 \
|| (ROCKSDB_MAJOR == 8 && ROCKSDB_MINOR > 0) \
|| (ROCKSDB_MAJOR == 8 && ROCKSDB_MINOR == 0 && ROCKSDB_PATCH >= 0)
#define IRCD_DB_HAS_CACHE_WRAPPER
#endif
#if ROCKSDB_MAJOR > 8 \
|| (ROCKSDB_MAJOR == 8 && ROCKSDB_MINOR > 1) \
|| (ROCKSDB_MAJOR == 8 && ROCKSDB_MINOR == 1 && ROCKSDB_PATCH >= 1)
#define IRCD_DB_HAS_CACHE_ASYNC
#endif

View file

@ -114,15 +114,13 @@ size_t
ircd::fs::aio::init::query_max_events() ircd::fs::aio::init::query_max_events()
{ {
size_t ret(0); size_t ret(0);
fs::dev::for_each("disk", [&ret] fs::dev::blk::for_each("disk", [&ret]
(const ulong &id, const fs::dev::blk &device) (const ulong &id, const fs::dev::blk &device)
{ {
ret = std::clamp ret = std::clamp
( (
device.queue_depth, ret, MAX_EVENTS device.queue_depth, ret, MAX_EVENTS
); );
return true;
}); });
if(!ret) if(!ret)
@ -691,7 +689,7 @@ ircd::fs::aio::system::handle_descriptor
// appears to excessively allocate and deallocate 120 bytes; this // appears to excessively allocate and deallocate 120 bytes; this
// is a simple asynchronous operation, we can do better (and perhaps // is a simple asynchronous operation, we can do better (and perhaps
// even better than this below). // even better than this below).
[](ios::handler &handler, const size_t &size) -> void * [](ios::handler &handler, const size_t size) -> void *
{ {
assert(ircd::fs::aio::system); assert(ircd::fs::aio::system);
auto &system(*ircd::fs::aio::system); auto &system(*ircd::fs::aio::system);
@ -707,7 +705,7 @@ ircd::fs::aio::system::handle_descriptor
}, },
// no deallocation; satisfied by class member unique_ptr // no deallocation; satisfied by class member unique_ptr
[](ios::handler &handler, void *const &ptr, const size_t &size) {}, [](ios::handler &handler, void *const ptr, const size_t size) {},
// continuation // continuation
true, true,

View file

@ -10,55 +10,6 @@
#include <RB_INC_SYS_SYSMACROS_H #include <RB_INC_SYS_SYSMACROS_H
bool
ircd::fs::dev::for_each(const blk_closure &closure)
{
return for_each(string_view{}, closure);
}
bool
ircd::fs::dev::for_each(const string_view &type,
const blk_closure &closure)
{
for(const auto &dir : fs::ls(blk::BASE_PATH)) try
{
const auto &[major, minor]
{
split(filename(path_scratch, dir), ':')
};
if(!major || !minor)
continue;
const ulong id
{
dev::id({lex_cast<ulong>(major), lex_cast<ulong>(minor)})
};
char dtbuf[32];
if(type && blk::devtype(dtbuf, id) != type)
continue;
if(!closure(id, blk(id)))
return false;
}
catch(const ctx::interrupted &)
{
throw;
}
catch(const std::exception &e)
{
log::error
{
log, "%s :%s",
dir,
e.what(),
};
}
return true;
}
ircd::string_view ircd::string_view
ircd::fs::dev::sysfs(const mutable_buffer &out, ircd::fs::dev::sysfs(const mutable_buffer &out,
const ulong &id, const ulong &id,
@ -123,6 +74,59 @@ ircd::fs::dev::blk::BASE_PATH
"/sys/dev/block" "/sys/dev/block"
}; };
bool
ircd::fs::dev::blk::for_each(const closure &closure)
{
return for_each(string_view{}, closure);
}
bool
ircd::fs::dev::blk::for_each(const string_view &type,
const closure &closure)
{
for(const auto &dir : fs::ls(blk::BASE_PATH)) try
{
const auto &[major, minor]
{
split(filename(path_scratch, dir), ':')
};
if(!major || !minor)
continue;
const ulong id
{
dev::id({lex_cast<ulong>(major), lex_cast<ulong>(minor)})
};
char dtbuf[32];
if(type && blk::devtype(dtbuf, id) != type)
continue;
if(!closure(id, blk(id)))
return false;
}
catch(const ctx::interrupted &)
{
throw;
}
catch(const std::exception &e)
{
log::error
{
log, "%s :%s",
dir,
e.what(),
};
}
return true;
}
//
// dev::blk::blk
//
ircd::fs::dev::blk::blk(const ulong &id) ircd::fs::dev::blk::blk(const ulong &id)
:type :type
{ {
@ -227,3 +231,100 @@ ircd::fs::dev::blk::devtype(const mutable_buffer &buf,
return ret; return ret;
} }
//
// dev::stats
//
ircd::fs::dev::stats
ircd::fs::dev::stats::get(const major_minor &id)
{
stats ret;
for_each([&id, &ret]
(const auto &stats)
{
if(stats.id == id)
{
ret = stats;
return false;
}
else return true;
});
return ret;
}
bool
ircd::fs::dev::stats::for_each(const closure &closure)
{
thread_local char buf[16_KiB];
const fs::fd fd
{
"/proc/diskstats", fs::fd::opts
{
.mode = std::ios::in,
},
};
fs::read_opts opts;
opts.aio = false;
const string_view read
{
fs::read(fd, buf, opts)
};
return tokens(read, '\n', [&closure]
(const auto &line)
{
return closure(stats(line));
});
}
//
// dev::stats::stats
//
ircd::fs::dev::stats::stats(const string_view &line)
{
string_view item[20];
const auto items
{
tokens(line, ' ', item)
};
id.first = lex_cast<uint64_t>(item[0]);
id.second = lex_cast<uint64_t>(item[1]);
strlcpy(name, item[2]);
read = lex_cast<uint64_t>(item[3]);
read_merged = lex_cast<uint64_t>(item[4]);
read_sectors = lex_cast<uint64_t>(item[5]);
read_time = lex_cast<milliseconds>(item[6]);
write = lex_cast<uint64_t>(item[7]);
write_merged = lex_cast<uint64_t>(item[8]);
write_sectors = lex_cast<uint64_t>(item[9]);
write_time = lex_cast<milliseconds>(item[10]);
io_current = lex_cast<uint64_t>(item[11]);
io_time = lex_cast<milliseconds>(item[12]);
io_weighted_time = lex_cast<milliseconds>(item[13]);
if(items <= 14)
return;
discard = lex_cast<uint64_t>(item[14]);
discard_merged = lex_cast<uint64_t>(item[15]);
discard_sectors = lex_cast<uint64_t>(item[16]);
discard_time = lex_cast<milliseconds>(item[17]);
if(items <= 18)
return;
flush = lex_cast<uint64_t>(item[18]);
flush_time = lex_cast<milliseconds>(item[19]);
if(items <= 20)
return;
}

View file

@ -151,7 +151,7 @@ namespace ircd::http::parser
const rule<string_view> reason { str ,"reason" }; const rule<string_view> reason { str ,"reason" };
const rule<string_view> head_key { raw[+(char_ - (illegal | ws | colon))] ,"head key" }; const rule<string_view> head_key { raw[+(char_ - (illegal | ws | colon))] ,"head key" };
const rule<string_view> head_val { str ,"head value" }; const rule<string_view> head_val { -str ,"head value" };
const rule<http::header> header { head_key >> *ws >> colon >> *ws >> head_val ,"header" }; const rule<http::header> header { head_key >> *ws >> colon >> *ws >> head_val ,"header" };
const rule<> headers { header % (*ws >> CRLF) ,"headers" }; const rule<> headers { header % (*ws >> CRLF) ,"headers" };

View file

@ -175,7 +175,7 @@ ircd::info::dump_cpu_info_x86()
const auto append{[&support] const auto append{[&support]
(const string_view &name, const bool &avail, const int &enable) (const string_view &name, const bool &avail, const int &enable)
{ {
strlcat(support, fmt::bsprintf<64> strlcat(support, fmt::bsprintf<32>
{ {
" %s:%c%s", " %s:%c%s",
name, name,
@ -193,7 +193,14 @@ ircd::info::dump_cpu_info_x86()
append("avx", hardware::x86::avx, simd::support::avx); append("avx", hardware::x86::avx, simd::support::avx);
append("avx2", hardware::x86::avx2, simd::support::avx2); append("avx2", hardware::x86::avx2, simd::support::avx2);
append("avx512f", hardware::x86::avx512f, simd::support::avx512f); append("avx512f", hardware::x86::avx512f, simd::support::avx512f);
append("constant_tsc", hardware::x86::tsc_constant, -1);
strlcat(support, fmt::bsprintf<32>
{
"%s%s%s",
hardware::x86::tsc? " tsc": "",
hardware::x86::tsc_constant? ":constant": "",
hardware::x86::tsc_nonstop? ":nonstop": "",
});
log::info log::info
{ {
@ -415,25 +422,31 @@ ircd::info::hardware::x86::avx
decltype(ircd::info::hardware::x86::avx2) decltype(ircd::info::hardware::x86::avx2)
ircd::info::hardware::x86::avx2 ircd::info::hardware::x86::avx2
{ {
bool(features & (uint128_t(1) << (32 + 5))) bool(extended_features & (uint128_t(1) << (32 + 5)))
}; };
decltype(ircd::info::hardware::x86::avx512f) decltype(ircd::info::hardware::x86::avx512f)
ircd::info::hardware::x86::avx512f ircd::info::hardware::x86::avx512f
{ {
bool(features & (uint128_t(1) << (32 + 16))) bool(extended_features & (uint128_t(1) << (32 + 16)))
}; };
decltype(ircd::info::hardware::x86::tsc) decltype(ircd::info::hardware::x86::tsc)
ircd::info::hardware::x86::tsc ircd::info::hardware::x86::tsc
{ {
bool(features & (uint128_t(1) << 4)) bool(features & (uint128_t(1) << (96 + 4)))
}; };
decltype(ircd::info::hardware::x86::tsc_constant) decltype(ircd::info::hardware::x86::tsc_constant)
ircd::info::hardware::x86::tsc_constant ircd::info::hardware::x86::tsc_constant
{ {
bool(_apmi & (uint128_t(1) << (8))) bool(_apmi & (uint128_t(1) << (96 + 8)))
};
decltype(ircd::info::hardware::x86::tsc_nonstop)
ircd::info::hardware::x86::tsc_nonstop
{
bool(_apmi & (uint128_t(1) << (96 + 24)))
}; };
#ifdef __x86_64__ #ifdef __x86_64__

View file

@ -39,6 +39,14 @@ ircd::ios::primary;
decltype(ircd::ios::main) decltype(ircd::ios::main)
ircd::ios::main; ircd::ios::main;
/// Indicates the user asio::executor is initialized.
decltype(ircd::ios::user_available)
ircd::ios::user_available;
/// Indicates the main asio::executor is initialized.
decltype(ircd::ios::main_available)
ircd::ios::main_available;
decltype(ircd::boost_version_api) decltype(ircd::boost_version_api)
ircd::boost_version_api ircd::boost_version_api
{ {
@ -71,12 +79,14 @@ ircd::ios::init(asio::executor &&user)
// Save a reference handle to the user's executor. // Save a reference handle to the user's executor.
ios::user = user; ios::user = user;
ios::user_available = bool(ios::user);
// Create our strand instance. // Create our strand instance.
ios::primary.emplace(static_cast<asio::io_context &>(user.context())); ios::primary.emplace(static_cast<asio::io_context &>(user.context()));
// Set the reference handle to our executor. // Set the reference handle to our executor.
ios::main = *ios::primary; ios::main = *ios::primary;
ios::main_available = bool(ios::main);
} }
[[using gnu: cold]] [[using gnu: cold]]
@ -180,13 +190,6 @@ ircd::ios::forked_parent()
#endif #endif
} }
bool
ircd::ios::available()
noexcept
{
return bool(main);
}
// //
// emption // emption
// //
@ -480,13 +483,6 @@ ircd::ios::descriptor::stats::stats(descriptor &d)
{ "name", stats_name(d, "free_bytes") }, { "name", stats_name(d, "free_bytes") },
}, },
} }
,slice_total
{
value + items++,
{
{ "name", stats_name(d, "slice_total") },
},
}
,slice_last ,slice_last
{ {
value + items++, value + items++,
@ -494,11 +490,11 @@ ircd::ios::descriptor::stats::stats(descriptor &d)
{ "name", stats_name(d, "slice_last") }, { "name", stats_name(d, "slice_last") },
}, },
} }
,latency_total ,slice_total
{ {
value + items++, value + items++,
{ {
{ "name", stats_name(d, "latency_total") }, { "name", stats_name(d, "slice_total") },
}, },
} }
,latency_last ,latency_last
@ -508,6 +504,13 @@ ircd::ios::descriptor::stats::stats(descriptor &d)
{ "name", stats_name(d, "latency_last") }, { "name", stats_name(d, "latency_last") },
}, },
} }
,latency_total
{
value + items++,
{
{ "name", stats_name(d, "latency_total") },
},
}
{ {
assert(items <= (sizeof(value) / sizeof(value[0]))); assert(items <= (sizeof(value) / sizeof(value[0])));
} }
@ -531,7 +534,7 @@ ircd::ios::handler::epoch;
[[gnu::cold]] [[gnu::cold]]
bool bool
ircd::ios::handler::fault(handler *const &handler) ircd::ios::handler::fault(handler *const handler)
noexcept noexcept
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
@ -568,7 +571,7 @@ noexcept
[[gnu::hot]] [[gnu::hot]]
void void
ircd::ios::handler::leave(handler *const &handler) ircd::ios::handler::leave(handler *const handler)
noexcept noexcept
{ {
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
@ -619,19 +622,14 @@ noexcept
[[gnu::hot]] [[gnu::hot]]
void void
ircd::ios::handler::enter(handler *const &handler) ircd::ios::handler::enter(handler *const handler)
noexcept noexcept
{ {
assert(!handler::current);
handler::current = handler;
++handler::epoch;
assert(handler && handler->descriptor); assert(handler && handler->descriptor);
auto &descriptor(*handler->descriptor); auto &descriptor(*handler->descriptor);
assert(descriptor.stats); assert(descriptor.stats);
auto &stats(*descriptor.stats); auto &stats(*descriptor.stats);
++stats.calls;
const auto last_ts const auto last_ts
{ {
@ -640,6 +638,11 @@ noexcept
stats.latency_last = handler->ts - last_ts; stats.latency_last = handler->ts - last_ts;
stats.latency_total += stats.latency_last; stats.latency_total += stats.latency_last;
++stats.calls;
assert(!handler::current);
handler::current = handler;
++handler::epoch;
if constexpr(profile::logging) if constexpr(profile::logging)
log::logf log::logf
@ -696,7 +699,6 @@ ircd::ios::dispatch::dispatch(descriptor &descriptor,
latch.wait(); latch.wait();
} }
[[gnu::hot]]
ircd::ios::dispatch::dispatch(descriptor &descriptor, ircd::ios::dispatch::dispatch(descriptor &descriptor,
defer_t, defer_t,
std::function<void ()> function) std::function<void ()> function)
@ -725,7 +727,6 @@ ircd::ios::dispatch::dispatch(descriptor &descriptor,
}; };
} }
[[gnu::hot]]
ircd::ios::dispatch::dispatch(descriptor &descriptor, ircd::ios::dispatch::dispatch(descriptor &descriptor,
std::function<void ()> function) std::function<void ()> function)
{ {

View file

@ -107,25 +107,21 @@ ircd::maintenance
} }
}; };
/// Coarse mode indicator for degraded operation known as "write-avoid" which /// Conf item dictates whether databases will be opened in slave mode; this
/// is similar to read_only but not hard-enforced. Writes may still occur, /// is a recent feature of RocksDB which may not be available. It allows
/// such as those manually triggered by an admin. All subsystems and background /// multiple processes to open the same database in a single-writer multiple
/// tasks otherwise depart from normal operation to avoid writes. /// reader configuration.
decltype(ircd::write_avoid) ///
ircd::write_avoid /// Originally this was intended as a maintenance mode to explore another live
/// running server's database. However it now allows listeners which can serve
/// as load-balanced mirrors and caches over a database shared at the fs level
/// locally or with nfs/ceph/intermezzo etc.
decltype(ircd::slave)
ircd::slave
{ {
{ { "name", "ircd.slave" },
{ "name", "ircd.write_avoid" }, { "default", false },
{ "default", false }, { "persist", false },
{ "persist", false },
},
[](conf::item<void> &)
{
if(!write_avoid)
return;
maintenance.set("true");
}
}; };
/// Coarse mode declaration for read-only behavior. All subsystems and feature /// Coarse mode declaration for read-only behavior. All subsystems and feature
@ -145,7 +141,11 @@ ircd::read_only
if(!read_only) if(!read_only)
return; return;
write_avoid.set("true"); // Not implict maintenance mode when slave mode is set.
if(slave)
return;
maintenance.set("true");
} }
}; };

Some files were not shown because too many files have changed in this diff Show more