Commit graph

58 commits

Author SHA1 Message Date
AnonymousRandomPerson
6340f95060 Closed files properly in redditcrawl 2022-06-11 12:39:10 -04:00
AnonymousRandomPerson
d441f34501 Run formatter in merge_out 2022-05-21 17:06:33 -04:00
AnonymousRandomPerson
9437f4cc7c Integrate whiteout scaling with merging 2022-05-21 16:57:19 -04:00
Hans5958
6e19e29983 Only extend on crawls 2022-05-21 08:14:58 +07:00
Cheng Hann Gan
d92eb66102 Revert "Revert "Removed unused contributors field from JSON""
This reverts commit 6a2b409fe5.

We need a proper discussion. Until then, this commit will stay.
2022-05-13 13:19:43 -04:00
Hans5958
6a2b409fe5 Revert "Removed unused contributors field from JSON"
This reverts commit 5deedbdea0.

ART, I'm not sure what is your intention, but it seems you are quite persistent. Please don't be distruptive.
2022-05-13 23:58:24 +07:00
Cheng Hann Gan
d90baab00a Revert "Revert "Removed unused contributors field from JSON""
This reverts commit 7f735f9abc.
2022-05-13 10:16:50 -04:00
Hans5958
7f735f9abc Revert "Removed unused contributors field from JSON"
This reverts commit 5deedbdea0.
2022-05-13 21:06:50 +07:00
Cheng Hann Gan
8921fdd0ad Fixed comma formatting in crawler 2022-05-12 12:54:27 -04:00
Cheng Hann Gan
5deedbdea0 Removed unused contributors field from JSON 2022-05-12 12:45:56 -04:00
Hans5958
710d4a7c6a Little rewrite on migrator, migrate submissions if needed, add writing log 2022-04-25 15:50:26 +07:00
Hans5958
dc5eda7ed6 Add missing spaces 2022-04-19 10:47:11 +07:00
Hans5958
d0d8add004 Use TXT instead for JSON for manual_atlas 2022-04-19 09:00:26 +07:00
Hans5958
37bfe79d3e Fix crawler and merger script, fix typo of formatter 2022-04-19 08:55:37 +07:00
Hans5958
0e5df81ea4 Edit instructions, move to top 2022-04-18 21:25:36 +07:00
Hans5958
3be58f2a02 Migrate contributors, move contributors to the end, simplify script
Forgot that there won't be submitted_by on the submissions
2022-04-18 21:21:02 +07:00
Hans5958
41ab1f0d20 Use different flair for edit 2022-04-16 22:25:11 +07:00
Hans5958
6faec6f11d Store and check read edit entry ids 2022-04-16 22:25:11 +07:00
Hans5958
1e1c007d31 Support editing on script, submitted_by to contributors, merge to atlas script 2022-04-16 22:25:10 +07:00
Hans5958
547ffc773e List ids that already been read in a seperate file 2022-04-10 21:34:06 +07:00
Hans5958
31101f8211 Forgot to avoid IndexError 2022-04-09 16:33:09 +07:00
Hans5958
a996899814 Add ensure_ascii=False 2022-04-09 16:33:09 +07:00
Hans5958
9053d6d106 Add docs on formatter, move path length checker to formatter 2022-04-09 16:33:09 +07:00
Hans5958
cc497d5178 Add forgotten parts on the port, move some parts for better git diff 2022-04-08 11:21:16 +07:00
Hans5958
121d8653a5 Update auth setup docs with more clarity
Also adapted from Nick

Co-authored-by: Nicolas Abram <abramlujan@gmail.com>
2022-04-08 11:09:35 +07:00
Hans5958
ad15cefb07 Remove zero width joiner before parsing 2022-04-08 11:03:18 +07:00
Hans5958
f04102f29c Port flair edit, small log change
Logic and port made/adapted from Nick's

Co-authored-by: Nicolas Abram <abramlujan@gmail.com>
2022-04-08 11:03:18 +07:00
Hans5958
4c4fb973f3 Fix trailing comma removal 2022-04-08 11:03:18 +07:00
Hans5958
33bda5f364 Expand error info 2022-04-08 11:03:17 +07:00
Hans5958
8913e781bf Forgot to silent on formatting 2022-04-08 11:03:17 +07:00
Hans5958
cd87520c87 Handle escaping escape characters 2022-04-08 11:03:17 +07:00
Hans5958
1da79b806b Take only the first object, remove escapes, add more info on errors 2022-04-08 11:03:17 +07:00
Hans5958
7571a92fd9 Simplify try-catch, assert that path length > 0, separator on fail file
Assertion migrated from ARP

Co-authored-by: Cheng Hann Gan <chenghanngan.us@gmail.com>
2022-04-08 11:03:14 +07:00
Hans5958
3a86822175 Happy little accidents 2022-04-08 11:03:12 +07:00
Hans5958
5a660759bf Improve and merge scripts, use JSON instead of regex 2022-04-08 11:03:12 +07:00
unknown
93e80dc7f9 New submissions, flair editing script 2022-04-07 17:00:30 -03:00
unknown
c7f4b927f5 Fix missing id in entry 2022-04-06 17:38:25 -03:00
unknown
7ad57eaef9 Make redditcrawl create a valid json for atlas_temp 2022-04-06 16:47:46 -03:00
unknown
f03a825c5d Fix crawler 2022-04-06 04:48:39 -03:00
unknown
443c210299 Fix Success counter 2022-04-05 22:33:57 -03:00
unknown
35672fdb47 Fix reddit crawl newline and quote handling 2022-04-05 19:00:31 -03:00
ash
2848f93f47
Merge branch 'master' into master 2022-04-05 22:38:13 +01:00
Nicolas Abram
2f45690040
Fix reddit crawling file encoding 2022-04-05 18:19:49 -03:00
ash
1eb81da6de
reddit contributions 2022-04-05 20:57:08 +01:00
ash
6206596da1
fix and update crawler 2022-04-05 13:47:36 +01:00
Aeywoo
5560a2f788
Merge branch 'master' into redditcrawl-fix 2022-04-05 20:48:48 +09:30
Someone Somewhere
9e77c4231c Fix encoding issue 2022-04-05 14:15:30 +03:00
ash
1933a5aebd
latest reddit updates and dupe prevention logic 2022-04-05 11:55:45 +01:00
ash
3e2f998116
update crawler to reject already submitted 2022-04-05 11:25:12 +01:00
ash
4d6fd79368
new submissions, and move to reddit-based ID 2022-04-05 08:39:51 +01:00