diff --git a/.github/workflows/validate-json.yml b/.github/workflows/validate-json.yml index b92b01fa..fb903685 100644 --- a/.github/workflows/validate-json.yml +++ b/.github/workflows/validate-json.yml @@ -1,4 +1,4 @@ -name: Validate JSON +name: Validate Atlas data on: push: paths: @@ -8,10 +8,20 @@ on: - web/atlas.json jobs: validate: - name: Validate JSON + name: Validate runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v3 - - name: Validate JSON - run: python3 tools/ci/validate_json.py web/atlas.json \ No newline at end of file + - name: Cache dependencies + uses: actions/cache@v3 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} + restore-keys: | + ${{ runner.os }}-pip- + - name: Validate + run: | + pip3 install -r tools/ci/requirements.txt + python3 tools/ci/validate_json.py web/atlas.json tools/schema/atlas.json + python3 tools/ci/validate_json.py data/patches tools/schema/patch.json \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6d41364b..ce433b55 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,8 +6,6 @@ You may contribute to the project by submitting a Pull Request on the GitHub rep ## New Atlas entries -> **Warning**: **WE ONLY ACCEPT NEW ENTRIES ON REDDIT!** - To contribute to the map, we require a certain format for artwork region and labels. This can be generated on [the drawing mode](https://place-atlas.stefanocoding.me?mode=draw) on the website. To add a new entry, go to [the drawing mode](https://place-atlas.stefanocoding.me?mode=draw) and draw a shape/polygon around the region you'd like to describe. You can use the Undo, Redo, and Reset buttons to help you creating a good polygon. Make sure that the lines you're drawing don't form a [self-intersecting polygon](https://upload.wikimedia.org/wikipedia/commons/thumb/0/0f/Complex_polygon.svg/288px-Complex_polygon.svg.png). @@ -26,34 +24,59 @@ When you're happy with the shape you've drawn, press Finish. You will All fields but the name are optional. For example, a country flag doesn't necessarily need a description. -Once you've entered all the information, you'll be presented with a pop-up window containing some [JSON](https://en.wikipedia.org/wiki/JSON)-formatted data. You can press the Post Direct to Reddit button and just press the send button on Reddit, or copy the entire JSON text and [create a new text post on the subreddit](https://www.reddit.com/r/placeAtlas2/submit). You don't need to add any other text; just directly send the data. +Once you've entered all the information, you'll be presented with a pop-up window containing some [JSON](https://en.wikipedia.org/wiki/JSON)-formatted submission data. Depending on the method, there are two preferred methods. + +### Through Reddit + +You can press the Post Direct to Reddit button and just press the send button on Reddit, or copy the entire JSON text and [create a new text post on the subreddit](https://www.reddit.com/r/placeAtlas2/submit). You don't need to add any other text; just directly send the data. Remember to flair your post with New Entry. On New Reddit, click the Flair button on the bottom part, and select New Entry. On Old Reddit, click the select button on the "choose a flair" section instead. +### Through GitHub + +If you know about Git and how to create a pull request on GitHub, you can try create a patch that will be merged, along with other patches, by one of the members. + +You can use the provided `tools/create_patch.py` script. This script helps you to create a working patch, along with additional data such as your name for attribution sakes. Simply run the script inside the `tools/` folder and follow the given instructions. + +If you want to do this manually (e.g. you don't have Python), you can create a patch by creating a `.json` file inside `data/patches`, with the content of the JSON-formatted data that is given earlier. You may add attribution by adding a `_author` key with the value of your Reddit username or your GitHub username plus a `gh:` prefix. + +```json5 +{ + "id": 0, + // ... + "_author": "Hans5958_", + // or... + "_author": "gh:Hans5958", +} +``` + +Once you have successfully created the patch, the file can be committed, and a pull request towards the `cleanup` branch can be created. A member will merge the pull request if it is adequate. + ## Edits to Atlas entries Other than adding new ones, you can edit existing atlas entries. ### Using the web interface -You can use the website to edit single entries easily. On the website, click Edit on an entry box. Afterwards, you are now on the drawing mode, editing the entry, in which you can follow the same instructions as [when creating a new entry](#new-atlas-entries). Upon submitting, please flair it as Edit Entry instead. +You can use the website to edit single entries easily. On the website, click Edit on an entry box. Afterwards, you are now on the drawing mode, editing the entry, in which you can follow the same instructions as [when creating a new entry](#new-atlas-entries). -As an alternative, you can also submit an issue on GitHub using [this form](https://github.com/placeAtlas/atlas/issues/new?assignees=&labels=entry+update&template=edit-entry.yml). +Upon submitting, if you use Reddit, please flair it as Edit Entry instead. The method stays the same if you use GitHub. + +As an alternative, you can also submit an issue on GitHub using [this form](https://github.com/placeAtlas/atlas/issues/new?assignees=&labels=entry+update&template=edit-entry.yml) or report it on our Discord server. ### Manually -Edits are also welcome on this repository through GitHub. You may use GitHub for bulk or large-scale changes, such as removing duplicates. +Edits are also welcome on this repository using Git through GitHub. You may use Git or GitHub for bulk or large-scale changes, such as removing duplicates. -`web/atlas.json` is where the Atlas data is located, in which you can edit on GitHub. Below is an example of an entry. The example has been expanded, but please save it in the way so each line is an entry which is minified. +`web/atlas.json` is where the Atlas data is located, in which you can edit on GitHub. The next section includes an example of an entry. -Upon creating a fork of this repository and pushing the changes, create a Pull Request against the `cleanup` branch. A member will merge the pull request if it is adequate. +Upon creating a fork of this repository and pushing the changes, create a pull request towards the `cleanup` branch. A member will merge the pull request if it is adequate. To help find duplicates, [use the Overlap mode](https://place-atlas.stefanocoding.me?mode=overlap). - ### Example -Hereforth is an example of the structured data. +Hereforth is an example of the structured data. The example has been expanded, but please save it in the way so each line is an entry which is minified. The `aformatter.py` script can help you with this. ```json5 { diff --git a/tools/aformatter.py b/tools/aformatter.py index 365a68c0..01913c9b 100644 --- a/tools/aformatter.py +++ b/tools/aformatter.py @@ -1,10 +1,12 @@ #!/usr/bin/python +from io import TextIOWrapper +from typing import List import re import json import math import traceback -from typing import List +import tqdm END_NORMAL_IMAGE = "164" END_WHITEOUT_IMAGE = "166" @@ -302,7 +304,6 @@ def floor_points(entry: dict): return entry - def validate(entry: dict): """ Validates the entry. Catch errors and tell warnings related to the entry. @@ -339,16 +340,17 @@ def validate(entry: dict): print(f"{key} of entry {entry['id']} is still invalid! {entry[key]}") return return_status -def per_line_entries(entries: list): +def per_line_entries(entries: list, file: TextIOWrapper): """ Returns a string of all the entries, with every entry in one line. """ - out = "[\n" - for entry in entries: - if entry: - out += json.dumps(entry, ensure_ascii=False) + ",\n" - out = out[:-2] + "\n]" - return out + file.write("[\n") + line_temp = "" + for entry in tqdm.tqdm(entries): + if line_temp: + file.write(line_temp + ",\n") + line_temp = json.dumps(entry, ensure_ascii=False) + file.write(line_temp + "\n]") def format_all(entry: dict, silent=False): """ @@ -387,7 +389,7 @@ def print_(*args, **kwargs): return entry def format_all_entries(entries): - for i in range(len(entries)): + for i in tqdm.trange(len(entries)): try: entry_formatted = format_all(entries[i], True) validation_status = validate(entries[i]) @@ -399,8 +401,6 @@ def format_all_entries(entries): except Exception: print(f"Exception occured when formatting ID {entries[i]['id']}") print(traceback.format_exc()) - if not (i % 200): - print(f"{i} checked.") def go(path): @@ -411,10 +411,10 @@ def go(path): format_all_entries(entries) - print(f"{len(entries)} checked. Writing...") + print(f"Writing...") with open(path, "w", encoding='utf-8', newline='\n') as f2: - f2.write(per_line_entries(entries)) + per_line_entries(entries, f2) print("Writing completed. All done.") diff --git a/tools/ci/build-prod.sh b/tools/ci/build-prod.sh index d73569bc..ebe64fb2 100644 --- a/tools/ci/build-prod.sh +++ b/tools/ci/build-prod.sh @@ -8,7 +8,7 @@ rm -rf .parcel-cache cp -r web/ dist-temp/ npm i -python tools/ci/cdn-to-local.py +python tools/ci/cdn_to_local.py npx parcel build dist-temp/index.html dist-temp/**.html --dist-dir "dist" --no-source-maps --no-content-hash rm -rf dist-temp diff --git a/tools/ci/cdn-to-local.py b/tools/ci/cdn_to_local.py similarity index 100% rename from tools/ci/cdn-to-local.py rename to tools/ci/cdn_to_local.py diff --git a/tools/ci/requirements.txt b/tools/ci/requirements.txt new file mode 100644 index 00000000..7b8f0158 --- /dev/null +++ b/tools/ci/requirements.txt @@ -0,0 +1 @@ +jsonschema \ No newline at end of file diff --git a/tools/ci/validate_json.py b/tools/ci/validate_json.py index 9f2bcfdf..cf698286 100644 --- a/tools/ci/validate_json.py +++ b/tools/ci/validate_json.py @@ -2,13 +2,38 @@ import sys import json +from jsonschema import validate, RefResolver +from pathlib import Path, PurePosixPath +import os -path = "./../../web/atlas.json" +instance_path = "../../web/atlas.json" # path override as 1st param: validate_json.py path_to_file.json if (len(sys.argv) > 1): - path = sys.argv[1] + instance_path = sys.argv[1] -json.load(open(path, "r", encoding='utf-8')) +schema_path = "../schema/atlas.json" + +# schema override as 2nd param: validate_json.py [...] path_to_schema.json +if (len(sys.argv) > 2): + schema_path = sys.argv[2] + +relative_path = "file:" + str(PurePosixPath(Path(os.getcwd(), schema_path))) + +schema = json.load(open(schema_path, "r", encoding='utf-8')) +# exit() + +resolver = RefResolver(relative_path, schema) +if os.path.isdir(instance_path): + for filename in os.listdir(instance_path): + f = os.path.join(instance_path, filename) + print(f) + + instance = json.load(open(f, "r", encoding='utf-8')) + validate(instance, schema, resolver=resolver) +elif os.path.isfile(instance_path): + print(instance_path) + instance = json.load(open(instance_path, "r", encoding='utf-8')) + validate(instance, schema, resolver=resolver) print("JSON is valid") \ No newline at end of file diff --git a/tools/create_patch.py b/tools/create_patch.py new file mode 100644 index 00000000..9e283ef3 --- /dev/null +++ b/tools/create_patch.py @@ -0,0 +1,37 @@ +import json +import os +import secrets +from pathlib import Path + +patches_dir = "../data/patches/" +Path(patches_dir).mkdir(parents=True, exist_ok=True) + +entry = None +entry_input = "" + +print("Write/paste your JSON-formatted submission data here.") +while entry is None: + + entry_input += input("> ") + try: + entry = json.loads(entry_input) + except: + pass +print() +print("Submission is valid!") +print() +print("Enter your username as the attribution to be shown on the About page.") +print("Leave it empty if you don't want to be attributed.") +print("You can use your Reddit username. Do not include the \"u/\" part.") +print("You can also your GitHub username, but add \"gh:\" before your username (e.g. \"gh:octocat\")") +author = input("Author: ") + +if author: + entry['_author'] = author + +filename = f'gh-{secrets.token_hex(2)}-{"-".join(entry["name"].split()).lower()}.json' +with open(f'{patches_dir}gh-{secrets.token_hex(2)}-{"-".join(entry["name"].split()).lower()}.json', 'w', encoding='utf-8') as out_file: + out_file.write(json.dumps(entry, ensure_ascii=False)) + +print("Patch created as " + filename + "!") +print("You can commit the created file directly, to which you can push and create a pull request after that.") \ No newline at end of file diff --git a/tools/merge_out.py b/tools/merge_out.py index 63181220..2371f10a 100644 --- a/tools/merge_out.py +++ b/tools/merge_out.py @@ -2,90 +2,108 @@ import os from aformatter import format_all_entries, per_line_entries import scale_back +import traceback from scale_back import ScaleConfig -merge_source_file = 'temp-atlas.json' - -with open(merge_source_file, 'r', encoding='UTF-8') as f1: - out_json = json.loads(f1.read()) - -format_all_entries(out_json) - -base_image_path = os.path.join('..', 'web', '_img', 'canvas', 'place30') -ScaleConfig.image1 = os.path.join(base_image_path, '159.png') -scale_back.swap_source_dest('164', '165', os.path.join(base_image_path, '163_159.png')) -scale_back.scale_back_entries(out_json) -scale_back.swap_source_dest('165', '166', os.path.join(base_image_path, '164_159.png')) -scale_back.scale_back_entries(out_json) -scale_back.swap_source_dest('166', '167', os.path.join(base_image_path, '165_159.png')) -scale_back.scale_back_entries(out_json) - -out_ids = set() -out_dupe_ids = set() +out_ids = [] atlas_ids = {} +authors = [] + +with open('../web/all-authors.txt', 'r', encoding='utf-8') as authors_file: + authors = authors_file.read().strip().split() + +with open('../data/read-ids.txt', 'r', encoding='utf-8') as ids_file: + out_ids = ids_file.read().strip().split() with open('../web/atlas.json', 'r', encoding='utf-8') as atlas_file: - atlas_json = json.loads(atlas_file.read()) + atlas_data = json.loads(atlas_file.read()) -for i, entry in enumerate(atlas_json): +# format_all_entries(atlas_data) + +# base_image_path = os.path.join('..', 'web', '_img', 'canvas', 'place30') +# ScaleConfig.image1 = os.path.join(base_image_path, '159.png') +# scale_back.swap_source_dest('164', '165', os.path.join(base_image_path, '163_159.png')) +# scale_back.scale_back_entries(atlas_data) +# scale_back.swap_source_dest('165', '166', os.path.join(base_image_path, '164_159.png')) +# scale_back.scale_back_entries(atlas_data) +# scale_back.swap_source_dest('166', '167', os.path.join(base_image_path, '165_159.png')) +# scale_back.scale_back_entries(atlas_data) + +last_id = 0 + +for i, entry in enumerate(atlas_data): atlas_ids[entry['id']] = i + id = entry['id'] + if id.isnumeric() and int(id) > last_id and int(id) - last_id < 100: + last_id = int(id) -last_existing_id = list(atlas_json[-1]['id']) +patches_dir = "../data/patches/" +if not os.path.exists(patches_dir): + print("Patches folder not found. Exiting.") + exit() + +for filename in os.listdir(patches_dir): + f = os.path.join(patches_dir, filename) + + print(f"{filename}: Processing...") -for entry in out_json: - if entry['id'] == 0 or entry['id'] == '0': - # "Increment" the last ID to derive a new ID. - current_index = -1 - while current_index > -(len(last_existing_id)): - current_char = last_existing_id[current_index] - - if current_char == 'z': - last_existing_id[current_index] = '0' - current_index -= 1 - else: - if current_char == '9': - current_char = 'a' - else: - current_char = chr(ord(current_char) + 1) - last_existing_id[current_index] = current_char - break - entry['id'] = ''.join(last_existing_id) - -for entry in out_json: - if entry['id'] in out_ids: - print(f"Entry {entry['id']} has duplicates! Please resolve this conflict. This will be excluded from the merge.") - out_dupe_ids.add(entry['id']) - out_ids.add(entry['id']) - -for entry in out_json: - if entry['id'] in out_dupe_ids: + if not os.path.isfile(f) or not f.endswith('json'): continue - if 'edit' in entry and entry['edit']: - assert entry['id'] in atlas_ids, "Edit failed! ID not found on Atlas." - index = atlas_ids[entry['id']] + try: + with open(f, 'r', encoding='utf-8') as entry_file: + entry = json.loads(entry_file.read()) - assert index != None, "Edit failed! ID not found on Atlas." + if '_reddit_id' in entry: + reddit_id = entry['_reddit_id'] + if reddit_id in out_ids: + print(f"{filename}: Submission from {entry['id']} has been included! This will be ignored from the merge.") + continue + out_ids.append(reddit_id) + del entry['_reddit_id'] - print(f"Edited {atlas_json[index]['id']} with {entry['edit']}") + # This wouldn't work if it is an edit. + # If needed, we can add a type to the patch to be more foolproof. + # if entry['id'] in out_ids: + # print(f"{filename}: Submission from {entry['id']} has been included! This will be ignored from the merge.") + # continue - del entry['edit'] - atlas_json[index] = entry - elif entry['id'] in atlas_ids: - print(f"Edited {entry['id']} manually.") - atlas_json[atlas_ids[entry['id']]] = entry - else: - print(f"Added {entry['id']}.") - atlas_json.append(entry) + if '_author' in entry: + author = entry['_author'] + if author not in authors: + authors.append(author) + del entry['_author'] + + if entry['id'] is int and entry['id'] < 1: + last_id += 1 + print(f"{filename}: Entry is new, assigned ID {last_id}") + entry['id'] = str(last_id) + elif entry['id'] not in out_ids: + out_ids.append(entry['id']) + + if entry['id'] in atlas_ids: + index = atlas_ids[entry['id']] + print(f"{filename}: Edited {atlas_data[index]['id']}.") + atlas_data[index] = entry + else: + print(f"{filename}: Added {entry['id']}.") + atlas_data.append(entry) + + os.remove(f) + + except: + print(f"{filename}: Something went wrong; patch couldn't be implemented. Skipping.") + traceback.print_exc() print('Writing...') with open('../web/atlas.json', 'w', encoding='utf-8') as atlas_file: - atlas_file.write(per_line_entries(atlas_json)) + per_line_entries(atlas_data, atlas_file) -with open('../data/read-ids.txt', 'a', encoding='utf-8') as read_ids_file: - with open('temp-read-ids.txt', 'r+', encoding='utf-8') as read_ids_temp_file: - read_ids_file.writelines(read_ids_temp_file.readlines()) - read_ids_temp_file.truncate(0) +with open('../data/read-ids.txt', 'w', encoding='utf-8') as ids_file: + ids_file.write("\n".join(out_ids) + "\n") + +with open('../web/all-authors.txt', 'w', encoding='utf-8') as authors_file: + authors_file.write("\n".join(authors) + "\n") print('All done.') \ No newline at end of file diff --git a/tools/oneoff/all-authors.py b/tools/oneoff/all_authors.py similarity index 100% rename from tools/oneoff/all-authors.py rename to tools/oneoff/all_authors.py diff --git a/tools/oneoff/migrate_atlas_format.py b/tools/oneoff/migrate_atlas_format.py index 7ec4e157..b507c39c 100644 --- a/tools/oneoff/migrate_atlas_format.py +++ b/tools/oneoff/migrate_atlas_format.py @@ -7,8 +7,10 @@ - submitted_by removed """ +from io import TextIOWrapper import re import json +import tqdm END_IMAGE = 166 INIT_CANVAS_RANGE = (1, END_IMAGE) @@ -73,16 +75,17 @@ def migrate_atlas_format(entry: dict): return toreturn -def per_line_entries(entries: list): +def per_line_entries(entries: list, file: TextIOWrapper): """ Returns a string of all the entries, with every entry in one line. """ - out = "[\n" - for entry in entries: - if entry: - out += json.dumps(entry, ensure_ascii=False) + ",\n" - out = out[:-2] + "\n]" - return out + file.write("[\n") + line_temp = "" + for entry in tqdm.tqdm(entries): + if line_temp: + file.write(line_temp + ",\n") + line_temp = json.dumps(entry, ensure_ascii=False) + file.write(line_temp + "\n]") if __name__ == '__main__': @@ -93,16 +96,14 @@ def go(path): with open(path, "r+", encoding='UTF-8') as f1: entries = json.loads(f1.read()) - for i in range(len(entries)): + for i in tqdm.trange(len(entries)): entry_formatted = migrate_atlas_format(entries[i]) entries[i] = entry_formatted - if not (i % 1000): - print(f"{i} checked.") print(f"{len(entries)} checked. Writing...") with open(path, "w", encoding='utf-8', newline='\n') as f2: - f2.write(per_line_entries(entries)) + per_line_entries(entries, f2) print("Writing completed. All done.") diff --git a/tools/redditcrawl.py b/tools/redditcrawl.py index 9f5a4103..9ad68aad 100755 --- a/tools/redditcrawl.py +++ b/tools/redditcrawl.py @@ -17,74 +17,90 @@ 1. Run the script 2. Input the next ID to use 3. Manually resolve errors in temp-atlas-manual.json -4 a. Use merge_out.py, or... - b. a. Copy temp-atlas.json entries into web/_js/atlas.js (mind the edits!) - b. Copy temp-read-ids.txt IDs into data/read-ids.txt +4. a. Use merge_out.py, or... + b. a. Copy temp-atlas.json entries into web/_js/atlas.js (mind the edits!) + b. Copy temp-read-ids.txt IDs into data/read-ids.txt 5. Create a pull request """ -import praw +from praw import Reddit +from praw.models import Submission import json import time import re import traceback from aformatter import format_all, validate +from pathlib import Path +import humanize +from datetime import datetime +import secrets -with open('temp-atlas.json', 'w', encoding='utf-8') as OUT_FILE, open('temp-read-ids.txt', 'w') as READ_IDS_FILE, open('temp-atlas-manual.txt', 'w', encoding='utf-8') as FAIL_FILE: +patches_dir = "../data/patches/" +Path(patches_dir).mkdir(parents=True, exist_ok=True) - OUT_FILE_LINES = ['[\n', ']\n'] +def set_flair(submission, flair): + if has_write_access and submission.link_flair_text != flair: + flair_choices = submission.flair.choices() + flair = next(x for x in flair_choices if x["flair_text_editable"] and flair == x["flair_text"]) + submission.flair.select(flair["flair_template_id"]) - with open('credentials', 'r') as file: - credentials = file.readlines() - client_id = credentials[0].strip() - client_secret = credentials[1].strip() - username = credentials[2].strip() if len(credentials) > 3 else "" - password = credentials[3].strip() if len(credentials) > 3 else "" - reddit = praw.Reddit( - client_id=client_id, - client_secret=client_secret, - username=username, - password=password, - user_agent='atlas_bot' - ) +with open('credentials', 'r') as file: + credentials = file.readlines() + client_id = credentials[0].strip() + client_secret = credentials[1].strip() + username = credentials[2].strip() if len(credentials) > 3 else "" + password = credentials[3].strip() if len(credentials) > 3 else "" - has_write_access = not reddit.read_only - if not has_write_access: - print("Warning: No write access. Post flairs will not be updated.") - time.sleep(5) +reddit = Reddit( + client_id=client_id, + client_secret=client_secret, + username=username, + password=password, + user_agent='atlas_bot' +) - existing_ids = [] +has_write_access = not reddit.read_only +if not has_write_access: + print("Warning: No write access. Post flairs will not be updated. Waiting 5 seconds...") + # time.sleep(5) - with open('../data/read-ids.txt', 'r') as edit_ids_file: - for id in [x.strip() for x in edit_ids_file.readlines()]: - existing_ids.append(id) +print("Running...") - def set_flair(submission, flair): - if has_write_access and submission.link_flair_text != flair: - flair_choices = submission.flair.choices() - flair = next(x for x in flair_choices if x["flair_text_editable"] and flair == x["flair_text"]) - submission.flair.select(flair["flair_template_id"]) +existing_ids = [] - total_all_flairs = 0 - duplicate_count = 0 - failcount = 0 - successcount = 0 - totalcount = 0 +with open('../data/read-ids.txt', 'r') as edit_ids_file: + for id in [x.strip() for x in edit_ids_file.readlines()]: + existing_ids.append(id) - for submission in reddit.subreddit('placeAtlas2').new(limit=2000): +total_all_flairs = 0 +count_dupe = 0 +count_fail = 0 +count_success = 0 +count_total = 0 + +with open('temp-atlas-manual.txt', 'w', encoding='utf-8') as FAIL_FILE: + + submission: Submission + for submission in reddit.subreddit('placeAtlas2').new(limit=1000): total_all_flairs += 1 - if (submission.id in existing_ids): - set_flair(submission, "Processed Entry") - print("Found first duplicate!") - duplicate_count += 1 - if (duplicate_count > 0): - break - else: - continue + print(f"{submission.id}: Submitted {humanize.naturaltime(datetime.utcnow() - datetime.utcfromtimestamp(submission.created_utc))}.") - if submission.link_flair_text == "New Entry" or submission.link_flair_text == "Edit Entry": + # print(patches_dir + 'reddit-' + submission.id + '.json') + if submission.id in existing_ids or Path(patches_dir + 'reddit-' + submission.id + '.json').is_file(): + set_flair(submission, "Processed Entry") + print(f"{submission.id}: Submission is a duplicate! Skipped.") + if (count_dupe == 1): + print(f"{submission.id}: Second duplicate. Stopped!") + break + print(f"{submission.id}: First duplicate. Continue running.") + count_dupe += 1 + continue + + print(f"{submission.id}: Processing...") + + if submission.link_flair_text == "New Entry" or submission.link_flair_text == "Edit Entry" or True: try: @@ -102,16 +118,11 @@ def set_flair(submission, flair): if submission_json: if submission.link_flair_text == "Edit Entry": - - assert submission_json["id"] != 0, "Edit invalid because ID is tampered, it must not be 0!" - - submission_json_dummy = {"id": submission_json["id"], "edit": submission.id} - + assert submission_json["id"] > 0, "Edit invalid because ID is tampered, it must not be 0 or -1!" else: - - assert submission_json["id"] == 0, "Edit invalid because ID is tampered, it must be 0!" - - submission_json_dummy = {"id": submission.id} + assert submission_json["id"] <= 0, "Addition invalid because ID is tampered, it must be 0 or -1!" + + submission_json_dummy = {"id": submission_json["id"], "_reddit_id": submission.id, "_author": submission.author.name} for key in submission_json: if not key in submission_json_dummy: @@ -121,13 +132,11 @@ def set_flair(submission, flair): assert validation_status < 3, \ "Submission invalid after validation. This may be caused by not enough points on the path." + + with open(f'{patches_dir}reddit-{submission.id}-{"-".join(submission["name"].split()).lower()}.json', 'w', encoding='utf-8') as out_file: + out_file.write(json.dumps(submission_json, ensure_ascii=False)) - add_comma_line = len(OUT_FILE_LINES) - 2 - if len(OUT_FILE_LINES[add_comma_line]) > 2: - OUT_FILE_LINES[add_comma_line] = OUT_FILE_LINES[add_comma_line].replace('\n', ',\n') - OUT_FILE_LINES.insert(len(OUT_FILE_LINES) - 1, json.dumps(submission_json, ensure_ascii=False) + '\n') - READ_IDS_FILE.write(submission.id + '\n') - successcount += 1 + count_success += 1 set_flair(submission, "Processed Entry") except Exception as e: @@ -140,12 +149,11 @@ def set_flair(submission, flair): "==== CLEAN ====" + "\n\n" + text + "\n\n" ) - failcount += 1 + count_fail += 1 set_flair(submission, "Rejected Entry") + print(f"{submission.id}: Something went wrong! Rejected.") - print("Wrote " + submission.id + ", submitted " + str(round(time.time()-submission.created_utc)) + " seconds ago") - totalcount += 1 + count_total += 1 + print(f"{submission.id}: Processed!") - OUT_FILE.writelines(OUT_FILE_LINES) - -print(f"\n\nTotal all flairs: {total_all_flairs}\nSuccess: {successcount}/{totalcount}\nFail: {failcount}/{totalcount}\nPlease check temp-atlas-manual.txt for failed entries to manually resolve.") +print(f"\n\nTotal all flairs: {total_all_flairs}\nSuccess: {count_success}/{count_total}\nFail: {count_fail}/{count_total}\nPlease check temp-atlas-manual.txt for failed entries to manually resolve.") diff --git a/tools/requirements.txt b/tools/requirements.txt index 9d9d90a4..23415418 100644 --- a/tools/requirements.txt +++ b/tools/requirements.txt @@ -1 +1,3 @@ -praw \ No newline at end of file +praw +tqdm +humanize \ No newline at end of file diff --git a/tools/scale_back.py b/tools/scale_back.py index 75cd0bf4..9cbccd2e 100644 --- a/tools/scale_back.py +++ b/tools/scale_back.py @@ -1,10 +1,12 @@ #!/usr/bin/python +from io import TextIOWrapper import json import traceback import numpy from PIL import Image, ImageDraw import gc +import tqdm """ # 166 to 164 with reference of 165 @@ -147,16 +149,17 @@ def remove_white(entry: dict): return entry -def per_line_entries(entries: list): +def per_line_entries(entries: list, file: TextIOWrapper): """ Returns a string of all the entries, with every entry in one line. """ - out = "[\n" - for entry in entries: - if entry: - out += json.dumps(entry, ensure_ascii=False) + ",\n" - out = out[:-2] + "\n]" - return out + file.write("[\n") + line_temp = "" + for entry in tqdm.tqdm(entries): + if line_temp: + file.write(line_temp + ",\n") + line_temp = json.dumps(entry, ensure_ascii=False) + file.write(line_temp + "\n]") def format_all(entry: dict, silent=False): def print_(*args, **kwargs): @@ -168,7 +171,7 @@ def print_(*args, **kwargs): return entry def scale_back_entries(entries): - for i in range(len(entries)): + for i in tqdm.trange(len(entries)): try: entry_formatted = format_all(entries[i], True) entries[i] = entry_formatted @@ -191,7 +194,7 @@ def go(path): print(f"{len(entries)} checked. Writing...") with open(path, "w", encoding='utf-8', newline='\n') as f2: - f2.write(per_line_entries(entries)) + per_line_entries(entries, f2) print("Writing completed. All done.") diff --git a/tools/schema/atlas.json b/tools/schema/atlas.json new file mode 100644 index 00000000..4559efb8 --- /dev/null +++ b/tools/schema/atlas.json @@ -0,0 +1,130 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "type": "array", + "definitions": { + "entry": { + "type": "object", + "properties": { + "id": { + "oneOf": [ + { + "type": "string" + }, + { + "type": "integer", + "minimum": 1 + }, + { + "type": "integer", + "minimum": -1, + "maximum": 0, + "description": "The ID of the entry. The value is a placeholder for new entries." + } + ], + "description": "The ID of the entry. Usually, this is a number (string or number) this is the post ID of the new entry submission." + }, + "name": { + "type": "string", + "description": "The short, descriptive name of the entry.", + "minLength": 1 + }, + "description": { + "type": "string", + "description": "The description of the entry. that will also be understood by somebody not familiar with the topic. Usually, the first sentence on Wikipedia is a good example." + }, + "links": { + "type": "object", + "description": "The links related to the entry.", + "properties": { + "subreddit": { + "type": "array", + "description": "Subreddits that's either most relevant to the topic, or that was responsible for creating the artwork, excluding the r/.", + "items": { + "type": "string", + "description": "A subreddit that's either most relevant to the topic, or that was responsible for creating the artwork.", + "pattern": "^[A-Za-z0-9][A-Za-z0-9_]{1,20}$", + "minItems": 1 + } + }, + "website": { + "type": "array", + "description": "URL to websites related to the entry, including the http/https protocol. If you're describing a project, the project's main website would be suitable here.", + "items": { + "type": "string", + "description": "The URL to a website related to the entry.", + "pattern": "^https?://[^\\s/$.?#].[^\\s]*$", + "minItems": 1 + } + }, + "discord": { + "type": "array", + "description": "Invite codes of Discord servers related to the entry (excluding discord.gg/)", + "items": { + "type": "string", + "description": "The invite code of a Discord server related to the entry.", + "minItems": 1, + "minLength": 1 + } + }, + "wiki": { + "type": "array", + "description": "Wiki pages related to the entry.", + "items": { + "type": "string", + "description": "The title of the wiki page related to the entry.", + "minItems": 1, + "minLength": 1 + } + } + }, + "additionalProperties": false + }, + "path": { + "type": "object", + "description": "The path of the entry.", + "patternProperties": { + "^(\\d+(-\\d+)?|\\w+(:\\d+(-\\d+)?)?)(, (\\d+(-\\d+)?|\\w+(:\\d+(-\\d+)?)?))*$": { + "type": "array", + "description": "A period containing the path points.", + "items": { + "type": "array", + "description": "A point.", + "items": { + "type": "number" + }, + "minItems": 2, + "maxItems": 2 + }, + "minItems": 3 + } + }, + "additionalProperties": false, + "minProperties": 1 + }, + "center": { + "type": "object", + "description": "The center of the entry.", + "patternProperties": { + "^(\\d+(-\\d+)?|\\w+(:\\d+(-\\d+)?)?)(, (\\d+(-\\d+)?|\\w+(:\\d+(-\\d+)?)?))*$": { + "type": "array", + "description": "A period containing the center point.", + "items": { + "type": "number", + "description": "A point." + }, + "minItems": 2, + "maxItems": 2 + } + }, + "additionalProperties": false, + "minProperties": 1 + } + }, + "required": ["id", "name", "description", "links", "path", "center"], + "additionalItems": true + } + }, + "items": { + "$ref": "#/definitions/entry" + } +} \ No newline at end of file diff --git a/tools/schema/patch.json b/tools/schema/patch.json new file mode 100644 index 00000000..081aa57a --- /dev/null +++ b/tools/schema/patch.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft-07/schema", + "$ref": "atlas.json#/definitions/entry", + "properties": { + "_author": { + "type": "string", + "description": "Patch only: Author of the entry.", + "minLength": 1 + }, + "_reddit_id": { + "type": "string", + "description": "Patch only: Submission ID, if submitted from Reddit.", + "minLength": 1 + } + } +} \ No newline at end of file diff --git a/web/_js/about.js b/web/_js/about.js index 7b444d3a..1dad15c7 100644 --- a/web/_js/about.js +++ b/web/_js/about.js @@ -5,17 +5,35 @@ * Licensed under AGPL-3.0 (https://place-atlas.stefanocoding.me/license.txt) */ -const redditWrapperEl = document.querySelector('#reddit-contributors-wrapper') +const contributorsEl = document.querySelector('#contributors-wrapper') + +// +const gitHubEl = document.createElement("i") +gitHubEl.ariaLabel = "GitHub:" +gitHubEl.className = "bi bi-github" + fetch('all-authors.txt') .then(response => response.text()) - .then(text => text.trim().split('\n').sort()) + .then(text => text.trim().split('\n').sort((a, b) => { + const aSplit = a.split(':') + const bSplit = b.split(':') + return aSplit[aSplit.length - 1] > bSplit[bSplit.length - 1] + })) .then(contributors => { - document.querySelector('#reddit-contributors-count').textContent = contributors.length + document.querySelector('#contributors-count').textContent = contributors.length for (const contributor of contributors) { const userEl = document.createElement('a') - userEl.href = 'https://reddit.com/user/' + contributor - userEl.textContent = contributor - redditWrapperEl.appendChild(userEl) - redditWrapperEl.appendChild(document.createTextNode(' ')) + const contributorSplit = contributor.split(':') + if (contributorSplit[0] === "gh") { + const contributor1 = contributorSplit[1] + userEl.href = 'https://github.com/' + contributor1 + userEl.appendChild(gitHubEl.cloneNode()) + userEl.appendChild(document.createTextNode(' ' + contributor1)) + } else { + userEl.href = 'https://reddit.com/user/' + contributor + userEl.textContent = contributor + } + contributorsEl.appendChild(userEl) + contributorsEl.appendChild(document.createTextNode(' ')) } }) \ No newline at end of file diff --git a/web/_js/main/draw.js b/web/_js/main/draw.js index 5cc07c07..cd1c0763 100644 --- a/web/_js/main/draw.js +++ b/web/_js/main/draw.js @@ -260,7 +260,7 @@ function initDraw() { function generateExportObject() { const exportObject = { - id: entryId ?? 0, + id: entryId ?? -1, name: nameField.value, description: descriptionField.value, links: {}, diff --git a/web/_js/main/main.js b/web/_js/main/main.js index 3174804f..4a60b860 100644 --- a/web/_js/main/main.js +++ b/web/_js/main/main.js @@ -500,7 +500,9 @@ async function init() { } function updateAtlasAll(atlas = atlasAll) { - for (const entry of atlas) { + for (const index in atlas) { + const entry = atlas[index] + entry._index = index const currentLinks = entry.links entry.links = { website: [], diff --git a/web/_js/main/time.js b/web/_js/main/time.js index 04af6408..f7911f68 100644 --- a/web/_js/main/time.js +++ b/web/_js/main/time.js @@ -106,18 +106,20 @@ async function updateBackground(newPeriod = currentPeriod, newVariation = curren } const canvas = document.createElement('canvas') const context = canvas.getContext('2d') - for await (const url of layerUrls) { + + layers.length = layerUrls.length + await Promise.all(layerUrls.map(async (url, i) => { const imageLayer = new Image() await new Promise(resolve => { imageLayer.onload = () => { context.canvas.width = Math.max(imageLayer.width, context.canvas.width) context.canvas.height = Math.max(imageLayer.height, context.canvas.height) - layers.push(imageLayer) + layers[i] = imageLayer resolve() } imageLayer.src = url }) - } + })) for (const imageLayer of layers) { context.drawImage(imageLayer, 0, 0) diff --git a/web/_js/main/view.js b/web/_js/main/view.js index 9b209a7b..b4f9e373 100644 --- a/web/_js/main/view.js +++ b/web/_js/main/view.js @@ -370,10 +370,10 @@ function buildObjectsList(filter, sort = defaultSort) { sortFunction = (a, b) => b.name.toLowerCase().localeCompare(a.name.toLowerCase()) break case "newest": - sortFunction = (a, b) => b.id.length - a.id.length || b.id.localeCompare(a.id) + sortFunction = (a, b) => b._index - a._index break case "oldest": - sortFunction = (a, b) => a.id.length - b.id.length || a.id.localeCompare(b.id) + sortFunction = (a, b) => a._index - b._index break case "area": sortFunction = (a, b) => calcPolygonArea(b.path) - calcPolygonArea(a.path) diff --git a/web/about.html b/web/about.html index 44ef6319..da1d4821 100644 --- a/web/about.html +++ b/web/about.html @@ -180,10 +180,10 @@
The 2022 Atlas would not have been possible without the help of our Reddit contributors.
+The 2022 Atlas would not have been possible without the help of our contributors.
Thank you to everyone who submitted new entries, amended existing ones, reported bugs and just supported the project in general.
- +Use the Post Direct to Reddit button or manually copy the text below and submit it as a new text post to r/placeAtlas2 on Reddit.
-Don't forget to flair it with the New Entry tag.
++ If you want to use Reddit, use the Post Direct to Reddit button or manually copy the text below and submit it as a new text post to r/placeAtlas2 on Reddit. + Don't forget to flair it with the New Entry flair.
+If you want to use GitHub, read the contributing guide to submit a patch.
We will then check it and add it to the Atlas.