Text processing docs (#2046)

* refactoring text normalization docs and tutorial

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* rename nemo tools to nemo text processing

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* rename nemo_tools to nemo_text_processing

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* rename docs

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* rename files

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* rename pytests

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix pytest

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix refactoring

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* renamed functions in ITN tutorial

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix typo

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix path to tutorial in readme for ITN

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix Jenkins

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

* fix Jenkins

Signed-off-by: Yang Zhang <yangzhang@nvidia.com>

Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com>

Dockerfile

@@ -85,9 +85,9 @@ COPY requirements .
 RUN for f in $(ls requirements*.txt); do pip install --disable-pip-version-check --no-cache-dir -r $f; done
 
 
-# install nemo_tools dependencies
-COPY nemo_tools/setup.sh nemo_tools_setup.sh
-RUN bash nemo_tools_setup.sh
+# install nemo_text_processing dependencies
+COPY nemo_text_processing/setup.sh nemo_text_processing_setup.sh
+RUN bash nemo_text_processing.sh
 
 #install TRT tools: PT quantization support and ONNX graph optimizer
 WORKDIR /tmp/trt_build
@@ -114,7 +114,8 @@ RUN --mount=from=nemo-src,target=/tmp/nemo cd /tmp/nemo && pip install ".[all]"
     python -c "import nemo.collections.asr as nemo_asr" && \
     python -c "import nemo.collections.nlp as nemo_nlp" && \
     python -c "import nemo.collections.tts as nemo_tts" && \
-    python -c "import nemo_tools.text_normalization as text_normalization"
+    python -c "import nemo_text_processing.text_normalization as text_normalization"
+
 # TODO: Remove once 21.04 container is base container
 # install latest numba version
 RUN conda update -c numba numba -y

Jenkinsfile

@@ -135,7 +135,7 @@ pipeline {
     //   }
     // }
 
-    stage('L2: NeMo tools') {
+    stage('L2: NeMo text processing') {
       when {
         anyOf {
           branch 'main'
@@ -146,10 +146,10 @@ pipeline {
       parallel {
         stage('L2: pynini export') {
           steps {
-            sh 'cd tools/text_denormalization && python pynini_export.py /home/TestData/nlp/text_denorm/output/ && ls -R /home/TestData/nlp/text_denorm/output/ && echo ".far files created "|| exit 1'
-            sh 'cd tools/text_denormalization && cp *.grm /home/TestData/nlp/text_denorm/output/'
+            sh 'cd tools/inverse_text_normalization_deployment && python pynini_export.py /home/TestData/nlp/text_denorm/output/ && ls -R /home/TestData/nlp/text_denorm/output/ && echo ".far files created "|| exit 1'
+            sh 'cd tools/inverse_text_normalization_deployment && cp *.grm /home/TestData/nlp/text_denorm/output/'
             sh 'ls -R /home/TestData/nlp/text_denorm/output/'
-            sh 'cd nemo_tools/text_denormalization/ &&  python run_predict.py --input=/home/TestData/nlp/text_denorm/ci/test.txt --output=/home/TestData/nlp/text_denorm/output/test.pynini.txt --verbose'
+            sh 'cd nemo_text_processing/inverse_text_normalization/ &&  python run_predict.py --input=/home/TestData/nlp/text_denorm/ci/test.txt --output=/home/TestData/nlp/text_denorm/output/test.pynini.txt --verbose'
             sh 'cmp --silent /home/TestData/nlp/text_denorm/output/test.pynini.txt /home/TestData/nlp/text_denorm/ci/test_goal_py.txt || exit 1'
             sh 'rm -rf /home/TestData/nlp/text_denorm/output/*'
           }

docs/source/index.rst

@@ -51,10 +51,10 @@ NVIDIA NeMo User Guide
 
 .. toctree::
    :maxdepth: 2
-   :caption: NeMo-Tools Package
-   :name: NeMo-Tools Package
+   :caption: NeMo-Text-Processing Package
+   :name: NeMo-Text-Processing Package
 
-   nemo_tools/intro
+   nemo_text_processing/intro
 
 .. toctree::
    :maxdepth: 2

docs/source/nemo_text_processing/intro.rst

@@ -0,0 +1,11 @@
+`nemo_text_processing` is a python package that is installed with the `nemo_toolkit`.
+
+See :doc:`NeMo Introduction <../starthere/intro>` for installation details.
+
+.. toctree::
+   :maxdepth: 1
+
+   text_normalization
+   inverse_text_normalization
+
+

docs/source/nemo_text_processing/inverse_text_normalization.rst

@@ -0,0 +1,27 @@
+Inverse Text Normalization
+==========================
+
+Inverse text normalization (ITN), also called denormalization, is a part of the Automatic Speech Recognition (ASR) post-processing pipeline.
+ITN is the task of converting the raw spoken output of the ASR model into its written form to improve text readability.
+
+For example, 
+`"in nineteen seventy"` -> `"in 1975"` 
+and `"one hundred and twenty three dollars"` -> `"$123"`.
+
+This tool is based on WFST-grammars :cite:`tools-itn-mohri2009`. We also provide a deployment route to C++ using Sparrowhawk -- an open-source version of Google Kestrel :cite:`tools-itn-ebden2015kestrel`.
+See :doc:`ITN Deployment <../tools/inverse_text_normalization_deployment>` for details.
+
+.. note::
+
+    For more details, see the tutorial `NeMo/tutorials/tools/Inverse_Text_Normalization.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/Inverse_Text_Normalization.ipynb>`__.
+
+
+
+
+References
+----------
+
+.. bibliography:: tools_all.bib
+    :style: plain
+    :labelprefix: TOOLS-ITN
+    :keyprefix: tools-itn-

docs/source/nemo_text_processing/text_normalization.rst

@@ -0,0 +1,40 @@
+Text Normalization
+==================
+
+This tool converts text from written form into its verbalized form, including numbers and dates, `10:00` -> `ten o'clock`, `10kg` -> `ten kilograms`.
+Text normalization is used as a preprocessing step before Text to Speech (TTS). It could also be used for preprocessing Automatic Speech Recognition (ASR) training transcripts.
+
+.. note::
+
+    We recommend you try the tutorial `NeMo/tutorials/tools/Text_Normalization_Tutorial.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/Text_Normalization_Tutorial.ipynb>`__.
+    
+
+Prediction
+----------------------------------
+
+Example prediction run:
+
+.. code::
+
+    python run_prediction.py  --input=<INPUT_TEXT_FILE> --output=<OUTPUT_PATH>
+
+
+Evaluation
+----------------------------------
+
+Example evaluation run:
+
+.. code::
+
+    python run_evaluation.py  --input=./en_with_types/output-00001-of-00100 [--cat CLASS_CATEGORY]
+
+
+References
+----------
+
+.. bibliography:: tools_all.bib
+    :style: plain
+    :labelprefix: TOOLS-NORM
+    :keyprefix: tools-norm-
+
+

docs/source/nemo_text_processing/tools_all.bib

@@ -0,0 +1,51 @@
+@inproceedings{kurzinger2020ctc,
+  title={CTC-segmentation of large corpora for german end-to-end speech recognition},
+  author={K{\"u}rzinger, Ludwig and Winkelbauer, Dominik and Li, Lujun and Watzel, Tobias and Rigoll, Gerhard},
+  booktitle={International Conference on Speech and Computer},
+  pages={267--278},
+  year={2020},
+  organization={Springer}
+}
+
+@article{ebden2015kestrel,
+  title={The Kestrel TTS text normalization system},
+  author={Ebden, Peter and Sproat, Richard},
+  journal={Natural Language Engineering},
+  volume={21},
+  number={3},
+  pages={333},
+  year={2015},
+  publisher={Cambridge University Press}
+}
+
+@article{sproat2016rnn,
+  title={RNN approaches to text normalization: A challenge},
+  author={Sproat, Richard and Jaitly, Navdeep},
+  journal={arXiv preprint arXiv:1611.00068},
+  year={2016}
+}
+
+@book{taylor2009text,
+  title={Text-to-speech synthesis},
+  author={Taylor, Paul},
+  year={2009},
+  publisher={Cambridge university press}
+}
+
+
+@Inbook{mohri2009,
+author="Mohri, Mehryar",
+editor="Droste, Manfred
+and Kuich, Werner
+and Vogler, Heiko",
+title="Weighted Automata Algorithms",
+bookTitle="Handbook of Weighted Automata",
+year="2009",
+publisher="Springer Berlin Heidelberg",
+address="Berlin, Heidelberg",
+pages="213--254",
+abstract="Weighted automata and transducers are widely used in modern applications in bioinformatics and text, speech, and image processing. This chapter describes several fundamental weighted automata and shortest-distance algorithms including composition, determinization, minimization, and synchronization, as well as single-source and all-pairs shortest distance algorithms over general semirings. It presents the pseudocode of these algorithms, gives an analysis of their running time complexity, and illustrates their use in some simple cases. Many other complex weighted automata and transducer algorithms used in practice can be obtained by combining these core algorithms.",
+isbn="978-3-642-01492-5",
+doi="10.1007/978-3-642-01492-5_6",
+url="https://doi.org/10.1007/978-3-642-01492-5_6"
+}

docs/source/nemo_tools/intro.rst

@@ -1,9 +0,0 @@
-NeMo's nemo_tools is a python package that is installed with the `nemo_toolkit`.
-
-.. toctree::
-   :maxdepth: 1
-
-   text_normalization
-   text_denormalization
-
-

docs/source/nemo_tools/text_denormalization.rst

@@ -1,13 +0,0 @@
-Text Denormalization
-====================
-
-Inverse text normalization (ITN), also called denormalization, is a part of the Automatic Speech Recognition (ASR) post-processing pipeline.
-ITN is the task of converting the raw spoken output of the ASR model into its written form to improve the text readability.
-For example, `in nineteen seventy` -> `in 1975` and `one hundred and twenty three dollars` -> `$123`.
-
-.. note::
-
-    For more details, see the tutorial `NeMo/tutorials/tools/Text_Denormalization_(Inverse_Text_Normalization).ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/Text_Denormalization_(Inverse_Text_Normalization).ipynb>`__.
-
-
-

docs/source/nemo_tools/text_normalization.rst

@@ -1,58 +0,0 @@
-Text Normalization
-==================
-
-Text normalization converts text into its verbalized form. That is, tokens belonging to special semiotic classes to denote things like 
-**numbers**, **times**, **dates**, **monetary amounts**, etc., that are often written in a way that differs from the way they are verbalized. 
-For example, `10:00` -> `ten o'clock`, `10:00 a.m.` -> `ten a m`, `10kg` -> `ten kilograms`.
-Text normalization is used as a preprocessing step before Text to Speech (TTS). It could also be used for preprocessing Automatic Speech Recognition (ASR) training transcripts.
-
-This tool offers prediction on text files and evaluation on `Google text normalization dataset <https://www.kaggle.com/richardwilliamsproat/text-normalization-for-english-russian-and-polish>`__ :cite:`tools-norm-ebden2015kestrel`, :cite:`tools-norm-sproat2016rnn`, :cite:`tools-norm-taylor2009text`.
-It reaches 81% in sentence accuracy on `output-00001-of-00100` of `Google text normalization dataset <https://www.kaggle.com/richardwilliamsproat/text-normalization-for-english-russian-and-polish>`__, 97.4% in token accuracy.
-
-.. note::
-
-    We recommend you try the tutorial `NeMo/tutorials/tools/Text_Normalization_Tutorial.ipynb <https://github.com/NVIDIA/NeMo/blob/main/tutorials/tools/Text_Normalization_Tutorial.ipynb>`__.
-    
-
-We use the same semiotic classes as in the Google Text normalization dataset: 
-**PLAIN**, **PUNCT**, **DATE**, **CARDINAL**, **LETTERS**, **VERBATIM**, **MEASURE**, **DECIMAL**, **ORDINAL**, **DIGIT**, **MONEY**, **TELEPHONE**, **ELECTRONIC**, **FRACTION**, **TIME**, **ADDRESS**. 
-We additionally added the class **WHITELIST** for all whitelisted tokens whose verbalizations are directly looked up from a user-defined list.
-
-NeMo rule-based system is divided into a tagger and a verbalizer: 
-the tagger is responsible for detecting and classifying semiotic classes in the underlying text, 
-the verbalizer takes the output of the tagger and carries out the normalization. 
-In the example `The alarm goes off at 10:30 a.m.`, the tagger for time detects `10:30 a.m.` as a valid time data with `hour=10`, `minutes=30`, `suffix=a.m.`, 
-the verbalizer then turns this into ten thirty a m. 
-
-The system is designed to be easily debuggable and extendable by more rules. 
-We provide a set of rules that covers the majority of semiotic classes as found in the Google Text normalization dataset for the English language. As with every language there is a long tail of special cases.
-
-Prediction
-----------------------------------
-
-Example prediction run:
-
-.. code::
-
-    python run_prediction.py  --input=<INPUT_TEXT_FILE> --output=<OUTPUT_PATH>
-
-
-Evaluation
-----------------------------------
-
-Example evaluation run:
-
-.. code::
-
-    python run_evaluation.py  --input=./en_with_types/output-00001-of-00100 [--cat CLASS_CATEGORY]
-
-
-References
-----------
-
-.. bibliography:: tools_all.bib
-    :style: plain
-    :labelprefix: TOOLS-NORM
-    :keyprefix: tools-norm-
-
-

docs/source/nemo_tools/tools_all.bib

@@ -1,33 +0,0 @@
-@inproceedings{kurzinger2020ctc,
-  title={CTC-segmentation of large corpora for german end-to-end speech recognition},
-  author={K{\"u}rzinger, Ludwig and Winkelbauer, Dominik and Li, Lujun and Watzel, Tobias and Rigoll, Gerhard},
-  booktitle={International Conference on Speech and Computer},
-  pages={267--278},
-  year={2020},
-  organization={Springer}
-}
-
-@article{ebden2015kestrel,
-  title={The Kestrel TTS text normalization system},
-  author={Ebden, Peter and Sproat, Richard},
-  journal={Natural Language Engineering},
-  volume={21},
-  number={3},
-  pages={333},
-  year={2015},
-  publisher={Cambridge University Press}
-}
-
-@article{sproat2016rnn,
-  title={RNN approaches to text normalization: A challenge},
-  author={Sproat, Richard and Jaitly, Navdeep},
-  journal={arXiv preprint arXiv:1611.00068},
-  year={2016}
-}
-
-@book{taylor2009text,
-  title={Text-to-speech synthesis},
-  author={Taylor, Paul},
-  year={2009},
-  publisher={Cambridge university press}
-}

docs/source/starthere/tutorials.rst

@@ -107,8 +107,8 @@ To run tutorials:
      - CTC Segmentation
      - `CTC Segmentation <https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tools/CTC_Segmentation_Tutorial.ipynb>`_
    * - Tools
-     - Text Normalization for Text To Speech
+     - Text Normalization for TTS
      - `Text Normalization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tools/Text_Normalization_Tutorial.ipynb>`_
    * - Tools
-     - Text Denormalization for Text To Speech
-     - `Text Denormalization(Inverse_Text_Normalization) <https://colab.research.google.com/github/NVIDIA/NeMo/blob/r1.0.0rc1/tutorials/tools/Text_Denormalization_(Inverse_Text_Normalization).ipynb>`_
+     - Inverse Text Normalization for ASR
+     - `Inverse Text Normalization <https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/tools/Inverse_Text_Normalization.ipynb>`_

docs/source/tools/intro.rst

@@ -9,5 +9,6 @@ NeMo provides a set of tools useful for developing Automatic Speech Recognitions
 
    ctc_segmentation
    speech_data_explorer
+   inverse_text_normalization_deployment
 
 

docs/source/tools/inverse_text_normalization_deployment.rst

@@ -1,19 +1,15 @@
-**NeMo/tools/text_denormalization**
-=========================================
-
-Introduction
-------------
-
-This folder provides scripts to deploy WFST-based grammars in `nemo_tools <https://github.com/NVIDIA/NeMo/blob/main/nemo_tools>`_ for
-Inverse Text Normalization system in production.
+Inverse Text Normalization Deployment
+===============================================
 
+This tool deploys :doc:`NeMo Inverse Text Normalization <../nemo_text_processing/inverse_text_normalization>` for production.
+It uses Sparrowhawk -- an open-source version of Google Kestrel :cite:`tools-itn-ebden2015kestrel`.
 
 Requirements
 ------------------------
 
-1) nemo_tools
+1) :doc:`nemo_text_processing <../nemo_text_processing/intro>` package
 
-See NeMo `README <https://github.com/NVIDIA/NeMo/blob/main/README.rst>`_ for installation guide.
+See :doc:`NeMo Introduction <../starthere/intro>` for installation details.
 
 
 Usage
@@ -27,7 +23,7 @@ Automatically start docker container with production backend with plugged in gra
 
 This script runs the following steps in sequence:
 
-Exports grammars `tokenize_and_classify_tmp.far` and `verbalize_tmp.far` from nemo_tools to directory `classify/` and `verbalize/` respectively
+Exports grammars `tokenize_and_classify_tmp.far` and `verbalize_tmp.far` from `nemo_text_processing` to directory `classify/` and `verbalize/` respectively
 
 .. code-block:: bash
 
@@ -61,4 +57,12 @@ Runs Inverse Text Normalization:
 
     echo "two dollars fifty" | ../../src/bin/normalizer_main --config=sparrowhawk_configuration.ascii_proto
 
-This returns $2.50.
+This returns $2.50.
+
+References
+----------
+
+.. bibliography:: tools_all.bib
+    :style: plain
+    :labelprefix: TOOLS-ITN
+    :keyprefix: tools-itn-

nemo_text_processing/README.rst

@@ -0,0 +1,8 @@
+**nemo_text_processing**
+==========================
+
+Introduction
+------------
+
+NeMo's `nemo_text_processing` is a Python package that is installed with the `nemo_toolkit`. 
+See `documentation <https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/>`_ for details.

nemo_text_processing/inverse_text_normalization/README.md

@@ -6,7 +6,7 @@ Install dependencies:
 bash setup.sh
 
 Example prediction run:
-python run_predict.py  --input=`INPUT_FILE` --output=`OUTPUT_FILE` [--denormalizer nemo]
+python run_predict.py  --input=`INPUT_FILE` --output=`OUTPUT_FILE` [--inverse_normalizer nemo]
 
 Example evaluation run:
-python run_evaluate.py  --input=./en_with_types/output-00001-of-00100 [--denormalizer nemo]
+python run_evaluate.py  --input=./en_with_types/output-00001-of-00100 [--inverse_normalizer nemo]

nemo_text_processing/inverse_text_normalization/clean_eval_data.py

@@ -16,7 +16,7 @@ from argparse import ArgumentParser
 from typing import List
 
 import regex as re
-from nemo_tools.text_denormalization.data_loader_utils import (
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import (
     EOS_TYPE,
     Instance,
     load_files,

nemo_text_processing/inverse_text_normalization/graph_utils.py

@@ -18,7 +18,7 @@ import string
 from pathlib import Path
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
 from pynini import Far
 from pynini.examples import plurals
 from pynini.lib import byte, pynutil, utf8

nemo_text_processing/inverse_text_normalization/inverse_normalize.py

@@ -18,9 +18,9 @@ from collections import OrderedDict
 from typing import List
 
 import pynini
-from nemo_tools.text_denormalization.taggers.tokenize_and_classify_final import ClassifyFinalFst
-from nemo_tools.text_denormalization.token_parser import PRESERVE_ORDER_KEY, TokenParser
-from nemo_tools.text_denormalization.verbalizers.verbalize_final import VerbalizeFinalFst
+from nemo_text_processing.inverse_text_normalization.taggers.tokenize_and_classify_final import ClassifyFinalFst
+from nemo_text_processing.inverse_text_normalization.token_parser import PRESERVE_ORDER_KEY, TokenParser
+from nemo_text_processing.inverse_text_normalization.verbalizers.verbalize_final import VerbalizeFinalFst
 from tqdm import tqdm
 
 tagger = ClassifyFinalFst()
@@ -142,7 +142,7 @@ def select_verbalizer(lattice: pynini.FstLike) -> str:
     return output
 
 
-def denormalize(text: str, verbose: bool) -> str:
+def inverse_normalize(text: str, verbose: bool) -> str:
     """
     main function. normalizes spoken tokens in given text to its written form
         e.g. twelve kilograms -> 12 kg
@@ -171,7 +171,7 @@ def denormalize(text: str, verbose: bool) -> str:
     raise ValueError()
 
 
-def denormalize_identity(texts: List[str], verbose=False) -> List[str]:
+def inverse_normalize_identity(texts: List[str], verbose=False) -> List[str]:
     """
     Identity function. Returns input unchanged
 
@@ -183,7 +183,7 @@ def denormalize_identity(texts: List[str], verbose=False) -> List[str]:
     return texts
 
 
-def denormalize_nemo(texts: List[str], verbose=False) -> List[str]:
+def inverse_normalize_nemo(texts: List[str], verbose=False) -> List[str]:
     """
     NeMo inverse text normalizer 
 
@@ -195,19 +195,19 @@ def denormalize_nemo(texts: List[str], verbose=False) -> List[str]:
     res = []
     for input in tqdm(texts):
         try:
-            text = denormalize(input, verbose=verbose)
+            text = inverse_normalize(input, verbose=verbose)
         except:
             raise Exception
         res.append(text)
     return res
 
 
-DENORMALIZERS = {
-    "identity": denormalize_identity,
-    "nemo": denormalize_nemo,
+INVERSE_NORMALIZERS = {
+    "identity": inverse_normalize_identity,
+    "nemo": inverse_normalize_nemo,
 }
 
 
 if __name__ == "__main__":
     s = sys.argv[1]  # input string
-    denormalize(s, verbose=True)
+    inverse_normalize(s, verbose=True)

nemo_text_processing/inverse_text_normalization/run_evaluate.py

@@ -14,15 +14,15 @@
 
 from argparse import ArgumentParser
 
-from nemo_tools.text_denormalization.clean_eval_data import filter_loaded_data
-from nemo_tools.text_denormalization.data_loader_utils import (
+from nemo_text_processing.inverse_text_normalization.clean_eval_data import filter_loaded_data
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import (
     evaluate,
     known_types,
     load_files,
     training_data_to_sentences,
     training_data_to_tokens,
 )
-from nemo_tools.text_denormalization.denormalize import DENORMALIZERS
+from nemo_text_processing.inverse_text_normalization.inverse_normalize import INVERSE_NORMALIZERS
 
 
 '''
@@ -35,9 +35,9 @@ def parse_args():
     parser = ArgumentParser()
     parser.add_argument("--input", help="input file path", type=str)
     parser.add_argument(
-        "--denormalizer",
+        "--inverse_normalizer",
         default='nemo',
-        help="denormalizer to use (" + ", ".join(DENORMALIZERS.keys()) + ")",
+        help="inverse_normalizer to use (" + ", ".join(DENORMALIZERS.keys()) + ")",
         type=str,
     )
     parser.add_argument(
@@ -57,7 +57,7 @@ if __name__ == "__main__":
     # python run_evaluate.py --input=<INPUT> --cat=<CATEGORY> --filter
     args = parse_args()
     file_path = args.input
-    denormalizer = DENORMALIZERS[args.denormalizer]
+    inverse_normalizer = INVERSE_NORMALIZERS[args.inverse_normalizer]
 
     print("Loading training data: " + file_path)
     training_data = load_files([file_path])
@@ -69,7 +69,7 @@ if __name__ == "__main__":
         print("Sentence level evaluation...")
         sentences_un_normalized, sentences_normalized, _ = training_data_to_sentences(training_data)
         print("- Data: " + str(len(sentences_normalized)) + " sentences")
-        sentences_prediction = denormalizer(sentences_normalized)
+        sentences_prediction = inverse_normalizer(sentences_normalized)
         print("- Denormalized. Evaluating...")
         sentences_accuracy = evaluate(
             preds=sentences_prediction, labels=sentences_un_normalized, input=sentences_normalized
@@ -83,7 +83,7 @@ if __name__ == "__main__":
         print("- Token type: " + token_type)
         tokens_un_normalized, tokens_normalized = tokens_per_type[token_type]
         print("  - Data: " + str(len(tokens_normalized)) + " tokens")
-        tokens_prediction = denormalizer(tokens_normalized)
+        tokens_prediction = inverse_normalizer(tokens_normalized)
         print("  - Denormalized. Evaluating...")
         token_accuracy[token_type] = evaluate(tokens_prediction, tokens_un_normalized, input=tokens_normalized)
         print("  - Accuracy: " + str(token_accuracy[token_type]))
@@ -105,7 +105,7 @@ if __name__ == "__main__":
         c2 = ['Num Tokens', len(sentences_normalized)] + [
             token_count_per_type[known_type] if known_type in tokens_per_type else '0' for known_type in known_types
         ]
-        c3 = [args.denormalizer, sentences_accuracy] + [
+        c3 = [args.inverse_normalizer, sentences_accuracy] + [
             token_accuracy[known_type] if known_type in token_accuracy else '0' for known_type in known_types
         ]
 
@@ -113,4 +113,4 @@ if __name__ == "__main__":
             print(f'{str(c1[i]):10s} | {str(c2[i]):10s} | {str(c3[i]):5s}')
     else:
         print(f'numbers\t{token_count_per_type[args.category]}')
-        print(f'{args.denormalizer}\t{token_accuracy[args.category]}')
+        print(f'{args.inverse_normalizer}\t{token_accuracy[args.category]}')

nemo_text_processing/inverse_text_normalization/run_predict.py

@@ -15,7 +15,7 @@
 from argparse import ArgumentParser
 from typing import List
 
-from nemo_tools.text_denormalization.denormalize import DENORMALIZERS
+from nemo_text_processing.inverse_text_normalization.inverse_normalize import INVERSE_NORMALIZERS
 
 
 '''
@@ -59,19 +59,19 @@ def parse_args():
     parser.add_argument("--input", help="input file path", required=True, type=str)
     parser.add_argument("--output", help="output file path", required=True, type=str)
     parser.add_argument("--verbose", help="print denormalization info. For debugging", action='store_true')
-    parser.add_argument("--denormalizer", default='nemo', type=str)
+    parser.add_argument("--inverse_normalizer", default='nemo', type=str)
     return parser.parse_args()
 
 
 if __name__ == "__main__":
     args = parse_args()
     file_path = args.input
-    denormalizer = DENORMALIZERS[args.denormalizer]
+    inverse_normalizer = INVERSE_NORMALIZERS[args.inverse_normalizer]
 
     print("Loading data: " + file_path)
     data = load_file(file_path)
 
     print("- Data: " + str(len(data)) + " sentences")
-    denormalizer_prediction = denormalizer(data, verbose=args.verbose)
-    write_file(args.output, denormalizer_prediction)
+    inverse_normalizer_prediction = inverse_normalizer(data, verbose=args.verbose)
+    write_file(args.output, inverse_normalizer_prediction)
     print(f"- Normalized. Writing out to {args.output}")

nemo_text_processing/inverse_text_normalization/taggers/cardinal.py

@@ -14,9 +14,15 @@
 
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import NEMO_DIGIT, NEMO_SIGMA, NEMO_SPACE, GraphFst, delete_space
-from nemo_tools.text_denormalization.utils import num_to_word
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_DIGIT,
+    NEMO_SIGMA,
+    NEMO_SPACE,
+    GraphFst,
+    delete_space,
+)
+from nemo_text_processing.inverse_text_normalization.utils import num_to_word
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/date.py

@@ -13,8 +13,13 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import NEMO_SIGMA, GraphFst, delete_extra_space, delete_space
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_SIGMA,
+    GraphFst,
+    delete_extra_space,
+    delete_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/decimal.py

@@ -13,8 +13,13 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import NEMO_DIGIT, GraphFst, delete_extra_space, delete_space
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_DIGIT,
+    GraphFst,
+    delete_extra_space,
+    delete_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/electronic.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
 
 
 class ElectronicFst(GraphFst):

nemo_text_processing/inverse_text_normalization/taggers/fraction.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
 
 
 class FractionFst(GraphFst):

nemo_text_processing/inverse_text_normalization/taggers/measure.py

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import (
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
     NEMO_SIGMA,
     GraphFst,
     convert_space,

nemo_text_processing/inverse_text_normalization/taggers/money.py

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import (
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
     NEMO_DIGIT,
     NEMO_SIGMA,
     GraphFst,

nemo_text_processing/inverse_text_normalization/taggers/ordinal.py

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, GraphFst
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_CHAR, GraphFst
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/punctuation.py

@@ -13,7 +13,7 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import GraphFst, convert_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst, convert_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/telephone.py

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
 
 
 class TelefoneFst(GraphFst):

nemo_text_processing/inverse_text_normalization/taggers/time.py

@@ -14,16 +14,16 @@
 
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import (
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
     GraphFst,
     convert_space,
     delete_extra_space,
     delete_space,
     insert_space,
 )
-from nemo_tools.text_denormalization.taggers.cardinal import CardinalFst
-from nemo_tools.text_denormalization.utils import num_to_word
+from nemo_text_processing.inverse_text_normalization.taggers.cardinal import CardinalFst
+from nemo_text_processing.inverse_text_normalization.utils import num_to_word
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/tokenize_and_classify.py

@@ -12,16 +12,16 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
-from nemo_tools.text_denormalization.taggers.cardinal import CardinalFst
-from nemo_tools.text_denormalization.taggers.date import DateFst
-from nemo_tools.text_denormalization.taggers.decimal import DecimalFst
-from nemo_tools.text_denormalization.taggers.measure import MeasureFst
-from nemo_tools.text_denormalization.taggers.money import MoneyFst
-from nemo_tools.text_denormalization.taggers.ordinal import OrdinalFst
-from nemo_tools.text_denormalization.taggers.time import TimeFst
-from nemo_tools.text_denormalization.taggers.whitelist import WhiteListFst
-from nemo_tools.text_denormalization.taggers.word import WordFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.taggers.cardinal import CardinalFst
+from nemo_text_processing.inverse_text_normalization.taggers.date import DateFst
+from nemo_text_processing.inverse_text_normalization.taggers.decimal import DecimalFst
+from nemo_text_processing.inverse_text_normalization.taggers.measure import MeasureFst
+from nemo_text_processing.inverse_text_normalization.taggers.money import MoneyFst
+from nemo_text_processing.inverse_text_normalization.taggers.ordinal import OrdinalFst
+from nemo_text_processing.inverse_text_normalization.taggers.time import TimeFst
+from nemo_text_processing.inverse_text_normalization.taggers.whitelist import WhiteListFst
+from nemo_text_processing.inverse_text_normalization.taggers.word import WordFst
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/tokenize_and_classify_final.py

@@ -13,9 +13,9 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import GraphFst, delete_extra_space, delete_space
-from nemo_tools.text_denormalization.taggers.punctuation import PunctuationFst
-from nemo_tools.text_denormalization.taggers.tokenize_and_classify import ClassifyFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst, delete_extra_space, delete_space
+from nemo_text_processing.inverse_text_normalization.taggers.punctuation import PunctuationFst
+from nemo_text_processing.inverse_text_normalization.taggers.tokenize_and_classify import ClassifyFst
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/whitelist.py

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import GraphFst, convert_space
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst, convert_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/taggers/word.py

@@ -13,8 +13,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.data_loader_utils import get_abs_path
-from nemo_tools.text_denormalization.graph_utils import NEMO_NOT_SPACE, GraphFst, convert_space
+from nemo_text_processing.inverse_text_normalization.data_loader_utils import get_abs_path
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_NOT_SPACE, GraphFst, convert_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/cardinal.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_NOT_QUOTE, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_NOT_QUOTE, GraphFst, delete_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/date.py

@@ -14,7 +14,12 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_NOT_QUOTE, GraphFst, delete_extra_space, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_NOT_QUOTE,
+    GraphFst,
+    delete_extra_space,
+    delete_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/decimal.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_NOT_QUOTE, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_NOT_QUOTE, GraphFst, delete_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/fraction.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
 
 
 class FractionFst(GraphFst):

nemo_text_processing/inverse_text_normalization/verbalizers/measure.py

@@ -14,9 +14,9 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, GraphFst, delete_space
-from nemo_tools.text_denormalization.verbalizers.cardinal import CardinalFst
-from nemo_tools.text_denormalization.verbalizers.decimal import DecimalFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_CHAR, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.verbalizers.cardinal import CardinalFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.decimal import DecimalFst
 from pynini.lib import pynutil
 
 decimal = DecimalFst()

nemo_text_processing/inverse_text_normalization/verbalizers/money.py

@@ -14,8 +14,8 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, GraphFst, delete_space
-from nemo_tools.text_denormalization.verbalizers.decimal import DecimalFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_CHAR, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.verbalizers.decimal import DecimalFst
 from pynini.lib import pynutil
 
 decimal = DecimalFst()

nemo_text_processing/inverse_text_normalization/verbalizers/ordinal.py

@@ -14,7 +14,12 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_NOT_QUOTE, NEMO_SIGMA, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_NOT_QUOTE,
+    NEMO_SIGMA,
+    GraphFst,
+    delete_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/punctuation.py

@@ -14,7 +14,12 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, NEMO_NOT_QUOTE, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_CHAR,
+    NEMO_NOT_QUOTE,
+    GraphFst,
+    delete_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/time.py

@@ -14,7 +14,13 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, NEMO_DIGIT, GraphFst, delete_space, insert_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import (
+    NEMO_CHAR,
+    NEMO_DIGIT,
+    GraphFst,
+    delete_space,
+    insert_space,
+)
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/verbalize.py

@@ -13,15 +13,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from nemo_tools.text_denormalization.graph_utils import GraphFst
-from nemo_tools.text_denormalization.verbalizers.cardinal import CardinalFst
-from nemo_tools.text_denormalization.verbalizers.date import DateFst
-from nemo_tools.text_denormalization.verbalizers.decimal import DecimalFst
-from nemo_tools.text_denormalization.verbalizers.measure import MeasureFst
-from nemo_tools.text_denormalization.verbalizers.money import MoneyFst
-from nemo_tools.text_denormalization.verbalizers.ordinal import OrdinalFst
-from nemo_tools.text_denormalization.verbalizers.time import TimeFst
-from nemo_tools.text_denormalization.verbalizers.whitelist import WhiteListFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.cardinal import CardinalFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.date import DateFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.decimal import DecimalFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.measure import MeasureFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.money import MoneyFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.ordinal import OrdinalFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.time import TimeFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.whitelist import WhiteListFst
 
 
 class VerbalizeFst(GraphFst):

nemo_text_processing/inverse_text_normalization/verbalizers/verbalize_final.py

@@ -14,10 +14,10 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import GraphFst, delete_extra_space, delete_space
-from nemo_tools.text_denormalization.verbalizers.punctuation import PunctuationFst
-from nemo_tools.text_denormalization.verbalizers.verbalize import VerbalizeFst
-from nemo_tools.text_denormalization.verbalizers.word import WordFst
+from nemo_text_processing.inverse_text_normalization.graph_utils import GraphFst, delete_extra_space, delete_space
+from nemo_text_processing.inverse_text_normalization.verbalizers.punctuation import PunctuationFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.verbalize import VerbalizeFst
+from nemo_text_processing.inverse_text_normalization.verbalizers.word import WordFst
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/whitelist.py

@@ -15,7 +15,7 @@
 
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, NEMO_SIGMA, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_CHAR, NEMO_SIGMA, GraphFst, delete_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/inverse_text_normalization/verbalizers/word.py

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 import pynini
-from nemo_tools.text_denormalization.graph_utils import NEMO_CHAR, NEMO_SIGMA, GraphFst, delete_space
+from nemo_text_processing.inverse_text_normalization.graph_utils import NEMO_CHAR, NEMO_SIGMA, GraphFst, delete_space
 from pynini.lib import pynutil
 
 

nemo_text_processing/text_normalization/normalize.py

@@ -15,8 +15,8 @@
 import sys
 from typing import List
 
-from nemo_tools.text_normalization.tag import Tag, TagType
-from nemo_tools.text_normalization.tagger import (
+from nemo_text_processing.text_normalization.tag import Tag, TagType
+from nemo_text_processing.text_normalization.tagger import (
     tag_cardinal,
     tag_date,
     tag_decimal,
@@ -27,7 +27,7 @@ from nemo_tools.text_normalization.tagger import (
     tag_verbatim,
     tag_whitelist,
 )
-from nemo_tools.text_normalization.verbalizer import (
+from nemo_text_processing.text_normalization.verbalizer import (
     expand_cardinal,
     expand_date,
     expand_decimal,
@@ -206,4 +206,4 @@ normalizers = {"identity": normalize_identity, "nemo": normalize_nemo}
 if __name__ == "__main__":
     # Example usage:
     s = sys.argv[1]  # input string
-    normalize_nemo(s, verbose=True)
+    normalize(s, verbose=True)

nemo_text_processing/text_normalization/run_evaluate.py

@@ -14,8 +14,8 @@
 
 from argparse import ArgumentParser
 
-from nemo_tools.text_normalization.normalize import normalizers
-from nemo_tools.text_normalization.utils import (
+from nemo_text_processing.text_normalization.normalize import normalizers
+from nemo_text_processing.text_normalization.utils import (
     evaluate,
     known_types,
     load_files,

nemo_text_processing/text_normalization/run_predict.py

@@ -16,7 +16,7 @@ from argparse import ArgumentParser
 from time import perf_counter
 from typing import List
 
-from nemo_tools.text_normalization.normalize import normalizers
+from nemo_text_processing.text_normalization.normalize import normalizers
 
 
 '''

nemo_text_processing/text_normalization/tagger.py

@@ -14,8 +14,8 @@
 
 
 import regex as re
-from nemo_tools.text_normalization.tag import Tag, TagType
-from nemo_tools.text_normalization.verbalizer import _currency_dict, _measurements_dict, _whitelist_dict
+from nemo_text_processing.text_normalization.tag import Tag, TagType
+from nemo_text_processing.text_normalization.verbalizer import _currency_dict, _measurements_dict, _whitelist_dict
 
 
 def make_re(re_inner: str, *args):

nemo_tools/README.rst

@@ -1,14 +0,0 @@
-**nemo_tools**
-==========================
-
-Introduction
-------------
-
-NeMo's nemo_tools is a python package that is installed with the nemo_toolkit. See NeMo `README.rst <https://github.com/NVIDIA/NeMo/blob/main/README.rst>`_ for installation guide.
-
-Additional requirements
-------------------------
-
-1) pynini=2.1.3
-
-setup.sh installs all additional requirements.

reinstall.sh

@@ -19,7 +19,7 @@ ${PIP} uninstall -y nemo_cv
 
 ${PIP} install -U setuptools
 
-echo 'Installing nemo and nemo_tools'
+echo 'Installing nemo and nemo_text_processing'
 if [[ "$INSTALL_OPTION" == "dev" ]]; then
     ${PIP} install --editable ".[all]"
 else
@@ -29,8 +29,7 @@ else
     ${PIP} install "${DIST_FILE}[all]"
 fi
 
-
-echo 'Installing nemo_tools requirements'
-bash nemo_tools/setup.sh
+echo 'Installing additional nemo_text_processing conda dependency'
+bash nemo_text_processing/setup.sh
 
 echo 'All done!'

setup.py

@@ -81,7 +81,7 @@ extras_require = {
     'cv': req_file("requirements_cv.txt"),
     'nlp': req_file("requirements_nlp.txt"),
     'tts': req_file("requirements_tts.txt"),
-    'tools': req_file("requirements_tools.txt"),
+    'text_processing': req_file("requirements_text_processing.txt"),
 }
 
 extras_require['all'] = list(chain(extras_require.values()))