1 line
41 KiB
Plaintext
1 line
41 KiB
Plaintext
{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","colab":{"name":"auto_mixed_precision_demo_cifar10.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"jupytext":{"formats":"ipynb,py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.8"}},"cells":[{"cell_type":"code","metadata":{"colab_type":"code","id":"tuOe1ymfHZPu","colab":{}},"source":["# Copyright 2019 NVIDIA Corporation. All Rights Reserved.\n","#\n","# Licensed under the Apache License, Version 2.0 (the \"License\");\n","# you may not use this file except in compliance with the License.\n","# You may obtain a copy of the License at\n","#\n","# http://www.apache.org/licenses/LICENSE-2.0\n","#\n","# Unless required by applicable law or agreed to in writing, software\n","# distributed under the License is distributed on an \"AS IS\" BASIS,\n","# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n","# See the License for the specific language governing permissions and\n","# limitations under the License.\n","# =============================================================================="],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"MfBg1C5NB3X0"},"source":["<img src=\"https://upload.wikimedia.org/wikipedia/en/thumb/6/6d/Nvidia_image_logo.svg/200px-Nvidia_image_logo.svg.png\" width=\"90px\" align=\"right\" style=\"margin-right: 0px;\">\n","\n","# Mixed Precision Training of CNN"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"xHxb-dlhMIzW"},"source":["## Overview\n","\n","In this example, we will speed-up the training of a simple CNN with mixed precision to perform image classification on the CIFAR10 dataset.\n","\n","By using mixed precision, we can reduce the training time without a significant impact on classification accuracy. For example, using the NVIDIA Tesla T4 GPU on Google Colab, we can reduce the training time (using the same model and batch size) over 10 epochs from about 900 seconds (FP32) to less than 600 seconds with mixed precision, without sacrificing classification accuracy.\n","\n","### How mixed precision works\n","\n","**Mixed precision** is the use of both float16 and float32 data types when training a model.\n","\n","Performing arithmetic operations in float16 takes advantage of the performance gains of using specialized processing units such as the Tensor cores. Due to the smaller representable range of float16, performing the entire training with float16 data type can result in underflow of the gradients, leading to convergence or model quality issues.\n","\n","However, *performing only select arithmetic operations* in float16 results in performance gains when using compatible hardware accelerators, decreasing training time and reducing memory usage, typically without sacrificing model performance.\n","\n","To learn more about mixed precision and how it works:\n","\n","* [Overview of Automatic Mixed Precision for Deep Learning](https://developer.nvidia.com/automatic-mixed-precision)\n","* [NVIDIA Mixed Precision Training Documentation](https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html)\n","* [NVIDIA Deep Learning Performance Guide](https://docs.nvidia.com/deeplearning/sdk/dl-performance-guide/index.html)\n","* [Information about NVIDIA Tensor Cores](https://developer.nvidia.com/tensor-cores)\n","* [Post on TensorFlow blog explaining Automatic Mixed Precision](https://medium.com/tensorflow/automatic-mixed-precision-in-tensorflow-for-faster-ai-training-on-nvidia-gpus-6033234b2540)\n","\n","### TensorFlow Automatic Mixed Precision API\n","\n","The method presented in this notebook is the API used in TensorFlow 1.14 and newer: `tf.train.experimental.enable_mixed_precision_graph_rewrite()`. \n","\n","This allows you to switch to using mixed precision by simply wrapping a `tf.keras.optimizers` Optimizer in `tf.train.experimental.enable_mixed_precision_graph_rewrite()`. For more details, you can consult the [relevant TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/train/experimental/enable_mixed_precision_graph_rewrite). \n"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"MUXex9ctTuDB"},"source":["## Setup and Requirements"]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"0yh5tPwanSjf"},"source":["**Hardware requirements**\n","\n","* NVIDIA Tensor Core GPU (Compute Capability >= `7.0`)\n","\n","**Software requirements**\n","\n","* TensorFlow version >= `1.14.0-rc0`\n","\n","The following section will import the necessary libraries and check if these requirements are met."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"IqR2PQG4ZaZ0","colab":{}},"source":["import time\n","import numpy as np\n","\n","import tensorflow.compat.v2 as tf\n","tf.enable_v2_behavior()\n","\n","from tensorflow.keras.models import Sequential\n","from tensorflow.keras.layers import *"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"SyOV7ojinSjf","outputId":"f290d073-73b5-46f3-a070-c360f3831756","colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["from tensorflow.python.client import device_lib\n","\n","def check_tensor_core_gpu_present():\n"," local_device_protos = device_lib.list_local_devices()\n"," for line in local_device_protos:\n"," if \"compute capability\" in str(line):\n"," compute_capability = float(line.physical_device_desc.split(\"compute capability: \")[-1])\n"," if compute_capability>=7.0:\n"," return True\n","\n","print(\"TensorFlow version is\", tf.__version__)\n","\n","try:\n"," # check and assert TensorFlow >= 1.14\n"," tf_version_list = tf.__version__.split(\".\")\n"," if int(tf_version_list[0]) < 2:\n"," assert int(tf_version_list[1]) >= 14\n","except:\n"," print(\"TensorFlow 1.14.0 or newer is required.\")\n"," \n","print(\"Tensor Core GPU Present:\", check_tensor_core_gpu_present())\n","if check_tensor_core_gpu_present():\n"," pass\n","else:\n"," !nvidia-smi\n"," assert check_tensor_core_gpu_present() == True"],"execution_count":0,"outputs":[{"output_type":"stream","text":["TensorFlow version is 1.14.0\n","Tensor Core GPU Present: True\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"QKp40qS-DGEZ"},"source":["## Import the Dataset\n","\n","Import the CIFAR10 image dataset from `tf.keras.datasets`"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"GQq1V2EmnSjj","colab":{}},"source":["# The data, split between train and test sets\n","\n","(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n","\n","num_classes = np.max(y_train) + 1\n","\n","# Convert class vectors to binary class matrices\n","\n","y_train = tf.keras.utils.to_categorical(y_train, num_classes)\n","y_test = tf.keras.utils.to_categorical(y_test, num_classes)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"EPSlI16QnSjl"},"source":["Preprocess the images by scaling the values from the range `0 ~ 255` to the range `0 ~ 1`"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"4W52Ac5-nSjl","colab":{}},"source":["def normalize(ndarray):\n"," ndarray = ndarray.astype(\"float32\")\n"," ndarray = ndarray/255.0\n"," return ndarray\n","\n","x_train = normalize(x_train)\n","x_test = normalize(x_test)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"0zTEfgJFnSjn"},"source":["## Define the Model\n","\n","Define a reusable helper function to return a simple CNN"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"edD6HTTmnSjo","colab":{}},"source":["def create_model(num_classes=10):\n"," \"\"\"\n"," Returns a simple CNN suitable for classifiying images from CIFAR10\n"," \"\"\"\n"," # model parameters\n"," act = \"relu\"\n"," pad = \"same\"\n"," ini = \"he_uniform\"\n"," \n"," model = tf.keras.models.Sequential([\n"," Conv2D(128, (3, 3), activation=act, padding=pad, kernel_initializer=ini,\n"," input_shape=(32,32,3)),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," MaxPooling2D(pool_size=(2,2)),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(512, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(512, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," MaxPooling2D(pool_size=(2,2)),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(256, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," Conv2D(128, (3, 3), activation=act, padding=pad, kernel_initializer=ini),\n"," MaxPooling2D(pool_size=(4,4)),\n"," Flatten(),\n"," BatchNormalization(),\n"," Dense(512, activation='relu'),\n"," Dense(num_classes, activation=\"softmax\")\n"," ])\n","\n"," return model"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"VsZy8zzwjz1V","outputId":"82c39333-3017-43d0-b74c-b8ad1166d755","colab":{"base_uri":"https://localhost:8080/","height":799}},"source":["model = create_model(num_classes)\n","model.summary()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Model: \"sequential\"\n","_________________________________________________________________\n","Layer (type) Output Shape Param # \n","=================================================================\n","conv2d (Conv2D) (None, 32, 32, 128) 3584 \n","_________________________________________________________________\n","conv2d_1 (Conv2D) (None, 32, 32, 256) 295168 \n","_________________________________________________________________\n","conv2d_2 (Conv2D) (None, 32, 32, 256) 590080 \n","_________________________________________________________________\n","conv2d_3 (Conv2D) (None, 32, 32, 256) 590080 \n","_________________________________________________________________\n","max_pooling2d (MaxPooling2D) (None, 16, 16, 256) 0 \n","_________________________________________________________________\n","conv2d_4 (Conv2D) (None, 16, 16, 256) 590080 \n","_________________________________________________________________\n","conv2d_5 (Conv2D) (None, 16, 16, 256) 590080 \n","_________________________________________________________________\n","conv2d_6 (Conv2D) (None, 16, 16, 512) 1180160 \n","_________________________________________________________________\n","conv2d_7 (Conv2D) (None, 16, 16, 512) 2359808 \n","_________________________________________________________________\n","max_pooling2d_1 (MaxPooling2 (None, 8, 8, 512) 0 \n","_________________________________________________________________\n","conv2d_8 (Conv2D) (None, 8, 8, 256) 1179904 \n","_________________________________________________________________\n","conv2d_9 (Conv2D) (None, 8, 8, 256) 590080 \n","_________________________________________________________________\n","conv2d_10 (Conv2D) (None, 8, 8, 256) 590080 \n","_________________________________________________________________\n","conv2d_11 (Conv2D) (None, 8, 8, 128) 295040 \n","_________________________________________________________________\n","max_pooling2d_2 (MaxPooling2 (None, 2, 2, 128) 0 \n","_________________________________________________________________\n","flatten (Flatten) (None, 512) 0 \n","_________________________________________________________________\n","batch_normalization (BatchNo (None, 512) 2048 \n","_________________________________________________________________\n","dense (Dense) (None, 512) 262656 \n","_________________________________________________________________\n","dense_1 (Dense) (None, 10) 5130 \n","=================================================================\n","Total params: 9,123,978\n","Trainable params: 9,122,954\n","Non-trainable params: 1,024\n","_________________________________________________________________\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"OB8kdGNTnSjr"},"source":["## Training the Model\n","\n","Train and benchmark the same model trained with and without mixed precision"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"vXHS_e8lnSjs","colab":{}},"source":["# training parameters\n","BATCH_SIZE = 320\n","N_EPOCHS = 10\n","opt = tf.keras.optimizers.SGD(learning_rate=0.02, momentum=0.5)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"G05htP5vnSju","colab":{}},"source":["def train_model(mixed_precision, optimizer):\n"," \"\"\"\n"," Trains a CNN to classify images on CIFAR10,\n"," and returns the training and classification performance\n"," \n"," Args:\n"," mixed_precision: `True` or `False`\n"," optimizer: An instance of `tf.keras.optimizers.Optimizer`\n"," \"\"\"\n"," model = create_model(num_classes)\n","\n"," if mixed_precision:\n"," import tensorflow\n"," optimizer = tensorflow.compat.v1.train.experimental.enable_mixed_precision_graph_rewrite(optimizer)\n","\n"," model.compile(loss=\"categorical_crossentropy\",\n"," optimizer=optimizer,\n"," metrics=[\"accuracy\"])\n"," \n"," train_start = time.time()\n","\n"," train_log = model.fit(x_train, y_train,\n"," batch_size=BATCH_SIZE,\n"," epochs=N_EPOCHS,\n"," use_multiprocessing=True,\n"," workers=2)\n","\n"," train_end = time.time()\n","\n"," \n"," \n"," results = {\"test_loss\": score[0],\n"," \"test_acc\": score[1],\n"," \"train_time\": train_end-train_start,\n"," \"train_log\": train_log}\n"," \n"," return results"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"B1o7YirynSjw"},"source":["### Training with FP32"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"jS4XPcLFnSjw","outputId":"0231bd28-1b2f-4a0a-a960-14b47137ee19","colab":{"base_uri":"https://localhost:8080/","height":479}},"source":["fp32_results = train_model(mixed_precision=False, optimizer=opt)\n","\n","test_acc = round(fp32_results[\"test_acc\"]*100, 1)\n","train_time = round(fp32_results[\"train_time\"], 1)\n","\n","print(test_acc, \"% achieved in\", train_time, \"seconds\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["WARNING: Logging before flag parsing goes to stderr.\n","W0803 17:03:43.563157 140238721070976 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.where in 2.0, which has the same broadcast rule as np.where\n"],"name":"stderr"},{"output_type":"stream","text":["Epoch 1/10\n","50000/50000 [==============================] - 105s 2ms/sample - loss: 1.8097 - accuracy: 0.3372\n","Epoch 2/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 1.3215 - accuracy: 0.5240\n","Epoch 3/10\n","50000/50000 [==============================] - 88s 2ms/sample - loss: 1.0472 - accuracy: 0.6292\n","Epoch 4/10\n","50000/50000 [==============================] - 88s 2ms/sample - loss: 0.8664 - accuracy: 0.6951\n","Epoch 5/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.7398 - accuracy: 0.7391\n","Epoch 6/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.6447 - accuracy: 0.7744\n","Epoch 7/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.5654 - accuracy: 0.8025\n","Epoch 8/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.4927 - accuracy: 0.8300\n","Epoch 9/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.4201 - accuracy: 0.8555\n","Epoch 10/10\n","50000/50000 [==============================] - 89s 2ms/sample - loss: 0.3560 - accuracy: 0.8794\n","10000/10000 [==============================] - 8s 823us/sample - loss: 1.1442 - accuracy: 0.6635\n","66.4 % achieved in 904.9 seconds\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab_type":"code","id":"QiRUF-v-omKO","colab":{}},"source":["# to ensure accuracy of timing benchmark\n","# we give the GPU 10 seconds to cool down\n","\n","tf.keras.backend.clear_session()\n","\n","time.sleep(10)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"lnA6jdbVnSjy"},"source":["### Training with Mixed Precision"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"KtylpxOmceaC","outputId":"5c94465c-93a8-412d-b524-bb0e35f49a7f","colab":{"base_uri":"https://localhost:8080/","height":391}},"source":["mp_results = train_model(mixed_precision=True, optimizer=opt)\n","\n","test_acc = round(mp_results[\"test_acc\"]*100, 1)\n","train_time = round(mp_results[\"train_time\"], 1)\n","\n","print(test_acc, \"% achieved in\", train_time, \"seconds\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","50000/50000 [==============================] - 72s 1ms/sample - loss: 1.7493 - accuracy: 0.3657\n","Epoch 2/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 1.2876 - accuracy: 0.5376\n","Epoch 3/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 1.0372 - accuracy: 0.6338\n","Epoch 4/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.8650 - accuracy: 0.6942\n","Epoch 5/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.7406 - accuracy: 0.7415\n","Epoch 6/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.6463 - accuracy: 0.7741\n","Epoch 7/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.5586 - accuracy: 0.8065\n","Epoch 8/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.4856 - accuracy: 0.8340\n","Epoch 9/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.4213 - accuracy: 0.8553\n","Epoch 10/10\n","50000/50000 [==============================] - 56s 1ms/sample - loss: 0.3483 - accuracy: 0.8821\n","10000/10000 [==============================] - 5s 544us/sample - loss: 0.9768 - accuracy: 0.7111\n","71.1 % achieved in 575.5 seconds\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"MI1bAx9XtbZD","colab_type":"text"},"source":["As we can see, FP32 and mixed precision training yield comparable or even better test accuracy."]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"1BR_XJP9nSj0"},"source":["### Evaluate the Model Performance"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"mMOeXVmbdilM","outputId":"02121424-9c48-40a6-9046-9a582b0823f0","colab":{"base_uri":"https://localhost:8080/","height":295}},"source":["import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","plt.plot(fp32_results[\"train_log\"].history[\"loss\"], label=\"FP32\")\n","plt.plot(mp_results[\"train_log\"].history[\"loss\"], label=\"Mixed Precision\")\n","plt.title(\"Performance Comparison\")\n","plt.ylabel(\"Training Loss\")\n","plt.xlabel(\"Epoch\")\n","plt.legend()\n","plt.show()"],"execution_count":0,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XdYVGf6//H3TVEQu6BRAcGGDUHF\nrok1MUXd9BgTk5jE9GSTLXFLNll3893s/mKKG9NjTLGnF429G3uviAiKoCI2sFLu3x8zssQIjMh4\nQO7Xdc0lc86ZM/ccdT485znneURVMcYYYwB8nC7AGGNM2WGhYIwxJp+FgjHGmHwWCsYYY/JZKBhj\njMlnoWCMMSafhYK5rETknyJySET2O12LKZ6IzBCR+5yuw1w+YvcpmKKISBJQD8gFTgAzgCdVNasE\n+woHdgCNVPVgadZZlolIdWAUcAtQGzgAfA/8U1UPOVmbMeezloLxxEBVrQq0B+KAv17sDkTEDwgH\nMkoSCO7XlzsiUgmYC7QGBgDVga5ABtDJwdKKJC72/VAB2V+68Ziq7sPVUmgDICI1ROQjEUkTkX3u\nU0O+7nX3i8hSEXldRDKABcBsoIGIZInIePd2g0Rki4gcFZEFItLy3PuJSJKIPC8iG4ETIuLnXvYH\nEdkoIifc71/PfZojU0TmiEitAvuYJiL7ReSYiCwSkdYF1o0XkbEi8qP7tStEpEmB9a1FZLaIHBaR\nAyLyZ/dyHxEZKSK7RCRDRKaKSO1CDtswXGF4s6puVdU8VT2oqv9Q1enu/bV0f/aj7mMx6Lwa33Z/\nviz3Mb1KRN4QkSMisl1E2p13zP4kIlvd6z8WkQD3uloi8oOIpLvX/SAioQVeu0BEXhaRpcBJoLF7\n2UPu9U1FZKH7WB4SkSkFXttNRFa5160SkW7n7fcf7tozRWSWiAQX+Y/NOMZCwXhMRMKAG4B17kXj\ngRygKdAOuBZ4qMBLOgOJuE4/9QeuB1JVtaqq3i8izYFJwG+BEGA68L37t+tzhgA3AjVVNce97Fb3\n/poDA3EF1Z/d+/ABni7w+hlAM6AusBaYcN7Hugv4O1ALSABedn/WasAc4CeggfszznW/5ingN8A1\n7nVHgLGFHLZ+wE+FnW4TEX9cp5JmuWt8CpggIlEFNrsDV+ssGDgD/Oz+LMHAF8Br5+12KHAd0ATX\nMTrXsvMBPgYa4QqqU8Bb5732XmAEUA1IPm/dP9x11gJCgf+6P0Nt4EdgDFDHXc+PIlKnwGvvBh5w\nf8ZKwO8vdDxMGaCq9rBHoQ8gCcgCjuL6kngbCMT1RX8GCCyw7RBgvvvn+4E95+2rF5BS4PkLwNQC\nz32AfUCvAu89/AL1DC3w/EvgnQLPnwK+KeSz1AQUqOF+Ph74sMD6G4DtBT7LukL2sw3oW+B5fSAb\n8LvAtrOBV4o4vj2B/YBPgWWTgJcK1PjBeZ9vW4Hn0cDR847Po+d9pl2FvHcscKTA8wXAqPO2WQA8\n5P75U+B9IPS8be4FVp637Gfg/gL7+GuBdY/jCkrH/33b49ePcnme1lx2v1HVOQUXiEg04A+kici5\nxT7A3gKbFfz5QhpQ4LdRVc0Tkb1Aw2L2caDAz6cu8Lyqu0ZfXL/5346rFZHn3iYYOOb+ueBVUCfP\nvRYIA3YVUncj4GsRySuwLBdXUO47b9sMXKFRmAbAXlUtuK9kfnkMPPq8BRQ8Zsnu90BEqgCv4+rb\nOHeKrZqI+Kpq7gVee74/4motrBSRI8BoVR3HeX+PhXyGwo6zKWPs9JEpqb24WgrBqlrT/aiuqq0L\nbFPcpW2puL5gAVfnJq4v44JfrJdyedzdwGBcp3BqABHn3sqD1+4FGhex7voCn7umqgaoq8/lfHOA\n60QkqJB9pQJh53XqhvPrcLkYYeftK9X98++AKKCzqlYHrnYvL3g8Cj3eqrpfVR9W1QbAI8DbItKU\n8/4eC7zvpXwG4xALBVMiqpqG6/zyaBGp7u58bSIi11zEbqYCN4pIX/e59d/hCpplpVRmNff+MoAq\nwP9dxGt/AOqLyG9FpLKIVBORzu517wIvi0gjABEJEZHBheznM1wh8qWItHAfpzoi8mcRuQFYges3\n5z+KiL+I9MLVTzL5Ij9rQU+ISKj7XP9fgHMdwtVwtSyOute9eDE7FZHbC3RMH8EVIHm4+oKai8jd\n4roY4E6gFa5jaMoZCwVzKYbh6jTciutL4guKPlXyC6q6A7gHV4flIVxfhgNV9Wwp1fcprtMY+9w1\nLr+I2jJxdWYPxHXqYyfQ2736TeA7YJaIZLr327mQ/ZzB1VLZjqt/4TiwEtcprBXuzzoQVyf8IVx9\nNsNUdfvFfNDzTMQV2Im4ToH90738DVz9QYfcNf90kfvtCKwQkSxcn/8ZVU1U1QzgJlyhnoHrNNNN\navdglEt285oxVxBx3Wz40Pl9QMZ4yloKxhhj8lkoGGOMyWenj4wxxuSzloIxxph85e7mteDgYI2I\niHC6DGOMKVfWrFlzSFVDituu3IVCREQEq1evdroMY4wpV0Tk/LvOL8hOHxljjMlnoWCMMSafhYIx\nxph85a5PwRjjPdnZ2aSkpHD69GmnSzElFBAQQGhoKP7+/iV6vYWCMSZfSkoK1apVIyIiggJDopty\nQlXJyMggJSWFyMjIEu3DTh8ZY/KdPn2aOnXqWCCUUyJCnTp1Lqml57VQEJFxInJQRDYXsr6GiHwv\nIhvc89I+4K1ajDGes0Ao3y7178+bLYXxuGZ4KswTwFZVjcE1TePo8+bmLVXJGSf4+/dbyM7NK35j\nY4ypoLwWCqq6CDhc1Ca4pgIUXFPzHcY1CbxXJBzM4uOlSXy5JsVbb2GMKQW+vr7ExsbmP5KSkliw\nYAE1atQgNjaWli1b8ve//x2AlStX5m8XExPD119/DcDevXvp3bs3rVq1onXr1rz55ptOfqRyxcmO\n5rdwTdSRimtGqDvPm6e2VPVpUZfYsJr8d14CN7dvSGU/X2+9lTHmEgQGBrJ+/fpfLEtKSqJnz578\n8MMPnDhxgtjYWAYOHEibNm1YvXo1fn5+pKWlERMTw8CBA/Hz82P06NG0b9+ezMxMOnToQP/+/WnV\nqpVDn6r8cLKj+TpgPa5Jv2OBt0Sk+oU2FJERIrJaRFanp6eX6M1EhN9d25x9R08xdVVx88kbY8qq\noKAgOnToQEJCAlWqVMHPz/W77enTp/PPp9evX5/27dsDUK1aNVq2bMm+fTZltCecbCk8ALyirrG7\nE0RkN9AC11SFv6Cq7wPvA8TFxZV4rO8eTYPpFFGbt+YncHtcGAH+1lowpjB//34LW1OPl+o+WzWo\nzosDWxe5zalTp4iNjQUgMjIy/5TQORkZGSxfvpwXXngBgBUrVjB8+HCSk5P57LPP8kPinKSkJNat\nW0fnzhecMdWcx8lQ2AP0BRaLSD0gCtecsl4jIjx3bXPuen85E1bs4cEeJbuO1xjjPRc6fQSwePFi\n2rVrh4+PDyNHjqR1a1e4dO7cmS1btrBt2zbuu+8+rr/+egICAgDIysri1ltv5Y033qB69QueiDDn\n8VooiMgkXFcVBYtICvAi4A+gqu8C/wDGi8gmQIDnL8dE310a16F70zq8syCBIZ3CqFLJ7t8z5kKK\n+43+cjvXp1CYli1bUrVqVTZv3kxcXBzZ2dnceuutDB06lFtuueUyVlq+ee0bUVWHFLM+FbjWW+9f\nlOf6R3HrO8v4ZFkyj/Vq4kQJxphSsHv3bsLCwvDz8yM5OZnt27cTERGBqvLggw/SsmVLnnvuOafL\nLFcqzh3NZ0/C2s9AlQ6NatErKoT3Fu0i83S205UZY0poyZIlxMTEEBsby80338zbb79NcHAwS5cu\n5bPPPmPevHn5l6xOnz7d6XLLhXI3R3NcXJyWaJKddZ/Dt0/AXROhxY1sTDnKoLeW8lz/5jzdt1np\nF2pMObRt2zZatmzpdBnmEl3o71FE1qhqXHGvrTgthbZ3QXBzmPMS5ObQNrQm/VvV44PFiRw7aa0F\nY4yBihQKvn7Q90U4FA/rPwfguf7NyTydw4dLvHrRkzHGlBsVJxQAWtwIYZ1h/r/g7Ela1q/OjW3r\nM27Jbg6fOOt0dcYY47iKFQoi0H8UZO2H5W8D8Gy/ZpzKzuW9RbscLs4YY5xXsUIBILwLRN0IS9+E\nExk0rVuNwbEN+WRZEgczbbYpY0zFVvFCAaDfi3A2Cxa/CsAzfZuRnau8s8BaC8aYiq1ihkJIFLS7\nB1Z+AEeSiAgO4tb2DZmwYg9px045XZ0xFZqIcM899+Q/z8nJISQkhJtuugmA7777jldeeaVU3qtq\n1aoXXH5u+O42bdpw++23c/LkyUt+r9WrV/P0008Xuj41NZXbbrvtkt/nUlXMUADo9Wfw8YN5/wTg\nqT7NUFXGzk9wuDBjKragoCA2b97MqVOuX9Bmz55Nw4YN89cPGjSIkSNHerWGc+Mvbd68mUqVKvHu\nu+/+Yr2qkpd3cSP9x8XFMWbMmELXN2jQgC+++KJE9ZamihsK1etD18dh0zRI20BY7SrcERfGlFV7\nSTly6b8VGGNK7oYbbuDHH38EYNKkSQwZ8r9Rc8aPH8+TTz4JwODBg/n0008BeO+99xg6dCgAu3bt\nYsCAAXTo0IGePXuyfft2wDUsRteuXYmOjuavf/2rR7X07NmThIQEkpKSiIqKYtiwYbRp04a9e/cy\na9YsunbtSvv27bn99tvJysoCYNWqVXTr1o2YmBg6depEZmYmCxYsyG/tLFy4MP9O63bt2pGZmUlS\nUhJt2rQBXMOAP/DAA0RHR9OuXTvmz5+f/9lvueUWBgwYQLNmzfjjH/94Scf5Qir2aHDdn4HVH8Ps\nF2HYNzzZpynT1qTw37kJ/Pu2tk5XZ4yzZoyE/ZtKd59XRcP1xZ/6ueuuuxg1ahQ33XQTGzduZPjw\n4SxevPhX273//vt0796dyMhIRo8ezfLlywEYMWIE7777Ls2aNWPFihU8/vjjzJs3j2eeeYbHHnuM\nYcOGMXbs2GLryMnJYcaMGQwY4JpZeOfOnXzyySd06dKFQ4cO8c9//pM5c+YQFBTEv//9b1577TVG\njhzJnXfeyZQpU+jYsSPHjx8nMDDwF/t99dVXGTt2LN27dycrKyt/VNdzxo4di4iwadMmtm/fzrXX\nXkt8fDwA69evZ926dVSuXJmoqCieeuopwsLCiv0snqq4LQWAgBpw9R8gcT7smkf9GoHc3SmcL9am\nkHTohNPVGVNhtW3blqSkJCZNmsQNN9xQ6Hb16tVj1KhR9O7dm9GjR1O7dm2ysrJYtmwZt99+O7Gx\nsTzyyCOkpaUBsHTp0vxWx7333lvofs/N6RAXF0d4eDgPPvggAI0aNaJLly4ALF++nK1bt9K9e3di\nY2P55JNPSE5OZseOHdSvX5+OHTsCUL169V/N8dC9e3eee+45xowZw9GjR3+1fsmSJfn9Ki1atKBR\no0b5odC3b19q1KhBQEAArVq1Ijk52ePj6omK3VIA6PggrHjH1VqI7MXjvZswedUexszdyWt3xjpd\nnTHO8eA3em8aNGgQv//971mwYAEZGRmFbrdp0ybq1KlDamoqAHl5edSsWfOCczIA+bOzFaWwOR2C\ngoLyf1ZV+vfvz6RJk35VT3FGjhzJjTfeyPTp0+nevTszZ878VWuhMJUrV87/2dfXl5yc0p3avmK3\nFAD8KkOfF2D/Rtj8JXWrBXBf1wi+Wb+PhIOZTldnTIU1fPhwXnzxRaKjowvdZuXKlcyYMYN169bx\n6quvsnv3bqpXr05kZCTTpk0DXF/eGzZsAFy/oU+ePBmACRMmXFJ9Xbp0YenSpSQkuC5OOXHiBPHx\n8URFRZGWlsaqVasAyMzM/NUX965du4iOjub555+nY8eO+X0e5/Ts2TO/vvj4ePbs2UNUVNQl1esp\nCwWANre5znXOGwU5Z3jkmiYE+vvy+pydTldmTIUVGhpa5CWcZ86c4eGHH2bcuHE0aNCA0aNHM3z4\ncFSVCRMm8NFHHxETE0Pr1q359ttvAXjzzTcZO3Ys0dHRlzxnc0hICOPHj2fIkCG0bduWrl27sn37\ndipVqsSUKVN46qmniImJoX///pw+/csbY9944w3atGlD27Zt8ff35/rrr//F+scff5y8vDyio6O5\n8847GT9+/C9aCN5UcYbOLk7CXPj8FhjwCnR5jFdn7uCt+QnMeKYnLevbNH6mYrChs68MNnR2aWja\nFxr3goX/gdPHeLhnY6oF+PH67HinKzPGmMvGa6EgIuNE5KCIbC5im14isl5EtojIQm/V4rF+f4dT\nh2Hpm9So4s9DPRoza+sBNqUcc7oyY4y5LLzZUhgPDChspYjUBN4GBqlqa+B2L9bimQaxrv6Fn9+G\n42kM7xFBzSr+vDZ7h9OVGXPZlLdTyuaXLvXvz2uhoKqLgMNFbHI38JWq7nFvf9BbtVyUvi9AXg4s\n+BfVAvwZcXVj5u9IZ03yEacrM8brAgICyMjIsGAop1SVjIwMjy9vvRAn71NoDviLyAKgGvCmqn7q\nYD0utSKg40Ow8j3o+gT3dW3CR4t38/rseD5/qLPT1RnjVaGhoaSkpJCenu50KaaEAgICCA0NLfHr\nnQwFP6AD0BcIBH4WkeWq+queXREZAYwACA8P935lV/8e1n0Oc0cRdNcEHuvVhH/+uI3liRl0aVzH\n++9vjEP8/f2JjIx0ugzjICevPkoBZqrqCVU9BCwCYi60oaq+r6pxqhoXEhLi/cqCgqHHM7D9B9iz\nnHu6NKJe9cq8NivemtXGmCuak6HwLdBDRPxEpArQGdjmYD2/1OVxqHoVzP4bAX4+PNG7KSuTDrMk\n4ZDTlRljjNd485LUScDPQJSIpIjIgyLyqIg8CqCq24CfgI3ASuBDVS308tXLrlIQ9BoJe1fAjunc\n2TGMBjUCGG2tBWPMFczuaC5Kbg6809X182M/M2lNKn/6ahPj7o+jT4t6l6cGY4wpBXZHc2nw9YO+\nL8KheFj/Obd1CCW8dhVem22tBWPMlclCoTgtboSwzjD/X/jnnubpvs3YvO84M7cccLoyY4wpdRYK\nxRGB/qMgaz8sf5vfxDagcXAQr8+OJy/PWgvGmCuLhYInwrtA1I2w9E38Th/hmX7N2HEgkx83pTld\nmTHGlCoLBU/1/RuczYLFrzKwbQOi6lXj9Tnx5OTmOV2ZMcaUGgsFT9VtAe3ugZUf4HMsmWf7NyMx\n/QTfrk91ujJjjCk1FgoXo9efwMcP5v2T61pfResG1Xlz7k6yrbVgjLlCWChcjOoNoMtjsGkasn8j\nz/Vvzp7DJ/lyTYrTlRljTKmwULhYPX4LgbVg9ov0aVGX2LCa/HdeAmdycp2uzBhjLpmFwsUKqAFX\n/wES5yOJ83muf3P2HT3F1FV7na7MGGMumYVCSXR8CGqGw+wX6dm0Nh0javHW/AROZ1trwRhTvlko\nlIRfZejzAuzfiGz+iuf6R3Hg+BkmrNjjdGXGGHNJLBRKqs1tcFU0zBtF10ZV6dakDu8sSODk2Ryn\nKzPGmBKzUCgpHx/o93c4ugdWj+N31zbnUNZZPlmW7HRlxhhTYhYKl6JpX2jcCxb+hw71fOkVFcJ7\ni3aReTrb6cqMMaZELBQuVb+X4NRhWDqG5/o35+jJbD5emuRwUcYYUzIWCpeqQTtX/8LPY2lb/RT9\nW9Xjg8WJHDtprQVjTPljoVAa+vwV8nJgwb94rn9zMk/n8OGSRKerMsaYi2ahUBpqR0LHB2HdZ7T0\n28+N0fUZt2Q3h0+cdboyY4y5KF4LBREZJyIHRWRzMdt1FJEcEbnNW7VcFlf/AfyDYO7f+W2/ZpzM\nzuW9RbucrsoYYy6KN1sK44EBRW0gIr7Av4FZXqzj8ggKhh7PwPYfaHZmC4NjGvDJsiQOZp52ujJj\njPGY10JBVRcBh4vZ7CngS+Cgt+q4rLo8DlWvgtkv8kzfZmTnKu8ssNaCMab8cKxPQUQaAjcD73iw\n7QgRWS0iq9PT071fXElVCoJeI2HvciIzFnJr+4ZMWLGHtGOnnK7MGGM84mRH8xvA86pa7Aw1qvq+\nqsapalxISMhlKO0StLsX6jSDOS/xVK9IVJWx8xOcrsoYYzziZCjEAZNFJAm4DXhbRH7jYD2lw9cP\n+r0Ih+IJS/6aO+LCmLJqLylHTjpdmTHGFMuxUFDVSFWNUNUI4AvgcVX9xql6SlWLmyC0Eyz4F0/2\nbICI8N+51lowxpR93rwkdRLwMxAlIiki8qCIPCoij3rrPcsMEeg/CjLTqL/1Y+7uFM4Xa1NIOnTC\n6cqMMaZI3rz6aIiq1ldVf1UNVdWPVPVdVX33Atver6pfeKsWRzTqClE3wNI3eaJzTfx9hTFzdzpd\nlTHGFMnuaPamvi/C2SxC1r3FsK4RfLN+HwkHM52uyhhjCmWh4E11W0DsUFj5AY+19SXA35fX51hr\nwRhTdlkoeFvvP4OPH7VW/D8e6B7BjxvT2JZ23OmqjDHmgiwUvK16A+jyGGyayqPNT1AtwI/XZsc7\nXZUxxlyQhcLl0P0ZCKxFtcX/4NFrmjB76wHeW2jDXxhjyh4LhcshsKZrFNXE+TwWuoeBMQ3414zt\nTFu91+nKjDHmF4oNBRG5RUSquX8eKSJTRSTW+6VdYTo+BDXC8Zn7IqNvi6Zns2BGfrWJOVsPOF2Z\nMcbk86Sl8JKqZopIN+AGYALwq3sNTDH8KrtmaNu/kUrbvuadezrQukF1npi4ltVJxQ0ma4wxl4cn\noZDr/vMm4D1V/Rao7L2SrmDRt8NV0TD371Q9m8HH93ekYc1Aho9fxY79dv+CMcZ5noRCmoiMBe4E\npotIJQ9fZ87n4wM3vQEnD8MnN1FHj/LJ8E4EVvJl2LgVNmieMcZxnny53wEsBG5U1SNAMDDSq1Vd\nyULjYOg0OLYPPrmJMP9MPh3emVNncxn20Uoyss44XaExpgLzJBSCgW9VdbuI9AB+Ayz1bllXuIju\nBYJhIFFBJ/no/o7sO3qK4eNXceJMjtMVGmMqKE9C4RsgT0SaAB8DzYCJXq2qIsgPhr3wyUA61slm\n7N3t2Zx6nEc/X8PZnGLnHjLGmFLnSSjkqWo2cAvwX1V9Fmjo3bIqiIjuMPSL/GDoFy68cks0i3ce\n4nfTNpCXp05XaIypYDwJhRwRuR24F/jBvczfeyVVMAWDYfxN3N6iMiOvb8H3G1IZ9cNWVC0YjDGX\njyehMBzoDfxHVRNFJBKY5N2yKpiCp5LG38Qj7avyUI9Ixi9LsvmdjTGXVbGhoKqbgaeB1SLSAtir\nqi97vbKKJqJHfjDIp4P489V1uKVdQ16dFc+klXucrs4YU0F4MsxFTyAB+AgYB8SLSHdvF1YhRfSA\nu6fC0T34fDaIfw+4it5RIfzl6038tDnN6eqMMRWAJ6ePXgduUNXuqtoNuBF4s7gXicg4ETkoIpsL\nWT9URDaKyCYRWSYiMRdX+hUqsmd+MPh/Ppixg0OJCavJ05PXszwxw+nqjDFXOE9CoZKqbj33RFW3\nAZU8eN14YEAR63cD16hqNPAP4H0P9lkxFAiGKpNu5uPbIgivXYWHP1nNltRjTldnjLmCeRIKa0Xk\nXRHp4X68A6wr7kWquggodKQ3VV3mvkMaYDkQ6lHFFcW5YDiSTM1pt/D5XZFUC/DjvnGrSM444XR1\nxpgrlCeh8CiQCPzR/UgERpRyHQ8CMwpbKSIjRGS1iKxOT08v5bcuwyJ7ujqfjyRz1dd3MGFIY3Ly\n8hg2biXpmTYchjGm9ElJroMXkQmqOtSD7SKAH1S1TRHb9AbeBnqoarEnzePi4nT16tUXUe0VYPci\nmHAH1IpgU7/PuOPzXTQOCWLyiC5UC7BbRowxxRORNaoaV9x2JR3ttGcJX/cLItIW+BAY7EkgVFiR\nV8PQqXAkiei5w/jwtkbs2J/JiE/XcDo7t/jXG2OMhxwbAltEwoGvgHtV1WayL07k1XD3FDi8m+5L\nhzNmUCg/J2bw28nrybXhMIwxpaTQUBCRtoU8YvBgmAsRmQT8DESJSIqIPCgij4rIo+5N/gbUAd4W\nkfUiUsHOCZVA42vyg+GGtY/w8rVX8dOW/bzw7WYbDsMYUyoK7VMQkcVFvVBVS+UU0sWqkH0K50tc\nCBPvhNqRjAl7jdeWHubpvs14rn9zpyszxpRRnvYp+BW2wqkvfeOBxtfA3ZNh4l08xXMciX2FMXN3\nUieoEvd1i3C6OmNMOWbTapZXjXvB3ZORw4n87fBIftO8Mi99v4UfNqY6XZkxphyzUCjPGveCu6cg\nhxN57fQL9Anz4dkp61my85DTlRljyikLhfKucS+4ewo+hxN5X0fRvk4uj3y2mo0pR52uzBhTDnky\nSuqFrkBqJCIWKGVF415w9xR8jyTyeaWXiahymvs/XkViepbTlRljyhlPvtg/AtYAnwKfAauBb4Gd\nItLXi7WZi9G4FwyZjP/R3XwV9Aq19Dj3frSSA8dPO12ZMaYc8SQUkoAOqhqrqjFAByAeuA4Y7cXa\nzMVq0huGTKbysd38UPP/wclDDPtoJcdOZjtdmTGmnPAkFFqq6sZzT1R1E9BKVW2eyLKoSW8YMonA\n47uZVXs0Rw6l8tCnq2w4DGOMRzwJhe0i8l8R6e5+jHEvqwzkeLk+UxJN+sCQSQRlJTE7+HV2JSfz\n5MS15OTmOV2ZMaaM8yQUhgEpwEj3IxW4D1cgWJ9CWeUOhhonkpgT/DprtiXw56832XAYxpgiFRsK\nqnpSVf+tqgPdj1dU9YSq5qqqTQNWljXpA3dNpPapZGbWGc2s1dv4z8wdTldljCnDPLkktYuIzBCR\nrSISf+5xOYozpaBpX7hrIiGn9zC95v9j0oL1fLg40emqjDFllCenjz7GNQlOP1zzKJx7mPKiaV9k\nyCTqZ+/l++r/4a0fV/L1uhSnqzLGlEGehMJxVf1eVVNV9cC5h9crM6XLHQyhuSl8U/U/vDxtKfO2\n21+jMeaXPAmFeSLyLxHpWPCuZq9XZkpf077IkIk00hSmBL7C7z+Zzxtz4u2qJGNMvmLnaC5kXgVV\n1au9U1LRbD6FUpAwB510Nwd96/Fg5ggCG3Xgjbva0bBmoNOVGWO8pNTmaFbVnhd4OBIIppQ07Yfc\n+xX1KufwXcBLXJM6joFvzGN/PAVlAAAY0klEQVT6pjSnKzPGOKyomdeGqOokEXn6QutVdYxXKyuE\ntRRK0akjMP2PsGkq8X7NefTECDrFdeZvA1tRpVKh8y8ZY8qh0mgp1HL/GVLIo7gCxonIQRHZXMh6\nEZExIpIgIhtFpH1x+zSlLLAW3PoB3D6eZv7pzAz4CwFrP2TQmEVsSbVbUIypiIrtUyjxjkWuBrKA\nT1W1zQXW3wA8BdwAdAbeVNXOxe3XWgpekrkfvnsKds5ipUTz+7OPcN/1PRjePQIRcbo6Y8wlKrU+\nBREJFpE/isjbIvL+uUdxr1PVRcDhIjYZjCswVFWXAzVFpH5x+zVeUu0quHsqDHyTjn67+KnS82ye\n/h7DP17JoawzTldnjLlMPLkk9VugHrAEmFvgcakaAnsLPE9xL/sVERkhIqtFZHV6enopvLW5IBHo\ncD/y2DICw9ryeqV3uDPpr9z5+o8sirfjbkxF4EkoBKnq71R1oqpOOffwemUFqOr7qhqnqnEhIcV2\nZ5hLVTsSuf9H6D+Ka/3WMy3vWcaPf5f/m76Nszl2T4MxVzJPQmGGiFzrhffeB4QVeB7qXmbKAh9f\n6P4MPiPmUzOkIeMqvUrksj9xz9tz2H3ohNPVGWO8xJNQeBT4SUSyROSwiBwRkaL6Cjz1HTDMfRVS\nF+CYqtqF8mXNVW3wGTEfejzHXX4Lef3w4/xtzHtMW73XhuE25grkSSgEA/5ADVyXogbj2SWpk4Cf\ngSgRSRGRB0XkURF51L3JdCARSAA+AB4vQf3mcvCrDP1eRIbPoF6NID7xGcXhb57nuYkrOH7apvo0\n5kpS1M1rzVR1Z2HjHBWcovNysktSHXYmi7xZL+CzZhw78sL4V+CzPD30FtqH1yr+tcYYx3h6SWpR\nofCRqj5oYx+ZC9o5m7NfPY6cOsybObcS2PtZHu3dAl8fu6fBmLLokkOhrLJQKENOHib7+2fx3/YN\na/Ka8dlVI3l+6I3Ur2ED6xlT1pRqKIhIC6AVEHBumapOvKQKS8hCoezRTV+Q/d2z5Jw9w+s+9xJ3\n6++5ro3dh2hMWVKadzT/FXgfeBe4HngDuO2SKzRXDIm+jUpPrYDwLvxFPyRgyh38e+o8TmfnOl2a\nMeYieXL10Z1AbyBNVe8FYoAgr1Zlyp/qDagy/Ftyrn+Vrv7xPLplKK+99jLb9x93ujJjzEXwJBRO\nqWoukCMi1YD9QCPvlmXKJRH8Oj9MpSeWIcHN+fOp0ex6+w4mL9xg9zQYU054EgrrRKQmMA5YDax0\nP4y5sDpNqP74XE70/AvX+ayi97xBvPHuOxw+cdbpyowxxSiyo1lcYyZfde5OYxFpClRX1bWXqb5f\nsY7m8iUvdQPHJjxArRO7+NLnWhre/ipdWlpD05jLrVQ6mtWVGLMLPE9wMhBM+ePTIIZav11GettH\nuDlvNldN7s/n06aSnWsD6xlTFnly+mi9iLTzeiXmyuUfQMgt/+HsPd9RvZIPQzaP4NtXR5B8sDSG\n0DLGlKZCQ0FEzk3S2w5YJSI7RGStiKwTEWstmIsW0PRqav9+Ffsib+O2U9M4PbYX8xbMc7osY0wB\nRQ1zsVZV24tIkwutV9VdXq2sENancGU4tOZbfH98hiq5mcyueSdNBz9Pi8bW12CMt3jap+BXxDoB\n5778zZUtuMNgcpp3I/GzJ7jp4AQyP/mK6bVvpengP9I8wsLBGKcU1VJIAV4r7IWqWug6b7KWwpUn\nM3k9ad+NonnGXDI1kKV1bqHZ4JE0aRTudGnGXDFK4+ojX6AqUK2QhzGlolqjWJo/9RXHH1hIanA3\nrs2YSL1xHZk15nESk/c4XZ4xFUqxfQqXuZ5iWUvhyncsaQOp340iKmMuJ6nM8uBbafabkTQKs5aD\nMSVVGi0FGxjfOKJGRAwtn/6S4w8sZE+dHvQ5NJHgD+OY99/H2JtiLQdjvKmoUOh72aow5gJqRsTQ\n6ukvOfrAInbXuZpehyZR+4M45r/1KPssHIzxikJDQVUv+c4iERngvr8hQURGXmB9uIjMd9/7sFFE\nbrjU9zRXntoRbWnz9BccuX8Ru+pcwzXpk6n1QRwLxz5K6j4LB2NKk9dmXhMRXyAe6A+kAKuAIaq6\ntcA27wPrVPUdEWkFTFfViKL2a30KJn33JlK/G0Wbw7M5QyXW1r2ZZjf/hXoNrM/BmMKU2iQ7l6AT\nkKCqiap6FpgMDD5vGwWqu3+uAaR6sR5zhQiJjCbmmWkcum8x8bV70fXgFKq/14FlYx8hPc1aDsZc\nCm+GQkNgb4HnKe5lBb0E3OO+J2I68NSFdiQiI0RktYisTk9P90atphyq1zia2Gemkn7fYrbV6k3n\ng1Oo+m4Hfn57BIcsHIwpEW+GgieGAONVNRS4AfhMRH5Vk6q+r6pxqhoXEhJy2Ys0ZdtVjaNp/9up\nHLxvCVtq9abTgalUfbc9K94eQYaFgzEXxZuhsA8IK/A81L2soAeBqQCq+jMQAAR7sSZzBavfuA1x\nv53K/mFL2FizHx0OTCPo3fasemcEh/dbOBjjCW+GwiqgmYhEikgl4C7gu/O22YP70lcRaYkrFOz8\nkLkkDZu0odOzk0m9dzEbavaj3f5pBL3TntXvPszRAxYOxhTFa6GgqjnAk8BMYBswVVW3iMgoERnk\n3ux3wMMisgGYBNyvNpmvKSXhTdvQ+dnJpNyzmLU1+xOb9gWBb7dn7bsPc+xAstPlGVMmee2SVG+x\nS1JNSe3euZnU71+m07GZ5OHDtvq/ofHNL1C9no3Kaq58ZeGSVGPKlMhmbej+3CT2DF3Eqhr9aZ32\nFQHvtGfDew+ReTDJ6fKMKROspWAqrPgdm9n/w//R9fhPqAjxtXtTs9sDhLYfAD6+TpdnTKnytKVg\noWAqvB3bt5A64z+0PzqbGnKCdJ8Q9kf8hoh+D1OtQZTT5RlTKiwUjLlIh44eY8OcSVTfPoX22evw\nFWVXYDR5MUNpcs1QfAKrF78TY8ooCwVjSkhV2Ra/g5QFH9Ms7TsiSeUUldkd0o/gHg9QN7ov+Fh3\nnClfLBSMKQWnz+awaslMstd8TlzWfKrLKQ761iOj6a1E9n2YgLqNnS7RGI9YKBhTylLTM9g0ZyK1\nd06jQ+5GfETZFdQO3/b30KjHXUjlqk6XaEyhLBSM8ZK8PGX95k3sXzye1gd/oJEc4CSB7Kl/LfWu\nHk6tFteA2MSFpmyxUDDmMsg8dZaVi36EdRPpfGoRVeU0B/0acDTqdiL6PEilOnZjnCkbLBSMucwS\n9x1g69wJXLX7C+J0C3kISdXiCOh0Lw263AH+gU6XaCowCwVjHJKTm8eqdevIWDqe2MPTCZVDnJAq\n7Gt4PQ2ueYiqTbva6SVz2VkoGFMGHM46zfJ531Fp00S6n11KoJzlQKUwTra8k/A+D+Jbo4HTJZoK\nwkLBmDJEVdmWtI+d8z8jbM83tGc7ufiQXLML1boMI6TDzeAf4HSZ5gpmoWBMGXU6O5efV60gc/ln\nxB37iQZymCypyv5GNxHa+yECwuPs9JIpdRYKxpQDaUeyWDnvG4K2TqZHznICJJsD/mEcbTKY0GuG\nEVTfxl4ypcNCwZhyRFVZG59M8qLPCd83nfa6FR9RdldqzpEmg4m45l5qX2WXt5qSs1AwppzKzVM2\nbt1G+vJJhO/7kRa6izwVtlZuy5Emg2l89RAa1rcOanNxLBSMuQKoKgnb1nFo2QTCUqcTmpfKWfVl\nTaWOHG0ymKY9bqNpwxDE+iBMMcpEKIjIAOBNwBf4UFVfucA2dwAvAQpsUNW7i9qnhYKpsFRJ3bac\n9J8/J3TfDOrkZZClASzz78KRJoNo3nUgMeEh+PhYQJhfczwURMQXiAf6AynAKmCIqm4tsE0zYCrQ\nR1WPiEhdVT1Y1H4tFIwB8nI5vHU+Gcsn0mDfTII0iwytxnzf7hxpPIiWnfrTuUkw/r42xLdx8TQU\n/LxYQycgQVUT3QVNBgYDWwts8zAwVlWPABQXCMYYNx9farfpR+02/SDnDCe2zOTUyokMSp1HpYSf\nSNkZzGfSnYzGg2nbvhtXN69LYCWbYtQUz5uh0BDYW+B5CtD5vG2aA4jIUlynmF5S1Z/O35GIjABG\nAISHh3ulWGPKLb/KBMUMIihmEJzJ5OyWH6m8ciL37f8B38Rv2ZEQyrt0Jz1iIHGx7ejboh41qvg7\nXbUpo7x5+ug2YICqPuR+fi/QWVWfLLDND0A2cAcQCiwColX1aGH7tdNHxnjoxCFyN39N1urJ1Eh3\n/Z9Zm9eU7/O6czD8Brq0bcl1repRt7rdSV0RlIXTR/uAsALPQ93LCkoBVqhqNrBbROKBZrj6H4wx\nlyIoGN/OD1Oj88NwdC95m76kxbrJtD/8Cbmpn7F0b2v+83030ur34+roJlzX+ioigoOcrto4zJst\nBT9cHc19cYXBKuBuVd1SYJsBuDqf7xORYGAdEKuqGYXt11oKxlyig9vRTdPI3jCNSseTOYs/c3Lb\n8W1uN/YF96RPmzCubX0VrRtUt0tdryCOX33kLuIG4A1c/QXjVPVlERkFrFbV78T1L240MADIBV5W\n1clF7dNCwZhSogr71sCmaeRu+hLfk+mclCpMz4nj29xuJAbF0rNFQ3pF1aVHs2CqVvbmiQXjbWUi\nFLzBQsEYL8jNgaTFsOkL8rZ+i8/ZTE5LIEvz2jArJ5YlxBIR2ZTeUXXp3aIujYODrBVRzlgoGGNK\nJvs0JC6AnTPR+JnIcVdX4E6fxvx0ti3zc2M5Uiuaa1rUp3eLunSOrE2Av13uWtZZKBhjLp0qHNwK\nO2dB/Cx07wpEc8n0qcG8nGjm5sSy0rcdbZo2ope7FdGwpk07WhZZKBhjSt+pI5AwF3bORhNmIycz\nyMOHTT5RzDzTlnl57dCQVvRuWY/eUSF0aFQLP7urukywUDDGeFdeLuxb6zrNtHMWkrYBgEO+Icw6\n25Z5ubFsrBRDx+Zh9ImqyzVRIQRXrexw0RWXhYIx5vI6ngYJsyF+Jpo4Hzl7gmypxGpa8dPZGBbk\nxVIzNIreUSH0jqpLdMMaNnjfZWShYIxxTs4Z2POzqx9i50wkIwGAFN9QfjrTlnl5sSQGRtM9qiF9\nWrguea0RaENveJOFgjGm7MjYBTtnu041JS1Bcs9y2qcKS/KimZkdw2KNpVGjxvRuUZc+LerSrG5V\nu+S1lFkoGGPKpjNZsHuh6zTTztlIZioAO32bMP1MW+bntuNQ9db0almP3lF16dYk2EZ4LQUWCsaY\nsk8VDmyG+JmwcxaasgrRPI771GR+bjRzs2NY5dOG5o2buPoiWtSlUR0bn6kkLBSMMeXPycPuS15n\noglzkFNHAEjyCWPB2Zb8nNea/bU7ENeiCb2j6tIxshaV/awV4QkLBWNM+ZaXC2nrYfci2L2IvOSf\n8ck5RR7CVo1gWW4r1vpE49+4O91aRdArKoT6NezGucJYKBhjriw5Z10D+O1eRG7iQiRlJT552eTg\nw4a8JizLa01qzThCWvWkZ6tw2oXVtBvnCrBQMMZc2bJPwd4VaOIiTu+cT+WDG/DRXM6qH2vymrPW\nN5ozYT2IjOnB1S0aUqeC3zhnoWCMqVhOH4c9yzmTMJ/T8QuodnQbPigntTKrNIqkah2o3LQXrTr0\npE1o7Qp345yFgjGmYjt5mLykJRzZPBeSFlHnZCIAx7UK63xac6ReF2q37kdMh67UqHLltyIsFIwx\npqDMA2TtWED6pllUTV1GSLbr/ogMrUZ8YCzZ4T0JbT+AyObRiM+V1xdhoWCMMUXIOZzM3rWzOLFj\nHvUOrSREDwFwkDrsqxVHpaa9aNzpegJDIh2utHRYKBhjjKdUSU/eRtLqGZC0iMaZa6kjxwE44Fef\nw/W6EdT6ekLbX49PQFWHiy2ZMhEKIjIAeBPXHM0fquorhWx3K/AF0FFVi/zGt1Awxnjb2exctm5Y\nzv4Nc6iWtpSY7A1UldOcwZ+EKrGcCO9D3Q4DadS0TbkZo8nxUBARXyAe6A+kAKuAIaq69bztqgE/\nApWAJy0UjDFlTdrhYySsmo3Gz6TR4aU0UtcUpUk0ILFWD2h2LU3i+hEeUrPMhoSnoeDnxRo6AQmq\nmuguaDIwGNh63nb/AP4N/MGLtRhjTInVr12D+tfdBtfdhqqSmriV/Wu+IzB5Lj2OfEWllVPJXBHI\nAt9YDta7mqA219O+dQsalMOpSb0ZCg2BvQWepwCdC24gIu2BMFX9UUQKDQURGQGMAAgPD/dCqcYY\n4xkRoUGT1jRo0hr4E3omk7T1s8ja9COx+xdSK+1nSPs3G2dGMqtyJ0406kt4m+50aRJCSLWyf+mr\nN0OhSCLiA7wG3F/ctqr6PvA+uE4febcyY4zxnFSuRv3Ot0LnW0GVvLRNHFr3PfXiZzLs2DR8EqaQ\nvrM6C/Ni2V61C75N+9AuKpIujWtTs0olp8v/FW/2KXQFXlLV69zP/wSgqv9yP68B7AKy3C+5CjgM\nDCqqX8H6FIwx5cbJw+TGz+L4xh8J3LOAgJzj5KgPqzWK+XmxJNfpSVizWLo2DaZjRG2qBXhv9rmy\n0NHsh6ujuS+wD1dH892quqWQ7RcAv7eOZmPMFSk3B/atJnf7T5zZNoMqR7YDsFdDmJ8by0JtR1aD\nrnRo0oBuTYLp0KhWqU4u5HgouIu4AXgD1yWp41T1ZREZBaxW1e/O23YBFgrGmIriWArsnE3ujpmQ\nuADf3FOcoTJL81oxLzeWJbSnbngzujWpQ9fGdYgNr3lJc0eUiVDwBgsFY8wVJ/s0JC+B+Fnkxc/E\n52gSAMm+jZhxpi3zcmPZ4teCx/u04IneTUv0FhYKxhhTHqlCRoJ7itKZaPIyJC+H075VSWr9JC1u\n+VOJdlsW7lMwxhhzsUQguJnr0e1J5PRxSJxPQPwsWjRt7vW3t1AwxpiyLKA6tBrselwGV974sMYY\nY0rMQsEYY0w+CwVjjDH5LBSMMcbks1AwxhiTz0LBGGNMPgsFY4wx+SwUjDHG5Ct3w1yISDqQXMKX\nBwOHSrGc8s6Oxy/Z8fgfOxa/dCUcj0aqGlLcRuUuFC6FiKz2ZOyPisKOxy/Z8fgfOxa/VJGOh50+\nMsYYk89CwRhjTL6KFgrvO11AGWPH45fsePyPHYtfqjDHo0L1KRhjjClaRWspGGOMKYKFgjHGmHwV\nJhREZICI7BCRBBEZ6XQ9ThKRMBGZLyJbRWSLiDzjdE1OExFfEVknIj84XYvTRKSmiHwhIttFZJuI\ndHW6JqeIyLPu/yObRWSSiAQ4XZO3VYhQEBFfYCxwPdAKGCIirZytylE5wO9UtRXQBXiigh8PgGeA\nbU4XUUa8Cfykqi2AGCrocRGRhsDTQJyqtgF8gbucrcr7KkQoAJ2ABFVNVNWzwGTg8sxtVwapapqq\nrnX/nInrP31DZ6tyjoiEAjcCHzpdi9NEpAZwNfARgKqeVdWjzlblKD8gUET8gCpAqsP1eF1FCYWG\nwN4Cz1OowF+CBYlIBNAOWOFsJY56A/gjkOd0IWVAJJAOfOw+nfahiAQ5XZQTVHUf8CqwB0gDjqnq\nLGer8r6KEgrmAkSkKvAl8FtVPe50PU4QkZuAg6q6xulaygg/oD3wjqq2A04AFbIPTkRq4TqjEAk0\nAIJE5B5nq/K+ihIK+4CwAs9D3csqLBHxxxUIE1T1K6frcVB3YJCIJOE6rdhHRD53tiRHpQApqnqu\n5fgFrpCoiPoBu1U1XVWzga+Abg7X5HUVJRRWAc1EJFJEKuHqLPrO4ZocIyKC65zxNlV9zel6nKSq\nf1LVUFWNwPXvYp6qXvG/DRZGVfcDe0Ukyr2oL7DVwZKctAfoIiJV3P9n+lIBOt39nC7gclDVHBF5\nEpiJ6wqCcaq6xeGynNQduBfYJCLr3cv+rKrTHazJlB1PARPcv0AlAg84XI8jVHWFiHwBrMV1xd46\nKsBwFzbMhTHGmHwV5fSRMcYYD1goGGOMyWehYIwxJp+FgjHGmHwWCsYYY/JZKBhzHhHJFZH1BR6l\ndkeviESIyObS2p8xpa1C3KdgzEU6paqxThdhjBOspWCMh0QkSUT+IyKbRGSliDR1L48QkXkislFE\n5opIuHt5PRH5WkQ2uB/nhkjwFZEP3OP0zxKRQMc+lDHnsVAw5tcCzzt9dGeBdcdUNRp4C9foqgD/\nBT5R1bbABGCMe/kYYKGqxuAaP+jcXfTNgLGq2ho4Ctzq5c9jjMfsjmZjziMiWapa9QLLk4A+qpro\nHlBwv6rWEZFDQH1VzXYvT1PVYBFJB0JV9UyBfUQAs1W1mfv584C/qv7T+5/MmOJZS8GYi6OF/Hwx\nzhT4ORfr2zNliIWCMRfnzgJ//uz+eRn/m6ZxKLDY/fNc4DHInwO6xuUq0piSst9QjPm1wAKjx4Jr\nvuJzl6XWEpGNuH7bH+Je9hSumcr+gGvWsnOjij4DvC8iD+JqETyGawYvY8os61MwxkPuPoU4VT3k\ndC3GeIudPjLGGJPPWgrGGGPyWUvBGGNMPgsFY4wx+SwUjDHG5LNQMMYYk89CwRhjTL7/D/Sw/cg4\njAKaAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"code","metadata":{"colab_type":"code","id":"kETVyzl4tJhv","outputId":"b3e70cd8-c9f7-4093-f405-dc2e754c08ba","colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["speed_up = int(100 * fp32_results[\"train_time\"]/mp_results[\"train_time\"]) - 100\n","\n","print(\"Performance Improvement:\", speed_up, \"%\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Performance Improvement: 57 %\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"78HBT9cQXJko"},"source":["## Conclusions\n","\n","* Mixed Precision training provides a significant speed-up over FP32 (single-precision) training without affecting final accuracy. \n","* Switch to using mixed precision by simply wrapping a `tf.keras.optimizers` Optimizer in `tf.train.experimental.enable_mixed_precision_graph_rewrite()`\n"]}]} |