{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "wJcYs_ERTnnI" }, "source": [ "##### Copyright 2021 The TensorFlow Authors." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "execution": { "iopub.execute_input": "2022-03-02T13:22:33.508596Z", "iopub.status.busy": "2022-03-02T13:22:33.507917Z", "iopub.status.idle": "2022-03-02T13:22:33.512450Z", "shell.execute_reply": "2022-03-02T13:22:33.511887Z" }, "id": "HMUDt0CiUJk9" }, "outputs": [], "source": [ "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "77z2OchJTk0l" }, "source": [ "# Migrate from TPU embedding_columns to TPUEmbedding layer\n", "\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " View on TensorFlow.org\n", " \n", " \n", " \n", " Run in Google Colab\n", " \n", " \n", " \n", " View source on GitHub\n", " \n", " Download notebook\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "meUTrR4I6m1C" }, "source": [ "This guide demonstrates how to migrate embedding training on on [TPUs](../../guide/tpu.ipynb) from TensorFlow 1's `embedding_column` API with `TPUEstimator` to TensorFlow 2's `TPUEmbedding` layer API with `TPUStrategy`.\n", "\n", "Embeddings are (large) matrices. They are lookup tables that map from a sparse feature space to dense vectors. Embeddings provide efficient and dense representations, capturing complex similarities and relationships between features.\n", "\n", "TensorFlow includes specialized support for training embeddings on TPUs. This TPU-specific embedding support allows you to train embeddings that are larger than the memory of a single TPU device, and to use sparse and ragged inputs on TPUs.\n", "\n", "- In TensorFlow 1, `tf.compat.v1.estimator.tpu.TPUEstimator` is a high level API that encapsulates training, evaluation, prediction, and exporting for serving with TPUs. It has special support for `tf.compat.v1.tpu.experimental.embedding_column`.\n", "- To implement this in TensorFlow 2, use the TensorFlow Recommenders' `tfrs.layers.embedding.TPUEmbedding` layer. For training and evaluation, use a TPU distribution strategy—`tf.distribute.TPUStrategy`—which is compatible with the Keras APIs for, for example, model building (`tf.keras.Model`), optimizers (`tf.keras.optimizers.Optimizer`), and training with `Model.fit` or a custom training loop with `tf.function` and `tf.GradientTape`.\n", "\n", "For additional information, refer to the `tfrs.layers.embedding.TPUEmbedding` layer's API documentation, as well as the `tf.tpu.experimental.embedding.TableConfig` and `tf.tpu.experimental.embedding.FeatureConfig` docs for additional information. For an overview of `tf.distribute.TPUStrategy`, check out the [Distributed training](../../guide/distributed_training.ipynb) guide and the [Use TPUs](../../guide/tpu.ipynb) guide. If you're migrating from `TPUEstimator` to `TPUStrategy`, check out [the TPU migration guide](tpu_estimator.ipynb)." ] }, { "cell_type": "markdown", "metadata": { "id": "YdZSoIXEbhg-" }, "source": [ "## Setup\n", "\n", "Start by installing [TensorFlow Recommenders](https://www.tensorflow.org/recommenders) and importing some necessary packages:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:33.521851Z", "iopub.status.busy": "2022-03-02T13:22:33.521120Z", "iopub.status.idle": "2022-03-02T13:22:35.285517Z", "shell.execute_reply": "2022-03-02T13:22:35.284906Z" }, "id": "tYE3RnRN2jNu" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting tensorflow-recommenders\r\n", " Downloading tensorflow_recommenders-0.6.0-py3-none-any.whl (85 kB)\r\n", "\u001b[?25l\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r\n", "\u001b[?25hRequirement already satisfied: absl-py>=0.1.6 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow-recommenders) (0.12.0)\r\n", "Requirement already satisfied: tensorflow>=2.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow-recommenders) (2.6.2)\r\n", "Requirement already satisfied: six in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from absl-py>=0.1.6->tensorflow-recommenders) (1.15.0)\r\n", "Requirement already satisfied: tensorflow-estimator<2.7,>=2.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (2.6.0)\r\n", "Requirement already satisfied: termcolor~=1.1.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.1.0)\r\n", "Requirement already satisfied: astunparse~=1.6.3 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.6.3)\r\n", "Requirement already satisfied: clang~=5.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (5.0)\r\n", "Requirement already satisfied: wheel~=0.35 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (0.37.1)\r\n", "Requirement already satisfied: keras-preprocessing~=1.1.2 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.1.2)\r\n", "Requirement already satisfied: wrapt~=1.12.1 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.12.1)\r\n", "Requirement already satisfied: keras<2.7,>=2.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (2.6.0)\r\n", "Requirement already satisfied: grpcio<2.0,>=1.37.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.44.0)\r\n", "Requirement already satisfied: h5py~=3.1.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (3.1.0)\r\n", "Requirement already satisfied: numpy~=1.19.2 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.19.5)\r\n", "Requirement already satisfied: google-pasta~=0.2 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (0.2.0)\r\n", "Requirement already satisfied: opt-einsum~=3.3.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (3.3.0)\r\n", "Requirement already satisfied: tensorboard<2.7,>=2.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (2.6.0)\r\n", "Requirement already satisfied: flatbuffers~=1.12.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (1.12)\r\n", "Requirement already satisfied: gast==0.4.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (0.4.0)\r\n", "Requirement already satisfied: typing-extensions~=3.7.4 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (3.7.4.3)\r\n", "Requirement already satisfied: protobuf>=3.9.2 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorflow>=2.6.0->tensorflow-recommenders) (3.19.4)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: cached-property in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from h5py~=3.1.0->tensorflow>=2.6.0->tensorflow-recommenders) (1.5.2)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: google-auth<2,>=1.6.3 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (1.35.0)\r\n", "Requirement already satisfied: requests<3,>=2.21.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (2.27.1)\r\n", "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (0.6.1)\r\n", "Requirement already satisfied: markdown>=2.6.8 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (3.3.6)\r\n", "Requirement already satisfied: setuptools>=41.0.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (59.6.0)\r\n", "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (1.8.1)\r\n", "Requirement already satisfied: werkzeug>=0.11.15 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (2.0.3)\r\n", "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (0.4.6)\r\n", "Requirement already satisfied: rsa<5,>=3.1.4 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (4.8)\r\n", "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/kbuilder/.local/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (4.2.4)\r\n", "Requirement already satisfied: pyasn1-modules>=0.2.1 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (0.2.8)\r\n", "Requirement already satisfied: requests-oauthlib>=0.7.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (1.3.1)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: importlib-metadata>=4.4 in /home/kbuilder/.local/lib/python3.6/site-packages (from markdown>=2.6.8->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (4.8.3)\r\n", "Requirement already satisfied: certifi>=2017.4.17 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (2021.10.8)\r\n", "Requirement already satisfied: idna<4,>=2.5 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (3.3)\r\n", "Requirement already satisfied: charset-normalizer~=2.0.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (2.0.12)\r\n", "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (1.26.8)\r\n", "Requirement already satisfied: dataclasses in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from werkzeug>=0.11.15->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (0.8)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: zipp>=0.5 in /home/kbuilder/.local/lib/python3.6/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (3.6.0)\r\n", "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (0.4.8)\r\n", "Requirement already satisfied: oauthlib>=3.0.0 in /tmpfs/src/tf_docs_env/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.7,>=2.6.0->tensorflow>=2.6.0->tensorflow-recommenders) (3.2.0)\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Installing collected packages: tensorflow-recommenders\r\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Successfully installed tensorflow-recommenders-0.6.0\r\n" ] } ], "source": [ "!pip install tensorflow-recommenders" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:35.290342Z", "iopub.status.busy": "2022-03-02T13:22:35.289724Z", "iopub.status.idle": "2022-03-02T13:22:37.381808Z", "shell.execute_reply": "2022-03-02T13:22:37.381184Z" }, "id": "iE0vSfMXumKI" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/tmpfs/src/tf_docs_env/lib/python3.6/site-packages/requests/__init__.py:104: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (2.3.0)/charset_normalizer (2.0.12) doesn't match a supported version!\n", " RequestsDependencyWarning)\n" ] } ], "source": [ "import tensorflow as tf\n", "import tensorflow.compat.v1 as tf1\n", "\n", "# TPUEmbedding layer is not part of TensorFlow.\n", "import tensorflow_recommenders as tfrs" ] }, { "cell_type": "markdown", "metadata": { "id": "Jsm9Rxx7s1OZ" }, "source": [ "And prepare a simple dataset for demonstration purposes:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.387601Z", "iopub.status.busy": "2022-03-02T13:22:37.386981Z", "iopub.status.idle": "2022-03-02T13:22:37.388986Z", "shell.execute_reply": "2022-03-02T13:22:37.388539Z" }, "id": "m7rnGxsXtDkV" }, "outputs": [], "source": [ "features = [[1., 1.5]]\n", "embedding_features_indices = [[0, 0], [0, 1]]\n", "embedding_features_values = [0, 5]\n", "labels = [[0.3]]\n", "eval_features = [[4., 4.5]]\n", "eval_embedding_features_indices = [[0, 0], [0, 1]]\n", "eval_embedding_features_values = [4, 3]\n", "eval_labels = [[0.8]]" ] }, { "cell_type": "markdown", "metadata": { "id": "4uXff1BEssdE" }, "source": [ "## TensorFlow 1: Train embeddings on TPUs with TPUEstimator" ] }, { "cell_type": "markdown", "metadata": { "id": "pc-WSeYG2oje" }, "source": [ "In TensorFlow 1, you set up TPU embeddings using the `tf.compat.v1.tpu.experimental.embedding_column` API and train/evaluate the model on TPUs with `tf.compat.v1.estimator.tpu.TPUEstimator`.\n", "\n", "The inputs are integers ranging from zero to the vocabulary size for the TPU embedding table. Begin with encoding the inputs to categorical ID with `tf.feature_column.categorical_column_with_identity`. Use `\"sparse_feature\"` for the `key` parameter, since the input features are integer-valued, while `num_buckets` is the vocabulary size for the embedding table (`10`)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.392949Z", "iopub.status.busy": "2022-03-02T13:22:37.392264Z", "iopub.status.idle": "2022-03-02T13:22:37.394379Z", "shell.execute_reply": "2022-03-02T13:22:37.393908Z" }, "id": "sO_y-IRT3dcM" }, "outputs": [], "source": [ "embedding_id_column = (\n", " tf1.feature_column.categorical_column_with_identity(\n", " key=\"sparse_feature\", num_buckets=10))" ] }, { "cell_type": "markdown", "metadata": { "id": "57e2dec8ed4a" }, "source": [ "Next, convert the sparse categorical inputs to a dense representation with `tpu.experimental.embedding_column`, where `dimension` is the width of the embedding table. It will store an embedding vector for each of the `num_buckets`." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.397784Z", "iopub.status.busy": "2022-03-02T13:22:37.397233Z", "iopub.status.idle": "2022-03-02T13:22:37.398866Z", "shell.execute_reply": "2022-03-02T13:22:37.399224Z" }, "id": "6d61c855011f" }, "outputs": [], "source": [ "embedding_column = tf1.tpu.experimental.embedding_column(\n", " embedding_id_column, dimension=5)" ] }, { "cell_type": "markdown", "metadata": { "id": "c6061452ee5a" }, "source": [ "Now, define the TPU-specific embedding configuration via `tf.estimator.tpu.experimental.EmbeddingConfigSpec`. You will pass it later to `tf.estimator.tpu.TPUEstimator` as an `embedding_config_spec` parameter." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.403432Z", "iopub.status.busy": "2022-03-02T13:22:37.402567Z", "iopub.status.idle": "2022-03-02T13:22:37.941886Z", "shell.execute_reply": "2022-03-02T13:22:37.942560Z" }, "id": "6abbf967fc82" }, "outputs": [], "source": [ "embedding_config_spec = tf1.estimator.tpu.experimental.EmbeddingConfigSpec(\n", " feature_columns=(embedding_column,),\n", " optimization_parameters=(\n", " tf1.tpu.experimental.AdagradParameters(0.05)))" ] }, { "cell_type": "markdown", "metadata": { "id": "BVWHEQj5a7rN" }, "source": [ "Next, to use a `TPUEstimator`, define: \n", "- An input function for the training data\n", "- An evaluation input function for the evaluation data\n", "- A model function for instructing the `TPUEstimator` how the training op is defined with the features and labels" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.953238Z", "iopub.status.busy": "2022-03-02T13:22:37.952398Z", "iopub.status.idle": "2022-03-02T13:22:37.953996Z", "shell.execute_reply": "2022-03-02T13:22:37.954380Z" }, "id": "lqe9obf7suIj" }, "outputs": [], "source": [ "def _input_fn(params):\n", " dataset = tf1.data.Dataset.from_tensor_slices((\n", " {\"dense_feature\": features,\n", " \"sparse_feature\": tf1.SparseTensor(\n", " embedding_features_indices,\n", " embedding_features_values, [1, 2])},\n", " labels))\n", " dataset = dataset.repeat()\n", " return dataset.batch(params['batch_size'], drop_remainder=True)\n", "\n", "def _eval_input_fn(params):\n", " dataset = tf1.data.Dataset.from_tensor_slices((\n", " {\"dense_feature\": eval_features,\n", " \"sparse_feature\": tf1.SparseTensor(\n", " eval_embedding_features_indices,\n", " eval_embedding_features_values, [1, 2])},\n", " eval_labels))\n", " dataset = dataset.repeat()\n", " return dataset.batch(params['batch_size'], drop_remainder=True)\n", "\n", "def _model_fn(features, labels, mode, params):\n", " embedding_features = tf1.keras.layers.DenseFeatures(embedding_column)(features)\n", " concatenated_features = tf1.keras.layers.Concatenate(axis=1)(\n", " [embedding_features, features[\"dense_feature\"]])\n", " logits = tf1.layers.Dense(1)(concatenated_features)\n", " loss = tf1.losses.mean_squared_error(labels=labels, predictions=logits)\n", " optimizer = tf1.train.AdagradOptimizer(0.05)\n", " optimizer = tf1.tpu.CrossShardOptimizer(optimizer)\n", " train_op = optimizer.minimize(loss, global_step=tf1.train.get_global_step())\n", " return tf1.estimator.tpu.TPUEstimatorSpec(mode, loss=loss, train_op=train_op)" ] }, { "cell_type": "markdown", "metadata": { "id": "QYnP3Dszc-2R" }, "source": [ "With those functions defined, create a `tf.distribute.cluster_resolver.TPUClusterResolver` that provides the cluster information, and a `tf.compat.v1.estimator.tpu.RunConfig` object.\n", "\n", "Along with the model function you have defined, you can now create a `TPUEstimator`. Here, you will simplify the flow by skipping checkpoint savings. Then, you will specify the batch size for both training and evaluation for the `TPUEstimator`." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:37.996438Z", "iopub.status.busy": "2022-03-02T13:22:37.974511Z", "iopub.status.idle": "2022-03-02T13:22:38.000394Z", "shell.execute_reply": "2022-03-02T13:22:37.999781Z" }, "id": "WAqyqawemlcl" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "All devices: []\n" ] } ], "source": [ "cluster_resolver = tf1.distribute.cluster_resolver.TPUClusterResolver(tpu='')\n", "print(\"All devices: \", tf1.config.list_logical_devices('TPU'))" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:38.011309Z", "iopub.status.busy": "2022-03-02T13:22:38.006014Z", "iopub.status.idle": "2022-03-02T13:22:38.068062Z", "shell.execute_reply": "2022-03-02T13:22:38.067499Z" }, "id": "HsOpjW5plH9Q" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:Estimator's model_fn () includes params argument, but params are not passed to Estimator.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpxc9fm1_q\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpxc9fm1_q', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n", "cluster_def {\n", " job {\n", " name: \"worker\"\n", " tasks {\n", " key: 0\n", " value: \"10.240.1.10:8470\"\n", " }\n", " }\n", "}\n", "isolate_session_state: true\n", ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_checkpoint_save_graph_def': True, '_service': None, '_cluster_spec': ClusterSpec({'worker': ['10.240.1.10:8470']}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.240.1.10:8470', '_evaluation_master': 'grpc://10.240.1.10:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=10, num_shards=None, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2, experimental_host_call_every_n_steps=1, experimental_allow_per_host_v2_parallel_get_next=False, experimental_feed_hook=None), '_cluster': }\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:_TPUContext: eval_on_tpu True\n" ] } ], "source": [ "tpu_config = tf1.estimator.tpu.TPUConfig(\n", " iterations_per_loop=10,\n", " per_host_input_for_training=tf1.estimator.tpu.InputPipelineConfig\n", " .PER_HOST_V2)\n", "config = tf1.estimator.tpu.RunConfig(\n", " cluster=cluster_resolver,\n", " save_checkpoints_steps=None,\n", " tpu_config=tpu_config)\n", "estimator = tf1.estimator.tpu.TPUEstimator(\n", " model_fn=_model_fn, config=config, train_batch_size=8, eval_batch_size=8,\n", " embedding_config_spec=embedding_config_spec)" ] }, { "cell_type": "markdown", "metadata": { "id": "Uxw7tWrcepaZ" }, "source": [ "Call `TPUEstimator.train` to begin training the model:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:38.072868Z", "iopub.status.busy": "2022-03-02T13:22:38.072107Z", "iopub.status.idle": "2022-03-02T13:22:48.458701Z", "shell.execute_reply": "2022-03-02T13:22:48.459157Z" }, "id": "WZPKFOMAcyrP" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Querying Tensorflow master (grpc://10.240.1.10:8470) for TPU system metadata.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Found TPU system:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Workers: 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 6349538157198932596)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 9059152445598865227)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 3922455949451878923)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, -6084187114011162725)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 8400191476321474241)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 5484621084550964852)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, -8416772895681308377)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 2490716523526845408)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7973779273400954871)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 6478654019570154047)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 7299189593611257732)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/tpu/feature_column_v2.py:479: IdentityCategoricalColumn._num_buckets (from tensorflow.python.feature_column.feature_column_v2) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "The old _FeatureColumn APIs are being deprecated. Please use the new FeatureColumn APIs instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Querying Tensorflow master (grpc://10.240.1.10:8470) for TPU system metadata.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Found TPU system:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Workers: 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 6349538157198932596)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 9059152445598865227)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 3922455949451878923)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, -6084187114011162725)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 8400191476321474241)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 5484621084550964852)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, -8416772895681308377)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 2490716523526845408)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7973779273400954871)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 6478654019570154047)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 7299189593611257732)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow/python/training/adagrad.py:77: calling Constant.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Call initializer instance with the dtype argument instead of passing it to the constructor\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Bypassing TPUEstimator hook\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:TPU job name worker\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:758: Variable.load (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Prefer Variable.assign which has equivalent behavior in 2.X.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initialized dataset iterators in 0 seconds\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Installing graceful shutdown hook.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Init TPU system\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initialized TPU in 8 seconds\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting infeed thread controller.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting outfeed thread controller.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Outfeed finished for iteration (0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:loss = 0.6467617, step = 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Stop infeed thread controller\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutting down InfeedController thread.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:InfeedController received shutdown signal, stopping.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Infeed thread finished, shutting down.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:infeed marked as finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Stop output thread controller\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutting down OutfeedController thread.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:OutfeedController received shutdown signal, stopping.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Outfeed thread finished, shutting down.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:outfeed marked as finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutdown TPU system.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Loss for final step: 0.6467617.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:training_loop marked as finished\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "estimator.train(_input_fn, steps=1)" ] }, { "cell_type": "markdown", "metadata": { "id": "ev1vjIz9euIw" }, "source": [ "Then, call `TPUEstimator.evaluate` to evaluate the model using the evaluation data:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:22:48.463431Z", "iopub.status.busy": "2022-03-02T13:22:48.462697Z", "iopub.status.idle": "2022-03-02T13:23:00.832627Z", "shell.execute_reply": "2022-03-02T13:23:00.832039Z" }, "id": "bqiKRiwWc0cz" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Could not find trained model in model_dir: /tmp/tmpxc9fm1_q, running initialization to evaluate.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Querying Tensorflow master (grpc://10.240.1.10:8470) for TPU system metadata.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Found TPU system:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Workers: 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 6349538157198932596)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 9059152445598865227)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 3922455949451878923)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, -6084187114011162725)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 8400191476321474241)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 5484621084550964852)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, -8416772895681308377)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 2490716523526845408)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 7973779273400954871)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 6478654019570154047)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 7299189593611257732)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py:3406: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Deprecated in favor of operator or tf.math.divide.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done calling model_fn.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting evaluation at 2022-03-02T13:22:48\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:TPU job name worker\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Graph was finalized.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Done running local_init_op.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Init TPU system\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initialized TPU in 11 seconds\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting infeed thread controller.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Starting outfeed thread controller.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initialized dataset iterators in 0 seconds\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Outfeed finished for iteration (0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Evaluation [1/1]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Stop infeed thread controller\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutting down InfeedController thread.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:InfeedController received shutdown signal, stopping.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Infeed thread finished, shutting down.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:infeed marked as finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Stop output thread controller\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutting down OutfeedController thread.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:OutfeedController received shutdown signal, stopping.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Outfeed thread finished, shutting down.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:outfeed marked as finished\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Shutdown TPU system.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Inference Time : 12.06464s\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Finished evaluation at 2022-03-02-13:23:00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Saving dict for global step 1: global_step = 1, loss = 0.16138805\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:evaluation_loop marked as finished\n" ] }, { "data": { "text/plain": [ "{'loss': 0.16138805, 'global_step': 1}" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "estimator.evaluate(_eval_input_fn, steps=1)" ] }, { "cell_type": "markdown", "metadata": { "id": "KEmzBjfnsxwT" }, "source": [ "## TensorFlow 2: Train embeddings on TPUs with TPUStrategy" ] }, { "cell_type": "markdown", "metadata": { "id": "UesuXNbShrbi" }, "source": [ "In TensorFlow 2, to train on the TPU workers, use `tf.distribute.TPUStrategy` together with the Keras APIs for model definition and training/evaluation. (Refer to the [Use TPUs](https://render.githubusercontent.com/guide/tpu.ipynb) guide for more examples of training with Keras Model.fit and a custom training loop (with `tf.function` and `tf.GradientTape`).)\n", "\n", "Since you need to perform some initialization work to connect to the remote cluster and initialize the TPU workers, start by creating a `TPUClusterResolver` to provide the cluster information and connect to the cluster. (Learn more in the *TPU initialization* section of the [Use TPUs](../../guide/tpu.ipynb) guide.)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:23:00.878464Z", "iopub.status.busy": "2022-03-02T13:23:00.877312Z", "iopub.status.idle": "2022-03-02T13:23:13.652851Z", "shell.execute_reply": "2022-03-02T13:23:13.653280Z" }, "id": "_TgdPNgXoS63" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Clearing out eager caches\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Clearing out eager caches\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Initializing the TPU system: grpc://10.240.1.10:8470\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Initializing the TPU system: grpc://10.240.1.10:8470\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Finished initializing TPU system.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Finished initializing TPU system.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "All devices: [LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:0', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:1', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:2', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:3', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:4', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:5', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:6', device_type='TPU'), LogicalDevice(name='/job:worker/replica:0/task:0/device:TPU:7', device_type='TPU')]\n" ] } ], "source": [ "cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='')\n", "tf.config.experimental_connect_to_cluster(cluster_resolver)\n", "tf.tpu.experimental.initialize_tpu_system(cluster_resolver)\n", "print(\"All devices: \", tf.config.list_logical_devices('TPU'))" ] }, { "cell_type": "markdown", "metadata": { "id": "94JBD0HxmdPI" }, "source": [ "Next, prepare your data. This is similar to how you created a dataset in the TensorFlow 1 example, except the dataset function is now passed a `tf.distribute.InputContext` object rather than a `params` dict. You can use this object to determine the local batch size (and which host this pipeline is for, so you can properly partition your data).\n", "\n", "- When using the `tfrs.layers.embedding.TPUEmbedding` API, it is important to include the `drop_remainder=True` option when batching the dataset with `Dataset.batch`, since `TPUEmbedding` requires a fixed batch size.\n", "- Additionally, the same batch size must be used for evaluation and training if they are taking place on the same set of devices.\n", "- Finally, you should use `tf.keras.utils.experimental.DatasetCreator` along with the special input option—`experimental_fetch_to_device=False`—in `tf.distribute.InputOptions` (which holds strategy-specific configurations). This is demonstrated below:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:23:13.662236Z", "iopub.status.busy": "2022-03-02T13:23:13.661398Z", "iopub.status.idle": "2022-03-02T13:23:13.663091Z", "shell.execute_reply": "2022-03-02T13:23:13.663548Z" }, "id": "9NTruOw6mcy9" }, "outputs": [], "source": [ "global_batch_size = 8\n", "\n", "def _input_dataset(context: tf.distribute.InputContext):\n", " dataset = tf.data.Dataset.from_tensor_slices((\n", " {\"dense_feature\": features,\n", " \"sparse_feature\": tf.SparseTensor(\n", " embedding_features_indices,\n", " embedding_features_values, [1, 2])},\n", " labels))\n", " dataset = dataset.shuffle(10).repeat()\n", " dataset = dataset.batch(\n", " context.get_per_replica_batch_size(global_batch_size),\n", " drop_remainder=True)\n", " return dataset.prefetch(2)\n", "\n", "def _eval_dataset(context: tf.distribute.InputContext):\n", " dataset = tf.data.Dataset.from_tensor_slices((\n", " {\"dense_feature\": eval_features,\n", " \"sparse_feature\": tf.SparseTensor(\n", " eval_embedding_features_indices,\n", " eval_embedding_features_values, [1, 2])},\n", " eval_labels))\n", " dataset = dataset.repeat()\n", " dataset = dataset.batch(\n", " context.get_per_replica_batch_size(global_batch_size),\n", " drop_remainder=True)\n", " return dataset.prefetch(2)\n", "\n", "input_options = tf.distribute.InputOptions(\n", " experimental_fetch_to_device=False)\n", "\n", "input_dataset = tf.keras.utils.experimental.DatasetCreator(\n", " _input_dataset, input_options=input_options)\n", "\n", "eval_dataset = tf.keras.utils.experimental.DatasetCreator(\n", " _eval_dataset, input_options=input_options)" ] }, { "cell_type": "markdown", "metadata": { "id": "R4EHXhN3CVmo" }, "source": [ "Next, once the data is prepared, you will create a `TPUStrategy`, and define a model, metrics, and an optimizer under the scope of this strategy (`Strategy.scope`).\n", "\n", "You should pick a number for `steps_per_execution` in `Model.compile` since it specifies the number of batches to run during each `tf.function` call, and is critical for performance. This argument is similar to `iterations_per_loop` used in `TPUEstimator`.\n", "\n", "The features and table configuration that were specified in TensorFlow 1 via the `tf.tpu.experimental.embedding_column` (and `tf.tpu.experimental.shared_embedding_column`) can be specified directly in TensorFlow 2 via a pair of configuration objects:\n", "- `tf.tpu.experimental.embedding.FeatureConfig`\n", "- `tf.tpu.experimental.embedding.TableConfig`\n", "\n", "(Refer to the associated API documentation for more details.)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:23:13.671880Z", "iopub.status.busy": "2022-03-02T13:23:13.671159Z", "iopub.status.idle": "2022-03-02T13:23:15.615425Z", "shell.execute_reply": "2022-03-02T13:23:15.615935Z" }, "id": "atVciNgPs0fw" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:Found TPU system:\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Found TPU system:\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores: 8\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Workers: 1\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Workers: 1\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Num TPU Cores Per Worker: 8\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 0, 0)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 0, 0)\n" ] } ], "source": [ "strategy = tf.distribute.TPUStrategy(cluster_resolver)\n", "with strategy.scope():\n", " optimizer = tf.keras.optimizers.Adagrad(learning_rate=0.05)\n", " dense_input = tf.keras.Input(shape=(2,), dtype=tf.float32, batch_size=global_batch_size)\n", " sparse_input = tf.keras.Input(shape=(), dtype=tf.int32, batch_size=global_batch_size)\n", " embedded_input = tfrs.layers.embedding.TPUEmbedding(\n", " feature_config=tf.tpu.experimental.embedding.FeatureConfig(\n", " table=tf.tpu.experimental.embedding.TableConfig(\n", " vocabulary_size=10,\n", " dim=5,\n", " initializer=tf.initializers.TruncatedNormal(mean=0.0, stddev=1)),\n", " name=\"sparse_input\"),\n", " optimizer=optimizer)(sparse_input)\n", " input = tf.keras.layers.Concatenate(axis=1)([dense_input, embedded_input])\n", " result = tf.keras.layers.Dense(1)(input)\n", " model = tf.keras.Model(inputs={\"dense_feature\": dense_input, \"sparse_feature\": sparse_input}, outputs=result)\n", " model.compile(optimizer, \"mse\", steps_per_execution=10)" ] }, { "cell_type": "markdown", "metadata": { "id": "FkM2VZyni98F" }, "source": [ "With that, you are ready to train the model with the training dataset:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:23:15.621814Z", "iopub.status.busy": "2022-03-02T13:23:15.621065Z", "iopub.status.idle": "2022-03-02T13:23:18.189860Z", "shell.execute_reply": "2022-03-02T13:23:18.189365Z" }, "id": "Kip65sYBlKiu" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/5\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "10/10 [==============================] - ETA: 0s - loss: 0.0057" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/10 [==============================] - 2s 175ms/step - loss: 0.0057\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 2/5\n", "\r", "10/10 [==============================] - ETA: 0s - loss: 0.0000e+00" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/10 [==============================] - 0s 3ms/step - loss: 0.0000e+00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 3/5\n", "\r", "10/10 [==============================] - ETA: 0s - loss: 0.0000e+00" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/10 [==============================] - 0s 3ms/step - loss: 0.0000e+00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 4/5\n", "\r", "10/10 [==============================] - ETA: 0s - loss: 0.0000e+00" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/10 [==============================] - 0s 3ms/step - loss: 0.0000e+00\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 5/5\n", "\r", "10/10 [==============================] - ETA: 0s - loss: 0.0000e+00" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "10/10 [==============================] - 0s 3ms/step - loss: 0.0000e+00\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.fit(input_dataset, epochs=5, steps_per_epoch=10)" ] }, { "cell_type": "markdown", "metadata": { "id": "r0AEK8sNjLOj" }, "source": [ "Finally, evaluate the model using the evaluation dataset:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "execution": { "iopub.execute_input": "2022-03-02T13:23:18.194710Z", "iopub.status.busy": "2022-03-02T13:23:18.194107Z", "iopub.status.idle": "2022-03-02T13:23:19.553551Z", "shell.execute_reply": "2022-03-02T13:23:19.552943Z" }, "id": "6tMRkyfKhqSL" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - ETA: 0s - loss: 12.2297" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "1/1 [==============================] - 1s 1s/step - loss: 12.2297\n" ] }, { "data": { "text/plain": [ "{'loss': 12.229663848876953}" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.evaluate(eval_dataset, steps=1, return_dict=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "a97b888c1911" }, "source": [ "## Next steps" ] }, { "cell_type": "markdown", "metadata": { "id": "gHx_RUL8xcJ3" }, "source": [ "Learn more about setting up TPU-specific embeddings in the API docs:\n", "\n", "- `tfrs.layers.embedding.TPUEmbedding`: particularly about feature and table configuration, setting the optimizer, creating a model (using the Keras [functional](https://www.tensorflow.org/guide/keras/functional) API or via [subclassing](../..guide/keras/custom_layers_and_models.ipynb) `tf.keras.Model`), training/evaluation, and model serving with `tf.saved_model`\n", "- `tf.tpu.experimental.embedding.TableConfig`\n", "- `tf.tpu.experimental.embedding.FeatureConfig`\n", "\n", "For more information about `TPUStrategy` in TensorFlow 2, consider the following resources:\n", "\n", "- Guide: [Use TPUs](../../guide/tpu.ipynb) (covering training with Keras `Model.fit`/a custom training loop with `tf.distribute.TPUStrategy`, as well as tips on improving the performance with `tf.function`)\n", "- Guide: [Distributed training with TensorFlow](../../guide/distributed_training.ipynb)\n", "- Guide: [Migrate from TPUEstimator to TPUStrategy](tpu_estimator.ipynb).\n", "\n", "To learn more about customizing your training, refer to:\n", "\n", "- Guide: [Customize what happens in Model.fit](../..guide/keras/customizing_what_happens_in_fit.ipynb)\n", "- Guide: [Writing a training loop from scratch](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch)\n", "\n", "TPUs—Google's specialized ASICs for machine learning—are available through [Google Colab](https://colab.research.google.com/), the [TPU Research Cloud](https://sites.research.google/trc/), and [Cloud TPU](https://cloud.google.com/tpu)." ] } ], "metadata": { "accelerator": "TPU", "colab": { "collapsed_sections": [], "name": "tpu_embedding.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 0 }