![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "NTIEyVwmLTBP"
+ },
+ "outputs": [],
+ "source": []
+ }
+ ]
+}
\ No newline at end of file
diff --git a/summary_be/archive/utils/lstm_bi25/fingerprint.pb b/summary_be/archive/utils/lstm_bi25/fingerprint.pb
new file mode 100644
index 0000000..70f8498
--- /dev/null
+++ b/summary_be/archive/utils/lstm_bi25/fingerprint.pb
@@ -0,0 +1 @@
+ȺƃXӍѹŰ ꋨW(猰(2
\ No newline at end of file
diff --git a/summary_be/archive/utils/lstm_bi25/keras_metadata.pb b/summary_be/archive/utils/lstm_bi25/keras_metadata.pb
new file mode 100644
index 0000000..e7d0857
--- /dev/null
+++ b/summary_be/archive/utils/lstm_bi25/keras_metadata.pb
@@ -0,0 +1,11 @@
+
+-root"_tf_keras_sequential*-{"name": "sequential", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": false, "class_name": "Sequential", "config": {"name": "sequential", "layers": [{"class_name": "InputLayer", "config": {"batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "dtype": "float32", "sparse": false, "ragged": false, "name": "bidirectional_input"}}, {"class_name": "Bidirectional", "config": {"name": "bidirectional", "trainable": true, "dtype": "float32", "batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "layer": {"class_name": "LSTM", "config": {"name": "lstm", "trainable": true, "dtype": "float32", "return_sequences": true, "return_state": false, "go_backwards": false, "stateful": false, "unroll": false, "time_major": false, "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 1}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 2}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 3}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}}, "merge_mode": "concat"}}, {"class_name": "TimeDistributed", "config": {"name": "time_distributed", "trainable": true, "dtype": "float32", "layer": {"class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 1, "activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}}, "bias_initializer": {"class_name": "Zeros", "config": {}}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}}}}]}, "shared_object_id": 11, "input_spec": [{"class_name": "InputSpec", "config": {"dtype": null, "shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "ndim": 3, "max_ndim": null, "min_ndim": null, "axes": {}}}], "build_input_shape": {"class_name": "TensorShape", "items": [null, null, 768]}, "is_graph_network": true, "full_save_spec": {"class_name": "__tuple__", "items": [[{"class_name": "TypeSpec", "type_spec": "tf.TensorSpec", "serialized": [{"class_name": "TensorShape", "items": [null, null, 768]}, "float32", "bidirectional_input"]}], {}]}, "save_spec": {"class_name": "TypeSpec", "type_spec": "tf.TensorSpec", "serialized": [{"class_name": "TensorShape", "items": [null, null, 768]}, "float32", "bidirectional_input"]}, "keras_version": "2.12.0", "backend": "tensorflow", "model_config": {"class_name": "Sequential", "config": {"name": "sequential", "layers": [{"class_name": "InputLayer", "config": {"batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "dtype": "float32", "sparse": false, "ragged": false, "name": "bidirectional_input"}, "shared_object_id": 0}, {"class_name": "Bidirectional", "config": {"name": "bidirectional", "trainable": true, "dtype": "float32", "batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "layer": {"class_name": "LSTM", "config": {"name": "lstm", "trainable": true, "dtype": "float32", "return_sequences": true, "return_state": false, "go_backwards": false, "stateful": false, "unroll": false, "time_major": false, "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 1}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 2}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 3}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 5}, "merge_mode": "concat"}, "shared_object_id": 6}, {"class_name": "TimeDistributed", "config": {"name": "time_distributed", "trainable": true, "dtype": "float32", "layer": {"class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 1, "activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 7}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 8}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "shared_object_id": 9}}, "shared_object_id": 10}]}}, "training_config": {"loss": "binary_crossentropy", "metrics": [[{"class_name": "SensitivityAtSpecificity", "config": {"name": "sensitivity_at_specificity", "dtype": "float32", "class_id": null, "num_thresholds": 1, "specificity": 0.5}, "shared_object_id": 13}]], "weighted_metrics": null, "loss_weights": null, "optimizer_config": {"class_name": "Custom>Adam", "config": {"name": "Adam", "weight_decay": null, "clipnorm": null, "global_clipnorm": null, "clipvalue": null, "use_ema": false, "ema_momentum": 0.99, "ema_overwrite_frequency": null, "jit_compile": true, "is_legacy_optimizer": false, "learning_rate": 0.0010000000474974513, "beta_1": 0.9, "beta_2": 0.999, "epsilon": 1e-07, "amsgrad": false}}}}2
+
root.layer_with_weights-0"_tf_keras_layer*
{"name": "bidirectional", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "Bidirectional", "config": {"name": "bidirectional", "trainable": true, "dtype": "float32", "batch_input_shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "layer": {"class_name": "LSTM", "config": {"name": "lstm", "trainable": true, "dtype": "float32", "return_sequences": true, "return_state": false, "go_backwards": false, "stateful": false, "unroll": false, "time_major": false, "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 1}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 2}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 3}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 5}, "merge_mode": "concat"}, "shared_object_id": 6, "input_spec": [{"class_name": "InputSpec", "config": {"dtype": null, "shape": null, "ndim": 3, "max_ndim": null, "min_ndim": null, "axes": {}}, "shared_object_id": 14}], "build_input_shape": {"class_name": "TensorShape", "items": [null, null, 768]}}2
+ root.layer_with_weights-1"_tf_keras_layer* {"name": "time_distributed", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "TimeDistributed", "config": {"name": "time_distributed", "trainable": true, "dtype": "float32", "layer": {"class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 1, "activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 7}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 8}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "shared_object_id": 9}}, "shared_object_id": 10, "input_spec": {"class_name": "InputSpec", "config": {"dtype": null, "shape": {"class_name": "__tuple__", "items": [null, null, 50]}, "ndim": 3, "max_ndim": null, "min_ndim": null, "axes": {}}, "shared_object_id": 15}, "build_input_shape": {"class_name": "TensorShape", "items": [null, null, 50]}}2
+ 'root.layer_with_weights-0.forward_layer"_tf_keras_rnn_layer* {"name": "forward_lstm", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "LSTM", "config": {"name": "forward_lstm", "trainable": true, "dtype": "float32", "return_sequences": true, "return_state": false, "go_backwards": false, "stateful": false, "unroll": false, "time_major": false, "zero_output_for_mask": true, "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 16}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 17}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 18}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 20, "input_spec": [{"class_name": "InputSpec", "config": {"dtype": null, "shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "ndim": 3, "max_ndim": null, "min_ndim": null, "axes": {}}, "shared_object_id": 21}], "build_input_shape": {"class_name": "TensorShape", "items": [null, null, 768]}}2
+ (root.layer_with_weights-0.backward_layer"_tf_keras_rnn_layer* {"name": "backward_lstm", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "LSTM", "config": {"name": "backward_lstm", "trainable": true, "dtype": "float32", "return_sequences": true, "return_state": false, "go_backwards": true, "stateful": false, "unroll": false, "time_major": false, "zero_output_for_mask": true, "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 22}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 23}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 24}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 26, "input_spec": [{"class_name": "InputSpec", "config": {"dtype": null, "shape": {"class_name": "__tuple__", "items": [null, null, 768]}, "ndim": 3, "max_ndim": null, "min_ndim": null, "axes": {}}, "shared_object_id": 27}], "build_input_shape": {"class_name": "TensorShape", "items": [null, null, 768]}}2
+ root.layer_with_weights-1.layer"_tf_keras_layer*{"name": "dense", "trainable": true, "expects_training_arg": false, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 1, "activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 7}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 8}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "shared_object_id": 9, "input_spec": {"class_name": "InputSpec", "config": {"dtype": null, "shape": null, "ndim": null, "max_ndim": null, "min_ndim": 2, "axes": {"-1": 50}}, "shared_object_id": 28}}2
+ L,root.layer_with_weights-0.forward_layer.cell"_tf_keras_layer*{"name": "lstm_cell_1", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "LSTMCell", "config": {"name": "lstm_cell_1", "trainable": true, "dtype": "float32", "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 16}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 17}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 18}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 19, "build_input_shape": {"class_name": "__tuple__", "items": [null, 768]}}2
+ U-root.layer_with_weights-0.backward_layer.cell"_tf_keras_layer*{"name": "lstm_cell_2", "trainable": true, "expects_training_arg": true, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "LSTMCell", "config": {"name": "lstm_cell_2", "trainable": true, "dtype": "float32", "units": 25, "activation": "tanh", "recurrent_activation": "sigmoid", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 22}, "recurrent_initializer": {"class_name": "Orthogonal", "config": {"gain": 1.0, "seed": null}, "shared_object_id": 23}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 24}, "unit_forget_bias": true, "kernel_regularizer": null, "recurrent_regularizer": null, "bias_regularizer": null, "kernel_constraint": null, "recurrent_constraint": null, "bias_constraint": null, "dropout": 0.0, "recurrent_dropout": 0.0, "implementation": 2}, "shared_object_id": 25, "build_input_shape": {"class_name": "__tuple__", "items": [null, 768]}}2
+froot.keras_api.metrics.0"_tf_keras_metric*{"class_name": "Mean", "name": "loss", "dtype": "float32", "config": {"name": "loss", "dtype": "float32"}, "shared_object_id": 29}2
+groot.keras_api.metrics.1"_tf_keras_metric*{"class_name": "SensitivityAtSpecificity", "name": "sensitivity_at_specificity", "dtype": "float32", "config": {"name": "sensitivity_at_specificity", "dtype": "float32", "class_id": null, "num_thresholds": 1, "specificity": 0.5}, "shared_object_id": 13}2
\ No newline at end of file
diff --git a/summary_be/archive/utils/lstm_bi25/saved_model.pb b/summary_be/archive/utils/lstm_bi25/saved_model.pb
new file mode 100644
index 0000000..18a6483
Binary files /dev/null and b/summary_be/archive/utils/lstm_bi25/saved_model.pb differ
diff --git a/summary_be/archive/utils/lstm_bi25/variables/variables.data-00000-of-00001 b/summary_be/archive/utils/lstm_bi25/variables/variables.data-00000-of-00001
new file mode 100644
index 0000000..6b33d28
Binary files /dev/null and b/summary_be/archive/utils/lstm_bi25/variables/variables.data-00000-of-00001 differ
diff --git a/summary_be/archive/utils/lstm_bi25/variables/variables.index b/summary_be/archive/utils/lstm_bi25/variables/variables.index
new file mode 100644
index 0000000..6d795c1
Binary files /dev/null and b/summary_be/archive/utils/lstm_bi25/variables/variables.index differ
diff --git a/summary_be/archive/utils/summarization.py b/summary_be/archive/utils/summarization.py
index ad87934..0cfd021 100644
--- a/summary_be/archive/utils/summarization.py
+++ b/summary_be/archive/utils/summarization.py
@@ -1,248 +1,50 @@
-# Actual summarization logic
-
-import nltk
-import re
-import math
-from nltk.stem import WordNetLemmatizer
-from nltk.tokenize import sent_tokenize, word_tokenize
-from nltk.corpus import stopwords
-
-nltk.data.path.append("/opt/python/nltk_data")
-
-INVALID = r"[^a-zA-Z\s]"
-STOP_WORDS = set(stopwords.words("english"))
-lemmatizer = WordNetLemmatizer()
-
-"""
- Function to clean and tokenize a chunk of text into words.
-
- Parameters
- ----------
- text : string
- - block of text to process
-
- Returns
- -------
- lemmatized_words : []
- - array of valid words
-"""
-
-
-def preprocess(text):
- # Remove invalid characters and digits
- text = re.sub(INVALID, "", text)
-
- # Remove stop words + convert to lower case
- words = word_tokenize(text)
- words = [word.lower() for word in words]
- stop_words_removed = [word for word in words if word not in STOP_WORDS]
-
- # Remove one character words
- valid_words = [word for word in stop_words_removed if len(word) > 1]
-
- # Lemmatize words, aka "builds" becomes "build" etc.
- lemmatized_words = [lemmatizer.lemmatize(word) for word in valid_words]
-
- return lemmatized_words
-
-
-"""
- Function to get frequency of words.
-
- Parameters
- ----------
- words : []
- - array of words
-
- Returns
- -------
- word_freq : {}
- - dictionary of word frequencies
- - key is a string
- - val is an int
-"""
-
-
-def get_word_freq(words):
- dict = {}
- unique_words = []
- for word in words:
- if word not in unique_words:
- unique_words.append(word)
- for word in unique_words:
- dict[word] = words.count(word)
- return dict
-
-
-"""
- Function to return sum of tf-idf score of words in a sentence.
-
- Parameters
- ----------
- sent : string
- - sentence containing word
- sentences : []
- - array of sentences in text
-
- Returns
- -------
- score : float
- - tf-idf score of sentence
-"""
-
-
-def tf_idf_score(sent, sentences):
- sent = re.sub(INVALID, "", sent)
- sent_len = len(word_tokenize(sent))
- words_in_given_sent = preprocess(sent)
- word_freq = get_word_freq(words_in_given_sent)
-
- score = 0
- for word in words_in_given_sent:
- tf = tf_score(word_freq[word], sent_len)
- idf = idf_score(word, sentences)
- tf_idf = tf * idf
- score += tf_idf
-
- return score
-
-
-"""
- Function to return term frequency score of a word.
-
- Parameters
- ----------
- freq : int
- - frequency of word in given sentence
- sent_len : int
- - total number of words in sentence
-
- Returns
- -------
- tf_score : float
- - tf score of word
-"""
-
-
-def tf_score(freq, sent_len):
- return freq / sent_len
-
-
-"""
- Function to return inverse document frequency of a word.
-
- Parameters
- ----------
- word : string
- - word to compute score
- setences : []
- - array of sentences in text
-
- Returns
- -------
- score : float
- - idf score of word
-"""
-
-
-def idf_score(word, sentences):
- num_sent_containing_word = 0
- for sent in sentences:
- words = preprocess(sent)
- if word in words:
- num_sent_containing_word += 1
- return math.log10(len(sentences) / num_sent_containing_word)
-
-
-"""
- Function to return sentence weights based on tf-idf score
-
- Parameters
- ----------
- sentences : []
- - array of sentences in text
-
- Returns
- -------
- sentence_weight : {}
- - dictionary of sentence weights
- - key is index of sentence in sentences
- - val is tf-idf score
-"""
-
-
-def get_sentence_weights(sentences):
- sentence_weight = {}
- for x in range(len(sentences)):
- sentence = sentences[x]
- importance = tf_idf_score(sentence, sentences)
- sentence_weight[x] = importance
- sentence_weight = dict(
- sorted(sentence_weight.items(), key=lambda item: item[1], reverse=True)
- )
- return sentence_weight
-
-
-"""
- Function to return indices of top scoring sentences
-
- Parameters
- ----------
- sentence_weight: {}
- - dictionary of sentence weights
- num_sent: int
- - number of top scoring sentences to return
-
- Returns
- -------
- sentence_idx : []
- - indices of top scoring sentences
-"""
-
-
-def get_top_scoring_sent(sentence_weight, num_sent):
- curr_num_sentences = 0
- sentence_idx = []
- for key in sentence_weight:
- if key == 0:
- continue
- if curr_num_sentences < num_sent - 1:
- sentence_idx.append(key)
- curr_num_sentences += 1
- else:
- break
- sentence_idx.sort()
- return sentence_idx
-
-
-"""
- Function to summarize a given text.
-
- Parameters
- ----------
- text : string
- - text to summarize
- num_sentences : int
- - number of sentences to include in summary
-
- Returns
- -------
- summary : string
- - extractive summary of text
-"""
-
-
-def summarize(text, num_sentences):
- if num_sentences < 1:
- raise Exception("Summary must have at least one sentence.")
- sentences = sent_tokenize(text)
- sentence_weight = get_sentence_weights(sentences)
- sentence_idx = get_top_scoring_sent(sentence_weight, num_sentences)
-
- summary = sentences[0]
- for x in range(len(sentences)):
- if x in sentence_idx:
- summary += " "
- summary += sentences[x]
-
- return summary
+import pandas as pd
+import numpy as np
+import spacy
+from spacy.language import Language
+import tensorflow as tf
+from tensorflow import keras
+from sentence_transformers import SentenceTransformer
+
+model = tf.keras.models.load_model('/lstm_bi25')
+
+def preprocess(text,
+ nlp = spacy.load("en_core_web_sm"),
+ embedder = SentenceTransformer('distilbert-base-nli-mean-tokens'),
+ min_len = 2):
+ @Language.component("custom_sentencizer")
+ def custom_sentencizer(doc):
+ for i, token in enumerate(doc[:-2]):
+ # Define sentence start if it occurs after "\n\n" or "\n"
+ if token.text == "\n\n" or token.text == "\n":
+ doc[i + 1].is_sent_start = True
+ return doc
+ nlp = spacy.load("en_core_web_sm")
+ nlp.add_pipe("custom_sentencizer", before="parser")
+ text = nlp(text)
+ sents = list(text.sents)
+
+ # remove sentences with length below threshold
+ sents_clean = [sent.text for sent in sents if len(sent) > min_len]
+ sents_clean = [sent for sent in sents_clean if len(sent)!=0]
+
+ # calculate sentence embeddings
+ sents_embedding= np.array(embedder.encode(sents_clean, convert_to_tensor=True))
+ return sents_clean, sents_embedding
+
+def summarize(text):
+ sentences, embeddings = preprocess(text)
+
+ reshape_func = lambda x: x.reshape(1, x.shape[0], x.shape[1])
+ x = reshape_func(embeddings)
+
+ y_pred = model.predict(x, verbose=0)
+ idx = np.argsort(y_pred.flatten())[-3:]
+ idx = sorted(idx)
+ pred_summary = ""
+ for i in idx:
+ pred_summary += ' '
+ pred_summary += sentences[i]
+ pred_summary.strip()
+
+ return pred_summary
\ No newline at end of file
diff --git a/summary_be/archive/utils/summarization_old.py b/summary_be/archive/utils/summarization_old.py
new file mode 100644
index 0000000..ad87934
--- /dev/null
+++ b/summary_be/archive/utils/summarization_old.py
@@ -0,0 +1,248 @@
+# Actual summarization logic
+
+import nltk
+import re
+import math
+from nltk.stem import WordNetLemmatizer
+from nltk.tokenize import sent_tokenize, word_tokenize
+from nltk.corpus import stopwords
+
+nltk.data.path.append("/opt/python/nltk_data")
+
+INVALID = r"[^a-zA-Z\s]"
+STOP_WORDS = set(stopwords.words("english"))
+lemmatizer = WordNetLemmatizer()
+
+"""
+ Function to clean and tokenize a chunk of text into words.
+
+ Parameters
+ ----------
+ text : string
+ - block of text to process
+
+ Returns
+ -------
+ lemmatized_words : []
+ - array of valid words
+"""
+
+
+def preprocess(text):
+ # Remove invalid characters and digits
+ text = re.sub(INVALID, "", text)
+
+ # Remove stop words + convert to lower case
+ words = word_tokenize(text)
+ words = [word.lower() for word in words]
+ stop_words_removed = [word for word in words if word not in STOP_WORDS]
+
+ # Remove one character words
+ valid_words = [word for word in stop_words_removed if len(word) > 1]
+
+ # Lemmatize words, aka "builds" becomes "build" etc.
+ lemmatized_words = [lemmatizer.lemmatize(word) for word in valid_words]
+
+ return lemmatized_words
+
+
+"""
+ Function to get frequency of words.
+
+ Parameters
+ ----------
+ words : []
+ - array of words
+
+ Returns
+ -------
+ word_freq : {}
+ - dictionary of word frequencies
+ - key is a string
+ - val is an int
+"""
+
+
+def get_word_freq(words):
+ dict = {}
+ unique_words = []
+ for word in words:
+ if word not in unique_words:
+ unique_words.append(word)
+ for word in unique_words:
+ dict[word] = words.count(word)
+ return dict
+
+
+"""
+ Function to return sum of tf-idf score of words in a sentence.
+
+ Parameters
+ ----------
+ sent : string
+ - sentence containing word
+ sentences : []
+ - array of sentences in text
+
+ Returns
+ -------
+ score : float
+ - tf-idf score of sentence
+"""
+
+
+def tf_idf_score(sent, sentences):
+ sent = re.sub(INVALID, "", sent)
+ sent_len = len(word_tokenize(sent))
+ words_in_given_sent = preprocess(sent)
+ word_freq = get_word_freq(words_in_given_sent)
+
+ score = 0
+ for word in words_in_given_sent:
+ tf = tf_score(word_freq[word], sent_len)
+ idf = idf_score(word, sentences)
+ tf_idf = tf * idf
+ score += tf_idf
+
+ return score
+
+
+"""
+ Function to return term frequency score of a word.
+
+ Parameters
+ ----------
+ freq : int
+ - frequency of word in given sentence
+ sent_len : int
+ - total number of words in sentence
+
+ Returns
+ -------
+ tf_score : float
+ - tf score of word
+"""
+
+
+def tf_score(freq, sent_len):
+ return freq / sent_len
+
+
+"""
+ Function to return inverse document frequency of a word.
+
+ Parameters
+ ----------
+ word : string
+ - word to compute score
+ setences : []
+ - array of sentences in text
+
+ Returns
+ -------
+ score : float
+ - idf score of word
+"""
+
+
+def idf_score(word, sentences):
+ num_sent_containing_word = 0
+ for sent in sentences:
+ words = preprocess(sent)
+ if word in words:
+ num_sent_containing_word += 1
+ return math.log10(len(sentences) / num_sent_containing_word)
+
+
+"""
+ Function to return sentence weights based on tf-idf score
+
+ Parameters
+ ----------
+ sentences : []
+ - array of sentences in text
+
+ Returns
+ -------
+ sentence_weight : {}
+ - dictionary of sentence weights
+ - key is index of sentence in sentences
+ - val is tf-idf score
+"""
+
+
+def get_sentence_weights(sentences):
+ sentence_weight = {}
+ for x in range(len(sentences)):
+ sentence = sentences[x]
+ importance = tf_idf_score(sentence, sentences)
+ sentence_weight[x] = importance
+ sentence_weight = dict(
+ sorted(sentence_weight.items(), key=lambda item: item[1], reverse=True)
+ )
+ return sentence_weight
+
+
+"""
+ Function to return indices of top scoring sentences
+
+ Parameters
+ ----------
+ sentence_weight: {}
+ - dictionary of sentence weights
+ num_sent: int
+ - number of top scoring sentences to return
+
+ Returns
+ -------
+ sentence_idx : []
+ - indices of top scoring sentences
+"""
+
+
+def get_top_scoring_sent(sentence_weight, num_sent):
+ curr_num_sentences = 0
+ sentence_idx = []
+ for key in sentence_weight:
+ if key == 0:
+ continue
+ if curr_num_sentences < num_sent - 1:
+ sentence_idx.append(key)
+ curr_num_sentences += 1
+ else:
+ break
+ sentence_idx.sort()
+ return sentence_idx
+
+
+"""
+ Function to summarize a given text.
+
+ Parameters
+ ----------
+ text : string
+ - text to summarize
+ num_sentences : int
+ - number of sentences to include in summary
+
+ Returns
+ -------
+ summary : string
+ - extractive summary of text
+"""
+
+
+def summarize(text, num_sentences):
+ if num_sentences < 1:
+ raise Exception("Summary must have at least one sentence.")
+ sentences = sent_tokenize(text)
+ sentence_weight = get_sentence_weights(sentences)
+ sentence_idx = get_top_scoring_sent(sentence_weight, num_sentences)
+
+ summary = sentences[0]
+ for x in range(len(sentences)):
+ if x in sentence_idx:
+ summary += " "
+ summary += sentences[x]
+
+ return summary
diff --git a/summary_be/lambda_handlers/summary.py b/summary_be/lambda_handlers/summary.py
index 93a79ab..a6c99d9 100644
--- a/summary_be/lambda_handlers/summary.py
+++ b/summary_be/lambda_handlers/summary.py
@@ -16,7 +16,7 @@ def get_summary(event, context):
}
try:
- result = summarize(text, num_sentences)
+ result = summarize(text)
num_words = len(result.split(" "))
return {
"statusCode": 200,
diff --git a/summary_be/ml_notebook/dynamic_chunking.ipynb b/summary_be/ml_notebook/dynamic_chunking.ipynb
deleted file mode 100644
index 7f3d88f..0000000
--- a/summary_be/ml_notebook/dynamic_chunking.ipynb
+++ /dev/null
@@ -1,634 +0,0 @@
-{
- "nbformat": 4,
- "nbformat_minor": 0,
- "metadata": {
- "colab": {
- "provenance": [],
- "authorship_tag": "ABX9TyNORzkxfdxKKcNI5yqF1enI",
- "include_colab_link": true
- },
- "kernelspec": {
- "name": "python3",
- "display_name": "Python 3"
- },
- "language_info": {
- "name": "python"
- }
- },
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "view-in-github",
- "colab_type": "text"
- },
- "source": [
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "# Dynamic Chunking\n",
- "This notebook implements dynamic chunking of text into semantically distinct paragraphs.\n",
- "\n",
- "* [SBERT](https://www.sbert.net/) is used to extract semantic feature vectors in sentences\n",
- "* Vectors can be compared to find sentences of similar meaning\n",
- "\n",
- "Dynamic chunking solves the following problems:\n",
- "\n",
- "\n",
- "* Summarization of long texts\n",
- "* Preservation of detail in the overall summary\n",
- "\n",
- "**TODO**\n",
- "* This notebook is not complete, missing examples and explanations\n",
- "* Compare time for each method\n"
- ],
- "metadata": {
- "id": "dTHsgGOLa8Vr"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "!pip install sentence_transformers"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "QewF-0LpSq1L",
- "outputId": "6a7b403a-e3ad-45e5-8d7b-7cbb71e0f7fe"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
- "Requirement already satisfied: sentence_transformers in /usr/local/lib/python3.7/dist-packages (2.2.2)\n",
- "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (4.64.1)\n",
- "Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (0.13.1+cu113)\n",
- "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (1.7.3)\n",
- "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (1.0.2)\n",
- "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (3.7)\n",
- "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (0.1.97)\n",
- "Requirement already satisfied: transformers<5.0.0,>=4.6.0 in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (4.24.0)\n",
- "Requirement already satisfied: huggingface-hub>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (0.10.1)\n",
- "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (1.21.6)\n",
- "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from sentence_transformers) (1.12.1+cu113)\n",
- "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (4.13.0)\n",
- "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (21.3)\n",
- "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (3.8.0)\n",
- "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (6.0)\n",
- "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (4.1.1)\n",
- "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from huggingface-hub>=0.4.0->sentence_transformers) (2.28.1)\n",
- "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.9->huggingface-hub>=0.4.0->sentence_transformers) (3.0.9)\n",
- "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (2022.6.2)\n",
- "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers<5.0.0,>=4.6.0->sentence_transformers) (0.13.2)\n",
- "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->huggingface-hub>=0.4.0->sentence_transformers) (3.10.0)\n",
- "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from nltk->sentence_transformers) (1.2.0)\n",
- "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from nltk->sentence_transformers) (7.1.2)\n",
- "Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2.1.1)\n",
- "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2.10)\n",
- "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (2022.9.24)\n",
- "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->huggingface-hub>=0.4.0->sentence_transformers) (1.24.3)\n",
- "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sentence_transformers) (3.1.0)\n",
- "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision->sentence_transformers) (7.1.2)\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "import pandas as pd\n",
- "import numpy as np\n",
- "import math\n",
- "import nltk\n",
- "nltk.download('punkt')\n",
- "from nltk import sent_tokenize\n",
- "from tqdm import tnrange\n",
- "from sentence_transformers import SentenceTransformer\n",
- "from sklearn.metrics.pairwise import cosine_similarity\n",
- "import seaborn as sns\n",
- "from scipy.signal import argrelextrema\n",
- "import matplotlib.pyplot as plt"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "q0VfRJ_sTNQ6",
- "outputId": "f4b03fce-2903-42b6-94f3-96a7d6d388fc"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stderr",
- "text": [
- "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
- "[nltk_data] Package punkt is already up-to-date!\n"
- ]
- }
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "MODEL = 'all-MiniLM-L6-v2'\n",
- "model = SentenceTransformer(MODEL)"
- ],
- "metadata": {
- "id": "ouSiySJaTOnJ"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "TEXT = \"\"\"Elon Musk sold $3.95 billion worth of Tesla stock since completing his purchase of Twitter late last month. \n",
- " Musk’s Tesla stock sales, totaling 19.5 million shares, have been widely anticipated ever since the Tesla CEO\n",
- " reached a deal to buy Twitter for $44 billion. Musk had sold blocks of Tesla shares worth a total of $15.4 billion\n",
- " earlier this year since his deal to buy Twitter was announced. Twitter confirmed Musk bought the social media company\n",
- " October 27, but he waited until November 4 to start selling additional Tesla shares. He also sold blocks of Tesla \n",
- " stock on Monday and Tuesday this week, according to filings to the Securities and Exchange Commission late Tuesday night. \n",
- " It’s not clear if the money Musk raised went toward the Twitter purchase, or to support losses at Twitter since he \n",
- " took over. Musk disclosed last week that Twitter has seen a “massive drop in revenue,” as a growing number of advertisers \n",
- " pause spending on the platform in the wake of his takeover of the company. He blamed “activist groups” pressuring \n",
- " advertisers for the loss of ad dollars. He has announced plans to charge users $8 a month to have verified accounts, \n",
- " and also announced deep staff cuts. This is not the best time to be selling Tesla shares, which have lost 46% of their \n",
- " value so far this year on disappointing sales caused by supply chain problems. Musk received an average price of $202.52 \n",
- " for the Tesla shares he sold since the Twitter deal closed, which is down 10% just since he closed on his deal to buy \n",
- " Twitter. Shares of Tesla fell 0.7% in after-hours trading Tuesday. The company is facing growing competition in the \n",
- " electric vehicle market from established automakers such as Volkswagen, Ford and General Motors. And some investors \n",
- " have expressed concerns that Musk will be too distracted by his purchase of Twitter to give enough attention to \n",
- " addressing Tesla’s problems.\"\"\""
- ],
- "metadata": {
- "id": "F8ko-i59hK24"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "code",
- "source": [
- "sns.set(rc={\"figure.dpi\":100, 'savefig.dpi':100})\n",
- "sns.set_context('notebook')\n",
- "sns.set_style(\"ticks\")"
- ],
- "metadata": {
- "id": "l89kojjIX6MI"
- },
- "execution_count": null,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Cosine-similarity matrix\n",
- "Sentence embeddings are compared using cosine-similarity to find similar sentences.\n",
- "\n",
- "The closer the value is to 1, the smaller the angle between the vectors and the more similar the sentences are to each other."
- ],
- "metadata": {
- "id": "iRwO2n5cd38_"
- }
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "n3r4ygRHSVn-"
- },
- "outputs": [],
- "source": [
- "def embed_sentences(text):\n",
- " sent_embeddings = sent_tokenize(text)\n",
- " for i in range (len(sent_embeddings)):\n",
- " sent_embeddings[i] = model.encode(sent_embeddings[i])\n",
- " return sent_embeddings\n",
- "\n",
- "def create_similarity_matrix(sent_embeddings):\n",
- " similarity_matrix = cosine_similarity(sent_embeddings)\n",
- " return similarity_matrix"
- ]
- },
- {
- "cell_type": "code",
- "source": [
- "similarity_matrix = create_similarity_matrix(embed_sentences(TEXT))\n",
- "sns.set(rc={'figure.figsize':(6.25, 5)})\n",
- "sns.heatmap(similarity_matrix).set_title('Cosine-Similarity Matrix')\n",
- "plt.show()"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 465
- },
- "id": "UFThhjjUZcFR",
- "outputId": "4274585d-f2a7-4eb5-8229-0c87915d23cc"
- },
- "execution_count": null,
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "| \n", - " | sentence | \n", - "embeddings | \n", - "cluster | \n", - "
|---|---|---|---|
| 0 | \n", - "Elon Musk sold $3.95 billion worth of Tesla st... | \n", - "[0.015738262, 0.049773764, 0.039228562, -0.002... | \n", - "1 | \n", - "
| 1 | \n", - "Musk’s Tesla stock sales, totaling 19.5 millio... | \n", - "[0.0046068756, 0.010958798, 0.06972741, 0.0108... | \n", - "1 | \n", - "
| 2 | \n", - "Musk had sold blocks of Tesla shares worth a t... | \n", - "[-0.014365158, 0.05811444, 0.053490557, 0.0085... | \n", - "1 | \n", - "
| 3 | \n", - "Twitter confirmed Musk bought the social media... | \n", - "[0.0058419057, -0.014379907, 0.07404005, 0.016... | \n", - "1 | \n", - "
| 4 | \n", - "He also sold blocks of Tesla \\n stock... | \n", - "[-0.054390516, 0.033562854, -0.004229972, 0.01... | \n", - "0 | \n", - "
| 5 | \n", - "It’s not clear if the money Musk raised went t... | \n", - "[0.007103605, 0.03970247, 0.091158584, -0.0287... | \n", - "1 | \n", - "
| 6 | \n", - "Musk disclosed last week that Twitter has seen... | \n", - "[0.017648496, 0.047859456, 0.077083685, -0.027... | \n", - "2 | \n", - "
| 7 | \n", - "He blamed “activist groups” pressuring \\n ... | \n", - "[0.0009142382, 0.03492736, -0.017508835, 0.071... | \n", - "2 | \n", - "
| 8 | \n", - "He has announced plans to charge users $8 a mo... | \n", - "[-0.04066035, -0.04515377, 0.045038592, -0.019... | \n", - "2 | \n", - "
| 9 | \n", - "This is not the best time to be selling Tesla ... | \n", - "[-0.02830206, 0.01951779, 0.061164685, 0.02635... | \n", - "0 | \n", - "
| 10 | \n", - "Musk received an average price of $202.52 \\n ... | \n", - "[0.035062186, 0.0321538, 0.091064826, 0.036340... | \n", - "1 | \n", - "
| 11 | \n", - "Shares of Tesla fell 0.7% in after-hours tradi... | \n", - "[-0.007510837, 0.034629587, 0.07932933, 0.1395... | \n", - "0 | \n", - "
| 12 | \n", - "The company is facing growing competition in t... | \n", - "[0.009982399, -0.004768099, 0.020714289, 0.011... | \n", - "0 | \n", - "
| 13 | \n", - "And some investors \\n have expressed ... | \n", - "[0.05617005, 0.018335959, 0.07072651, 0.013934... | \n", - "1 | \n", - "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "9it0AqXuPR3Y"
- },
- "source": [
- "# fastBART\n",
- "Adaptation of [fastT5](https://github.com/Ki6an/fastT5) for BART text summarization.\n",
- "\n",
- "* BART is converted to ONNX format and quantized\n",
- "* Model is \"flattened\" into a directed graph with nodes being operators\n",
- "* Quantization truncates floating point model weights to 8-bit integers\n",
- "* We should expect a significant decrease in model size and inference time\n",
- "\n",
- "**Work in progress**\n",
- "* The ONNX model does not have the same outputs as the PyTorch model -> bug\n",
- "* ONNX did not increase inference speed as much as expected -> possibly related to bug above\n",
- "* CoLab keeps crashing during quantization"
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Transformers architecture\n",
- ""
- ],
- "metadata": {
- "id": "C8xyZZU4qQZD"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "### Decoder inefficiencies\n",
- "\n",
- "* Note in **Figure 1** that the encoder output is fed as the input to the decoder. This output is called a hidden state, and the same hidden state is used for each subsequent computation in the decoder.\n",
- "* The encoder output can be computed once, saved, and reused for each subsequent step.\n",
- "* The generic Transformers class in PyTorch does not automatically save the encoder output, so the encoder recomputes hidden states during each decoder timestep.\n"
- ],
- "metadata": {
- "id": "2Er0jnSWOfPc"
- }
- },
- {
- "cell_type": "markdown",
- "source": [
- "### Token embedding inefficiencies\n",
- "\n",
- "\n",
- "* **Figure 2** illustrates how the embedding of a decoded token depends only on the previous token decoded before it.\n",
- "* Caching can be applied to prevent unnecessary computation, shown in **Figure 3**.\n",
- "* Seperating the encoder and decoder allows us to control the input/output, ensuring cached values are being passed and only computations related to updating the last token are computed.\n",
- "\n",
- "