3rd Iteration of Incubator Tools #700

sana-google · 2023-12-19T10:22:30Z

Added 9 new tools
Updated README.md

review-notebook-app · 2023-12-19T10:22:35Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

holtskinner · 2023-12-19T18:48:52Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Provide more description of what these are.

Reply via ReviewNB

This is taken care, adding the description here also The objective of the tool is to evaluate the character error rate which measures the rate of errors at the character level in the generated output compared to the reference. and word error rate which measures the rate of errors at the word level in the generated output compared to the reference.Both CER and WER can be used to provide a quantitative measure of accuracy, with lower values indicating better performance.

holtskinner · 2023-12-19T18:48:52Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Most of this download from GCS code should be brought out into the utilities

Reply via ReviewNB

It is possible only for json files, the variable groundtruth_files are text files and cannot be downloaded with our functions in utilities. This is an exception so we decided not to include the text downloading function to our utilities module.

holtskinner · 2023-12-19T18:48:52Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Line #79. print(f"Mean CER = {round(mean_cer,2)}%, Mean WER = {round(mean_wer,2)}%")
You can replace the round functions with {mean_cer:.2f} and {mean_wer:.2f}

Reply via ReviewNB

Acknowledged. Updated the code.

holtskinner · 2023-12-19T18:48:52Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Line #82. if __name__ == "__main__":
This isn't needed in a Notebook

Reply via ReviewNB

Marking as done. Replaced the condition and called the main function without the condition.

holtskinner · 2023-12-19T18:48:56Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Provide more description of what these are.

Reply via ReviewNB

Acknowledged.

holtskinner · 2023-12-19T18:48:56Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Most of this download from GCS code should be brought out into the utilities

Reply via ReviewNB

Acknowledged.

holtskinner · 2023-12-19T18:48:56Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Line #79. print(f"Mean CER = {round(mean_cer,2)}%, Mean WER = {round(mean_wer,2)}%")
You can replace the round functions with {mean_cer:.2f} and {mean_wer:.2f}

Reply via ReviewNB

Acknowledged.

holtskinner · 2023-12-19T18:48:56Z

incubator-tools/cer_wer/cer_wer.ipynb

@@ -0,0 +1,296 @@
+{


Line #82. if __name__ == "__main__":
This isn't needed in a Notebook

Reply via ReviewNB

Acknowledged.

holtskinner · 2023-12-19T19:47:45Z

incubator-tools/currency_normalization/currency_normalization.ipynb

@@ -0,0 +1,354 @@
+{


Make these variable names in code font

Reply via ReviewNB

Changed the variable names according to code font

holtskinner · 2023-12-19T19:47:45Z

incubator-tools/currency_normalization/currency_normalization.ipynb

@@ -0,0 +1,354 @@
+{


Line #17. return json_data
This function should be in the utilities

Reply via ReviewNB

holtskinner · 2023-12-19T19:47:46Z

incubator-tools/currency_normalization/currency_normalization.ipynb

@@ -0,0 +1,354 @@
+{


Line #34. df = pd.read_excel(excel_path)
There should be a built in function for pandas to convert this data frame to a dictionary

Reply via ReviewNB

A specified format is needed that's why we have used this approach(key_names, values)

holtskinner · 2023-12-19T19:47:46Z

incubator-tools/currency_normalization/currency_normalization.ipynb

@@ -0,0 +1,354 @@
+{


Line #85. )
Why does this variable need to be set twice?

Reply via ReviewNB

holtskinner · 2023-12-19T19:47:46Z

incubator-tools/currency_normalization/currency_normalization.ipynb

@@ -0,0 +1,354 @@
+{


Line #89. ] # IF MENTION TEXT NEED NOT BE CHANGED COMMENT THIS LINE
I think this comment should go above the section (after formatting)

Reply via ReviewNB

holtskinner · 2023-12-19T19:47:47Z

...er_document_page_based_on_unique_strings/reorder_document_page_based_on_unique_strings.ipynb

@@ -0,0 +1,363 @@
+{


I'm not sure I understand what this notebook does or what the use case is

Reply via ReviewNB

This tool uses OCR to reorder PDF pages based on unique strings (logical identifiers) provided by the user, ideal for organizing out of sequence PDF pages into their correct order.

holtskinner · 2023-12-19T19:47:47Z

incubator-tools/test_harness_guide/test_harness_guide.ipynb

@@ -0,0 +1,1455 @@
+{


Remove the comments

Reply via ReviewNB

Comments are removed in updated notebook

holtskinner · 2023-12-19T19:47:47Z

incubator-tools/test_harness_guide/test_harness_guide.ipynb

@@ -0,0 +1,1455 @@
+{


Overall, this seems over complicated. This code can be simplified/made more modular and readable.

Reply via ReviewNB

This comment taken-care in updated notebook

holtskinner · 2023-12-19T19:47:47Z

incubator-tools/test_harness_guide/test_harness_guide.ipynb

@@ -0,0 +1,1455 @@
+{


Line #490. def batch_process_documents(
Not sure why this extra functuion is needed

Reply via ReviewNB

In updated notebook, this staging function removed.

holtskinner · 2023-12-19T19:47:47Z

incubator-tools/tif_to_jpeg_image_conversion/tif_to_jpeg_image_conversion.ipynb

@@ -0,0 +1,243 @@
+{


This seems like a very basic tool using base Python libraries. Maybe should just be in utilities? Or not included.

Reply via ReviewNB

Removed the notebook. Updated it as a function in the utilities.py

…ples Syncing

sana-google added 2 commits December 19, 2023 13:02

3rd Iteration of DocAI Incubator Tools

3248f5c

Format ipynb files

8651f0e

Edited README files based on failed lint check

68d53a4

sana-google marked this pull request as ready for review December 19, 2023 11:15

sana-google requested a review from a team as a code owner December 19, 2023 11:15

Merge branch 'main' into main

f238bb1

holtskinner reviewed Dec 19, 2023

View reviewed changes

sana-google and others added 4 commits December 29, 2023 11:53

Resolve comments on PR 3

d379fb1

Merge branch 'main' of https://github.com/sana-google/document-ai-sam…

8925288

…ples Syncing

Resolve lint errors in utilities.py

e86c252

Merge branch 'main' into main

b693f11

holtskinner approved these changes Jan 2, 2024

View reviewed changes

Merge branch 'main' into main

c77617a

holtskinner merged commit f4c03bd into GoogleCloudPlatform:main Jan 2, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3rd Iteration of Incubator Tools #700

3rd Iteration of Incubator Tools #700

sana-google commented Dec 19, 2023

review-notebook-app bot commented Dec 19, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

Zaidcoderepos Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

ssnvenkatesh888 Dec 28, 2023

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

ssnvenkatesh888 Dec 21, 2023

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

vignesh-waran-k Dec 20, 2023

holtskinner Dec 19, 2023 •

edited

Loading

vijay-google Dec 26, 2023

holtskinner Dec 19, 2023 •

edited

Loading

vijay-google Dec 26, 2023

holtskinner Dec 19, 2023 •

edited

Loading

vijay-google Dec 26, 2023

holtskinner Dec 19, 2023 •

edited

Loading

vignesh-waran-k Dec 28, 2023

3rd Iteration of Incubator Tools #700

3rd Iteration of Incubator Tools #700

Conversation

sana-google commented Dec 19, 2023

review-notebook-app bot commented Dec 19, 2023

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading

holtskinner Dec 19, 2023 •

edited

Loading