docs: Add README for advanced image processing (#969)

Azure-Samples · May 24, 2024 · 59f55ea · 59f55ea
1 parent 3a497d7
commit 59f55ea
Show file tree

Hide file tree

Showing 4 changed files with 41 additions and 0 deletions.
diff --git a/docs/advanced_image_processing.md b/docs/advanced_image_processing.md
@@ -0,0 +1,41 @@
+# Advanced Image Processing
+
+The Chat With Your Data solution accelerator supports advanced image processing using Azure AI Vision and GPT-4 Vision to obtain more meaningful data from images.
+
+With advanced image processing enabled, GPT-4 vision is used to generate rich captions of images, and Azure Computer Vision is used to generate embeddings of images, in addition to the text embeddings generated from the image captions.
+
+When querying data in Azure AI Search, vector search is performed using embeddings created for the search query using both Azure OpenAI and Azure Computer Vision.
+
+Finally, if any image documents are retrieved from Azure AI Search, the images themselves are passed to the LLM as well as the text caption generated for the image.
+
+All of this allows a user to ask questions about the meaning of images in the knowledge store, rather than just the text in an image.
+
+Compare the results when uploading the following image:
+
+![image](./images/architecture.png)
+
+Without advanced image processing:
+
+![image](./images/without_advanced_image_processing.png)
+
+With advanced image processing:
+
+![image](./images/with_advanced_image_processing.png)
+
+## Enabling Advanced Image Processing
+
+To enable advanced image processing:
+
+- If deploying from the UI, set "Use Advanced Image Processing" to "true".
+
+- If deploying using azd, run the following before deploying:
+
+```bash
+azd env set USE_ADVANCED_IMAGE_PROCESSING true
+```
+
+Once enabled, advanced image processing will be enabled for all supported image types, however it can be enabled/disabled for individual image types in the Admin app.
+
+![image](./images/enable_advanced_image_processing.png)
+
+Advanced image processing is only used in the `custom` conversation flow and not the `byod` flow, as Azure OpenAI On Your Data only supports Ada embeddings. It is currently not possible to use advanced image processing when integrated vectorization is enabled.
diff --git a/docs/images/enable_advanced_image_processing.png b/docs/images/enable_advanced_image_processing.png
diff --git a/docs/images/with_advanced_image_processing.png b/docs/images/with_advanced_image_processing.png
diff --git a/docs/images/without_advanced_image_processing.png b/docs/images/without_advanced_image_processing.png