Skip to content

Commit

Permalink
[vision] Migrate vision projects to use snippets extraction (#33222)
Browse files Browse the repository at this point in the history 
### Packages impacted by this PR

- @azure-rest/ai-vision-image-analysis

### Issues associated with this PR

- #32416

### Describe the problem that is addressed by this PR

Updates all projects under `vision` to use snippets extraction.

### What are the possible designs available to address the problem? If
there are more than one possible design, why was the one in this PR
chosen?


### Are there test cases added in this PR? _(If not, why?)_


### Provide a list of related PRs _(if any)_


### Command used to generate this PR:**_(Applicable only to SDK release
request PRs)_

### Checklists
- [ ] Added impacted package name to the issue description
- [ ] Does this PR needs any fixes in the SDK Generator?** _(If so,
create an Issue in the
[Autorest/typescript](https://github.com/Azure/autorest.typescript)
repository and link it here)_
- [ ] Added a changelog (if necessary)
  • Loading branch information
@mpodwysocki
mpodwysocki authored Feb 27, 2025
1 parent af3ed50 commit 16c3124
Show file tree
Hide file tree
Showing 18 changed files with 424 additions and 129 deletions.
250 changes: 176 additions & 74 deletions sdk/vision/ai-vision-image-analysis-rest/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
The Image Analysis service provides AI algorithms for processing images and returning information about their content. In a single service call, you can extract one or more visual features from the image simultaneously, including getting a caption for the image, extracting text shown in the image (OCR) and detecting objects. For more information on the service and the supported visual features, see [Image Analysis overview][image_analysis_overview], and the [Concepts][image_analysis_concepts] page.

Use the Image Analysis client library to:
* Authenticate against the service
* Set what features you would like to extract
* Upload an image for analysis, or send an image URL
* Get the analysis result

[Product documentation][image_analysis_overview]
- Authenticate against the service
- Set what features you would like to extract
- Upload an image for analysis, or send an image URL
- Get the analysis result

[Product documentation][image_analysis_overview]
| [Samples](https://aka.ms/azsdk/image-analysis/samples/js)
| [Vision Studio][vision_studio]
| [API reference documentation](https://aka.ms/azsdk/image-analysis/ref-docs/js)
Expand All @@ -28,9 +29,9 @@ See our [support policy](https://github.com/Azure/azure-sdk-for-js/blob/main/SUP

- An [Azure subscription](https://azure.microsoft.com/free).
- A [Computer Vision resource](https://portal.azure.com/#create/Microsoft.CognitiveServicesComputerVision) in your Azure subscription.
* You will need the key and endpoint from this resource to authenticate against the service.
* You can use the free pricing tier (`F0`) to try the service, and upgrade later to a paid tier for production.
* Note that in order to run Image Analysis with the `Caption` or `Dense Captions` features, the Azure resource needs to be from one of the following GPU-supported regions: `East US`, `France Central`, `Korea Central`, `North Europe`, `Southeast Asia`, `West Europe`, or `West US`.
- You will need the key and endpoint from this resource to authenticate against the service.
- You can use the free pricing tier (`F0`) to try the service, and upgrade later to a paid tier for production.
- Note that in order to run Image Analysis with the `Caption` or `Dense Captions` features, the Azure resource needs to be from one of the following GPU-supported regions: `East US`, `France Central`, `Korea Central`, `North Europe`, `Southeast Asia`, `West Europe`, or `West US`.

### Install the `@azure-rest/ai-vision-image-analysis` package

Expand Down Expand Up @@ -64,9 +65,9 @@ For more information about these features, see [Image Analysis overview][image_a

Image Analysis works on images that meet the following requirements:

* The image must be presented in JPEG, PNG, GIF, BMP, WEBP, ICO, TIFF, or MPO format
* The file size of the image must be less than 20 megabytes (MB)
* The dimensions of the image must be greater than 50 x 50 pixels and less than 16,000 x 16,000 pixels
- The image must be presented in JPEG, PNG, GIF, BMP, WEBP, ICO, TIFF, or MPO format
- The file size of the image must be less than 20 megabytes (MB)
- The dimensions of the image must be greater than 50 x 50 pixels and less than 16,000 x 16,000 pixels

### ImageAnalysisClient

Expand All @@ -78,26 +79,20 @@ The `ImageAnalysisClient` is the primary interface for developers interacting wi

Here's an example of how to create an `ImageAnalysisClient` instance using a key-based authentication.


```javascript Snippet:const endpoint = "<your_endpoint>";
const key = "<your_key>";
const credential = new AzureKeyCredential(key);

const client = new ImageAnalysisClient(endpoint, credential);

const { ImageAnalysisClient } = require("@azure-rest/ai-vision-image-analysis");
const { AzureKeyCredential } = require('@azure/core-auth');
```ts snippet:ReadmeSampleCreateClient_KeyCredential
import { AzureKeyCredential } from "@azure/core-auth";
import ImageAnalysisClient from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const key = "<your_key>";
const credential = new AzureKeyCredential(key);

const client = new ImageAnalysisClient(endpoint, credential);
const client = ImageAnalysisClient(endpoint, credential);
```

#### Create ImageAnalysisClient with a Microsoft Entra ID Credential

**Prerequisites for Entra ID Authentication**:

- The role `Cognitive Services User` assigned to you. Role assignment can be done via the "Access Control (IAM)" tab of your Computer Vision resource in the Azure portal.
- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed.
- You are logged into your Azure account by running `az login`.
Expand All @@ -110,92 +105,199 @@ Client subscription key authentication is used in most of the examples in this g
npm install @azure/identity
```

```javascript Snippet:ImageAnalysisEntraIDAuth
```ts snippet:ReadmeSampleCreateClient_DefaultAzureCredential
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();

const client = new ImageAnalysisClient(endpoint, credential);
const client = ImageAnalysisClient(endpoint, credential);
```

### Analyze an image from URL

The following example demonstrates how to analyze an image using the Image Analysis client library for JavaScript.

```javascript Snippet:ImageAnalysisFromUrl
```ts snippet:ReadmeSampleAnalyzeImageFromUrl
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imageUrl = "https://example.com/image.jpg";
const features = ["Caption", "DenseCaptions", "Objects", "People", "Read", "SmartCrops", "Tags"];

async function analyzeImageFromUrl() {
const result = await client.path("/imageanalysis:analyze").post({
body: {
url: imageUrl,
},
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/json",
});
const result = await client.path("/imageanalysis:analyze").post({
body: {
url: imageUrl,
},
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/json",
});
if (isUnexpected(result)) {
throw result.body.error;
}

console.log(`Model Version: ${result.body.modelVersion}`);
console.log(`Image Metadata: ${JSON.stringify(result.body.metadata)}`);

if (result.body.captionResult) {
console.log(
`Caption: ${result.body.captionResult.text} (confidence: ${result.body.captionResult.confidence})`,
);
}

if (result.body.denseCaptionsResult) {
for (const denseCaption of result.body.denseCaptionsResult.values) {
console.log(`Dense Caption: ${JSON.stringify(denseCaption)}`);
}
}

if (result.body.objectsResult) {
for (const object of result.body.objectsResult.values) {
console.log(`Object: ${JSON.stringify(object)}`);
}
}

if (result.body.peopleResult) {
for (const person of result.body.peopleResult.values) {
console.log(`Person: ${JSON.stringify(person)}`);
}
}

if (result.body.readResult) {
for (const block of result.body.readResult.blocks) {
console.log(`Text Block: ${JSON.stringify(block)}`);
}
}

console.log("Image analysis result:", result.body);
if (result.body.smartCropsResult) {
for (const smartCrop of result.body.smartCropsResult.values) {
console.log(`Smart Crop: ${JSON.stringify(smartCrop)}`);
}
}

analyzeImageFromUrl();
if (result.body.tagsResult) {
for (const tag of result.body.tagsResult.values) {
console.log(`Tag: ${JSON.stringify(tag)}`);
}
}
```

### Analyze an image from a local file

In this example, we will analyze an image from a local file using the Image Analysis client library for JavaScript.

```javascript Snippet:ImageAnalysisFromLocalFile
const fs = require("fs");
```ts snippet:ReadmeSampleAnalyzeImageFromFile
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";
import { readFileSync } from "node:fs";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imagePath = "./path/to/your/image.jpg";
const features = ["Caption", "DenseCaptions", "Objects", "People", "Read", "SmartCrops", "Tags"];

async function analyzeImageFromFile() {
const imageBuffer = fs.readFileSync(imagePath);
const imageBuffer = readFileSync(imagePath);

const result = await client.path("/imageanalysis:analyze").post({
body: imageBuffer,
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/octet-stream",
});
if (isUnexpected(result)) {
throw result.body.error;
}

console.log(`Model Version: ${result.body.modelVersion}`);
console.log(`Image Metadata: ${JSON.stringify(result.body.metadata)}`);

const result = await client.path("/imageanalysis:analyze").post({
body: imageBuffer,
queryParameters: {
features: features,
"smartCrops-aspect-ratios": [0.9, 1.33],
},
contentType: "application/octet-stream",
});
if (result.body.captionResult) {
console.log(
`Caption: ${result.body.captionResult.text} (confidence: ${result.body.captionResult.confidence})`,
);
}

if (result.body.denseCaptionsResult) {
for (const denseCaption of result.body.denseCaptionsResult.values) {
console.log(`Dense Caption: ${JSON.stringify(denseCaption)}`);
}
}

console.log("Image analysis result:", result.body);
if (result.body.objectsResult) {
for (const object of result.body.objectsResult.values) {
console.log(`Object: ${JSON.stringify(object)}`);
}
}

if (result.body.peopleResult) {
for (const person of result.body.peopleResult.values) {
console.log(`Person: ${JSON.stringify(person)}`);
}
}

analyzeImageFromFile();
if (result.body.readResult) {
for (const block of result.body.readResult.blocks) {
console.log(`Text Block: ${JSON.stringify(block)}`);
}
}

if (result.body.smartCropsResult) {
for (const smartCrop of result.body.smartCropsResult.values) {
console.log(`Smart Crop: ${JSON.stringify(smartCrop)}`);
}
}

if (result.body.tagsResult) {
for (const tag of result.body.tagsResult.values) {
console.log(`Tag: ${JSON.stringify(tag)}`);
}
}
```

### Extract text from an image Url

This example demonstrates how to extract printed or hand-written text for the image file [sample.jpg](https://aka.ms/azsdk/image-analysis/sample.jpg) using the ImageAnalysisClient. The method call returns an ImageAnalysisResult object. The ReadResult property on the returned object includes a list of text lines and a bounding polygon surrounding each text line. For each line, it also returns a list of words in the text line and a bounding polygon surrounding each word.
``` javascript Snippet:readmeText
const client: ImageAnalysisClient = createImageAnalysisClient(endpoint, credential);

const features: string[] = [
'Read'
];
```ts snippet:ReadmeSampleExtractTextFromImageUrl
import { DefaultAzureCredential } from "@azure/identity";
import ImageAnalysisClient, { isUnexpected } from "@azure-rest/ai-vision-image-analysis";

const endpoint = "<your_endpoint>";
const credential = new DefaultAzureCredential();
const client = ImageAnalysisClient(endpoint, credential);

const imageUrl: string = 'https://aka.ms/azsdk/image-analysis/sample.jpg';
const features: string[] = ["Read"];
const imageUrl: string = "https://aka.ms/azsdk/image-analysis/sample.jpg";

client.path('/imageanalysis:analyze').post({
const result = await client.path("/imageanalysis:analyze").post({
body: { url: imageUrl },
queryParameters: { features: features },
contentType: 'application/json'
}).then(result => {
const iaResult: ImageAnalysisResultOutput = result.body as ImageAnalysisResultOutput;

// Process the response
if (iaResult.readResult && iaResult.readResult.blocks.length > 0) {
iaResult.readResult.blocks.forEach(block => {
console.log(`Detected text block: ${JSON.stringify(block)}`);
});
} else {
console.log('No text blocks detected.');
contentType: "application/json",
});
if (isUnexpected(result)) {
throw result.body.error;
}

// Process the response
const imageAnalysisResult = result.body;
if (imageAnalysisResult.readResult && imageAnalysisResult.readResult.blocks.length > 0) {
for (const block of imageAnalysisResult.readResult.blocks) {
console.log(`Detected text block: ${JSON.stringify(block)}`);
}
} else {
console.log("No text blocks detected.");
}
```

## Troubleshooting
Expand All @@ -204,8 +306,8 @@ client.path('/imageanalysis:analyze').post({

Enabling logging may help uncover useful information about failures. In order to see a log of HTTP requests and responses, set the `AZURE_LOG_LEVEL` environment variable to `info`. Alternatively, logging can be enabled at runtime by calling `setLogLevel` in the `@azure/logger`:

```javascript
const { setLogLevel } = require("@azure/logger");
```ts snippet:SetLogLevel
import { setLogLevel } from "@azure/logger";

setLogLevel("info");
```
Expand All @@ -228,4 +330,4 @@ If you'd like to contribute to this library, please read the [contributing guide
[image_analysis_concepts]: https://learn.microsoft.com/azure/ai-services/computer-vision/concept-tag-images-40
[vision_studio]: https://aka.ms/vision-studio/image-analysis
[azure_identity]: https://learn.microsoft.com/javascript/api/overview/azure/identity-readme
[azure_identity_dac]: https://learn.microsoft.com/javascript/api/@azure/identity/defaultazurecredential
[azure_identity_dac]: https://learn.microsoft.com/javascript/api/@azure/identity/defaultazurecredential
Loading

0 comments on commit 16c3124

Please sign in to comment.