Display available credits/quotas from service providers

[jeffpaul]

Is your enhancement related to a problem? Please describe.
In chatting with @dingman, one item that would be nice to expose if we can get it via service provider APIs is the current usage / credits / quotas a user has with their account.

Describe the solution you'd like

• on the Language and Image Processing settings pages, display the available credits/quotas from enabled services
• Azure Computer Vision would ideally show Pricing Tier, Subscription, Quota Info (Total Calls, Remaining Calls, Rest Period), and Total Cost
• Watson NLU would ideally show NLU Items Available, NLU Items Used, API Rate Limit, and Total Cost

Designs

• designs needed

Describe alternatives you've considered
n/a

Additional context
NOTE: A NLU item is based on the number of data units enriched and the number of enrichment features applied. A data unit is 10,000 characters or less. For example: extracting Entities and Sentiment from 15,000 characters of text is (2 Data Units * 2 Enrichment Features) = 4 NLU Items.

[jeffpaul]

Here's a screenshot of what I'm shown within the NLU resource page on the Watson site:

https://cloudup.com/ccu0dbfTGYP

[dkotter]

One other thing that may be nice to add here is a setting that can be toggled on to disable functionality once limits have been reached. This would allow site owners to test out features within Classifai with no risk of going over their free limits

[jeffpaul]

Thinking this might be helpful as a WP Admin dashboard widget, so something like...

Widget Title: ClassifAI Costs and Quotas

Widget Content:
Language Processing: <Service Provider> <-- Only display if a Language Processing feature is enabled, in which case display the active Service Provider (e.g. IBM Watson, Azure AI, OpenAI)
<cost/quota info> <-- Display whatever data we're able to retrieve from the active Service Provider. If not data is available, then attempt to deep-link into the Service Providers pricing information page.

Image Processing: <Service Provide> <-- Only display if an Image Processing feature is enabled, in which case display the active Service Provider (e.g. IBM Watson, Azure AI, OpenAI)
<cost/quota info> <-- Display whatever data we're able to retrieve from the active Service Provider. If not data is available, then attempt to deep-link into the Service Providers pricing information page.

View ClassifAI Settings <-- link to the ClassifAI Settings page

[kirtangajjar]

Okay, so after some investigation, here's what I've found so far:

Most of the APIs I've seen don't return in tokens, but they return charges (in USD or other currency). This was one doubt that I had in my mind on what if some APIs return tokens while others return in currency, but thankfully that's not the case.

I had a chance to poke around with OpenAI's API, but unfortunately, I didn't get a chance to explore IBM Watson/Azure since my credit card didn't work with them. So, I've relied on docs for this.

OpenAI

OpenAI does not have an official public-facing Usage API that's documented by them, but inspecting request on their usage page, I was able to find an API endpoint that can work for our purpose.

The API endpoint is:

https://api.openai.com/dashboard/billing/usage?end_date=&start_date=

i.e. https://api.openai.com/dashboard/billing/usage?end_date=2023-08-01&start_date=2023-07-01

You can open it in the browser and just enter your email and API key, and it should work. The response looks like:

{
  "object": "list",
  "daily_costs": [
    {
      "timestamp": 1688688000.0,
      "line_items": [
        {
          "name": "GPT-4",
          "cost": 0.0
        },
        {
          "name": "Chat models",
          "cost": 0.0
        },
        {
          "name": "InstructGPT",
          "cost": 0.00116
        },
        {
          "name": "Fine-tuning models",
          "cost": 0.0
        },
        {
          "name": "Embedding models",
          "cost": 0.0
        },
        {
          "name": "Image models",
          "cost": 0.0
        },
        {
          "name": "Audio models",
          "cost": 0.0
        }
      ]
    }
  ],
  "total_usage": 0.00116
}

So we get usage broken down by day and in turn broken down by models, which is super helpful.

IBM Cloud

IBM Cloud has https://cloud.ibm.com/apidocs/metering-reporting, which is the official Usage API. To fetch data from it, we first need to authenticate and then to get total cost in a month, we can use account usage summary endpoint, and if we want breakdown of charges, we can use account usage endpoint(which does not have total charges). However, I couldn't find charges for Watson API in the example response, so it's something that we might need to have a look at.

Azure

Azure has Usage Details API that we can make use of. There's BillingAccountUsageDetailsList and CustomerUsageDetailsList that looks promising, but I'm unsure about the difference between them and we'll only be able to figure out more once someone with azure account trys to figure it out.

[dkotter]

OpenAI does not have an official public-facing Usage API that's documented by them, but inspecting request on their usage page, I was able to find an API endpoint that can work for our purpose

My concern here is that they may not want us to use this endpoint if it's not publicly documented. I'd hate to build something around this and then end up having the endpoint be blocked.

One thing I have considered is that most of OpenAI's APIs will return how many tokens are used. If we were to store that information, it would be fairly easy to then do the math ourselves to figure out how much all those requests cost. This could be broken down into time periods (day, week, month, etc) but also might be neat to be able to see how much an individual action cost (for instance, how much did generating excerpts on this particular post cost me).

Now this doesn't solve other providers as tokens are specific to OpenAI (at least with the providers we've integrated with) but is something to consider.

[kirtangajjar]

One thing I have considered is that most of OpenAI's APIs will return how many tokens are used. If we were to store that information, it would be fairly easy to then do the math ourselves to figure out how much all those requests cost. This could be broken down into time periods (day, week, month, etc) but also might be neat to be able to see how much an individual action cost (for instance, how much did generating excerpts on this particular post cost me).

This is really a great idea, but then as other providers don't expose this sort of info on individual operations(AFAIK) and only show the aggregate of usage across UI and API, the problems I see here is that:

1. Will we be able to represent our data in a that we're able to convey this difference to the users? That OpenAI usage is just from what is used by ClassifAI instance on this particular site, while usage data of other services is overall? This sounds a bit complicated to me.
2. A future idea that we have in mind is that we're also planning to put limits here. I'm assuming people might also be using OpenAI services i.e. GPT-4 directly via web interface or via other tools such as AutoGPT or other plugins. I'm not sure we might be able to put limits effectively in that case.

I like the idea that perhaps we can use individual API cost to maybe show cost per post or show history of API calls and how much it cost in that sense, but I think mixing both the overall cost as shown in other providers with OpenAI individual API cost approach might pose some problems.

[jeffpaul]

I'd say to start with something simple, especially given the difference in available data (and how hacky some of the data access currently is). Probably even fine to start only with pulling OpenAI data as that seems to require the least additional effort from a site owner to get access for ClassifAI to pull data on Watson/Azure. So to that end, perhaps tracking monthly spend (pulled on the 1st of the month for the previous month?) and displaying the current/previous month and all time spend from OpenAI?

[kirtangajjar]

Sounds like a plan!