Initial pipeline config_ConfigureExports run after scope config failing

[lundejd]

I went to deploy a fresh version .4 today and everything deployed fine, but when I configured for my EA scope I see the following error:

Operation on target ForEach Export Scope failed: The function 'length' expects its parameter to be an array or a string. The provided value is of type 'Object'.

I do notice the .json config file scope is different than .3 (there are no [] characters).

Also there is no Author > Pipelines > msexports_backfill pipeline. I wanted to do it this way vs. doing manual monthly backfill exports like before. Is this just not ready yet?

[arthurclares]

@lundejd sorry for the delay on this one, John.
For the first issue with the configureExports pipeline, can you double check if your config file looks like this?
{ "$schema": "https://aka.ms/finops/hubs/settings-schema", "type": "HubInstance", "version": "0.4", "learnMore": "https://aka.ms/finops/hubs", "scopes": [ { "scope": "xxxxx" } ], "retention": { "msexports": { "days": 0 }, "ingestion": { "months": 13 } } }
About the second question, there is a mistake on the docs and the pipeline name is wrong, it should say "config_RunBackfill" instead of msexports_backfill. Can you try to run this pipeline instead?
Just make sure you have assigned the correct permissions to the Data Factory managed identity at the correct scope (subscription or billing account)

[soyroberto]

my file does look like this: {
"$schema": "https://aka.ms/finops/hubs/settings-schema",
"type": "HubInstance",
"version": "0.4",
"learnMore": "https://aka.ms/finops/hubs",
"scopes": {
"scope": "/providers/Microsoft.Billing/billingAccounts/xxxxxxxx/"
},
"retention": {
"msexports": {
"days": 0
},
"ingestion": {
"months": 13
}
}
}
I've tried to run using the date format yyyy-mm-dd both with '' and without - no luck
but doesn't run

[flanakin]

@soyroberto Can you update settings.json to use:

"scopes": [{
    "scope": "/providers/Microsoft.Billing/billingAccounts/xxxxxxxx/"
}],

You're missing brackets around the curly braces. Also let us know if you updated this manually or via a tool/script. I just want to make sure we have the right guidance and bugs are resolved.

[soyroberto]

Thanks for the callout, modified it to:

I'm doing the configuration of this file manually using VScode. copy/pasting azure portal-VScode-Azureportal

[flanakin]

I see the problem. For some reason, the settings.json file is created without the brackets. I made a change in 0.5 which will handle it with or without the brackets to avoid this going forward.

Thanks for sharing the details and, again, my apologies for the inconvenience here.

[saleemmy]

I am having the same error.

[flanakin]

I wonder if you're running into the same issue as @soyroberto. Make sure the scopes property in settings.json is an array that is surrounded by [ and ].

[soyroberto]

I'm confused documentation says: 'FinOps hubs leverage Cost Management exports to obtain cost data. . am I supposed to be dropping the files myself in the ingestion folder? exported from the Cost management?
+
Step 1: Verify Cost Management export
Go to Cost Management exports and make sure the export status is “Successful”.

Sources:
https://microsoft.github.io/finops-toolkit/hubs/data
https://microsoft.github.io/finops-toolkit/help/troubleshoot#-validate-your-finops-hub-deployment

Would this be the reason the DF run fails? Isn't the DF supposed to generated the cost files itself?

[soyroberto]

But the same documentation says: '

• Configure managed exports Managed exports allow FinOps hubs to setup and maintain Cost Management exports for you'

• For the most seamless experience, we recommend allowing FinOps hubs to manage exports for you when possible. This is the simplest to setup and requires the least effort to maintain over time.
  Quite confusing

[flanakin]

Configure exports into the msexports container. The ADF pipeline transforms the data to parquet and moves it to the ingestion container. If you give your hub access to your scopes, it can manage the exports for you, but the exports are the way we get cost data.

[soyroberto]

We have permissions to the principal at this level shown in the picture:

how can i confirm it does generate a new export every month?

[flanakin]

When you add a scope, exports are created. Cost Management runs the exports on a schedule automatically.

[soyroberto]

I think we finally it got it working,
1st: We configured an export using the FOCUS format and I assume this trigger the DF, would like to see where that logic sits generated a .parquet file, see pictures:

Would go to PBI and see how that goes

[saleemmy]

I updated the settings.json file with the scope. Now I get this error Operation on target ForEach Export Scope failed: Activity failed because an inner activity failed; Inner activity name: Get exports for scope, Error: {"error":{"code":"RBACAccessDenied","message":"The client does not have authorization to perform action. Request ID: NoHttpContext"}}

{
"$schema": "https://aka.ms/finops/hubs/settings-schema",
"type": "HubInstance",
"version": "0.4",
"learnMore": "https://aka.ms/finops/hubs",
"scopes": [{
"scope":
"/providers/Microsoft.Billing/billingAccounts/xxxxxxxxxxxxx"
}],
"retention": {
"ingestion": {
"months": 13
},
"msexports": {
"days": 0
}
}
}

[flanakin]

This means the ADF managed identity does not have access to create exports for the billing account. Did you give it access?

Email ftk-support@ms and we can setup time to make sure this is working for you.

[soyroberto]

I got it working, Once all settles, I'd be planning to write a blog about, still I have a key issue, I configured the report to cover 1 year of costs as specified here: https://microsoft.github.io/finops-toolkit/power-bi/setup#-setup-your-first-report

Change RangeStart and RangeEnd to the desired start/end dates for your report

Thing is, it only brings 1 month of data (august). Also, 1 yr of data is 1.7M less than the 2M warning specified.

See images below

I tried setting the dates to:
1/1/2024
1/9/2024
9 months and it keeps showing only august, i cannot even select dates, thoughts?

[soyroberto]

I just reviewed the export settings there's no way to export 12 months, that's why I only see 1 month data, wonder how do I get 12 months of data?

[flanakin]

From the UI, you can open the export details and export one month at a time for the last 12 months.

From PowerShell, you can go back 7 years. See https://aka.ms/ftk/Start-FinOpsCostExport

[MS-mikeo]

@flanakin - I ran into this issue with a customer today. We deployed .5 and still had to add the [ ] within the scopes section to get the pipeline to run.

@arthurclares - It looks like maybe the pipeline run name changed even from your comment earlier in this issue. The documentation needs to be updated withe pipeline "config_BackfillData" when configuring the managed exports.

[flanakin]

Sorry, missed this. We investigated this further and discovered there were a few other places that need to be updated to resolve the [ ] issue. We fixed those in 0.6.

Side note: We changed the names of these pipelines to hopefully be clearer. Docs have also been updated accordingly, so they should be aligned. Let us know if you spot any issues.

[lundejd]

One thing I realized after having failure after failure is that the export script seems to only take a single month per run, not a whole range of months (similar to how Cost Management manual back-fill jobs are done) I may have missed it but this is nowhere in the documentation, nor is the format of the start and end date strings so I had to do a lot of trial and error to find something that works.

StartDate: 2024-08-01
EndDate: 2024-08-31

Seems to work just fine, but 20240801 and 20240831 or other combos will generate an error.

[soyroberto]

I made a similar comment about it - #817 (reply in thread) . I still have the question, how can I visualize more than 1 month of data?

[flanakin]

Ah, yes. I noticed the pipeline only supports one month at a time as well. We need to update that to scale better. For now, I would suggest trying the Start-FinOpsCostExport command. If it'd help, we could create a template to deploy that to Azure Automation.

[flanakin]

I made a similar comment about it - #817 (reply in thread) . I still have the question, how can I visualize more than 1 month of data?

@soyroberto

1. Export the months you need from Cost Management
2. Verify the data was pushed to the msexports container in storage
3. Verify ADF moved the data to the ingestion container in storage
4. Clear RangeStart/RangeEnd parameters to pull all data from storage (or set Number of Months in 0.5 or 0.6 reports)
5. Make sure the date picker in the report (top-left) is covering all the months you want

[soyroberto]

Running Start-FinOpsCostExport is where the key of exporting data is. There's no way to do the same on the webui. The webUI will work for a month's export, afterwards the PScommands must be run.

If they're run successfully the structure on the storage account will look like + the data factory:

Once the data is updated, if needed refresh the PowerBI reports.

@flanakin RE: If it'd help, we could create a template to deploy that to Azure Automation. This would be fantastic. Thanks

[flanakin]

Can you create an issue to track that request?

[RamNadal]

Hello @flanakin , i tried to chagne the scope under the settings.json to subscriptions and got the below error

Operation on target ForEach Export Scope failed: Activity failed because an inner activity failed; Inner activity name: Create or update open month focus export, Error: {"error":{"code":"RBACAccessDenied","message":"The client does not have authorization to perform action. Request ID: dc725b8a-0af9-4886-b244-0aa69189256d"}}

[flanakin]

This sounds like the Data Factory managed identity does not have access to the subscription you're adding. Please refer to the first step in the configure scopes doc.

[RamNadal]

@flanakin When i chang ethe scope to my billing profile ..i get the belwo error message

Operation on target ForEach Export Scope failed: Activity failed because an inner activity failed; Inner activity name: Create or update open month focus export, Error: {"error":{"code":"MissingSubscription","message":"The request did not have a subscription or a valid tenant level resource provider."}}

[flanakin]

This sounds like the scope ID may be invalid. Make sure subscriptions are added using the scope format as /subscriptions/############-####-####-####-########.