Reading delta table using Dataset type File(Machine learning studio)

[J2OG]

Operating System

Windows

Version Information

N/A

Steps to reproduce

Folder structure:
/
└── my_data
├── _delta_log
├── part-0000-xxx.parquet
└── part-0001-xxx.parquet
Relative path for dataset_name set up in dataset type(from azure ML v1 APIs) file: my_data/*.parquet

from azureml.core import Workspace,Dataset
import pandas as pd
ws = Workspace.from_config()
dataset = Dataset.get_by_name(ws, "Dataset_name")
downloaded_parquets = dataset.download()
df_list = list()
for file in downloaded_parquets:
df_list.append(pd.read_parquet(file, engine='pyarrow'))
df = pd.concat(df_list)
df.head(6)

Expected behavior

id name
3 USA
1 FR
2 UK
5 SPAIN

Actual behavior

id name
1 FR
4 INDIA
5 Crude
2 UK
3 USA
5 SPAIN

The actual data it is showing is all the data that was added, doesn't show the most recent version

Addition information

No response

Tasks

Beta Give feedback

An error occurred while loading your tasklist. Please try refreshing the page, or edit + save the issue body.

No tasks being tracked yet.