ImportError: cannot import name '_DATASET_OUTPUT_ARGUMENT_TEMPLATE'

[saleung]

In the notebook 02_AutoML_Training cell "Build many model training steps" section running cell results in ImportError: cannot import name '_DATASET_OUTPUT_ARGUMENT_TEMPLATE'

from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder

train_steps = AutoMLPipelineBuilder.get_many_models_train_steps(experiment=experiment,
                                                                automl_settings=automl_settings,
                                                                train_data=filedst_10_models_input,
                                                                compute_target=compute,
                                                                partition_column_names=partition_column_names,
                                                                node_count=2,
                                                                process_count_per_node=8,
                                                                run_invocation_timeout=3700,
                                                                output_datastore=dstore)

Full error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-16-5cb613553b0a> in <module>
----> 1 from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder
      2 
      3 train_steps = AutoMLPipelineBuilder.get_many_models_train_steps(experiment=experiment,
      4                                                                 automl_settings=automl_settings,
      5                                                                 train_data=filedst_10_models_input,

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/automl/pipeline/steps/__init__.py in <module>
      6 
      7 """
----> 8 from .automl_pipeline_builder import AutoMLPipelineBuilder
      9 
     10 __all__ = ["AutoMLPipelineBuilder"]

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/automl/pipeline/steps/automl_pipeline_builder.py in <module>
     19 from azureml.core import ComputeTarget, Datastore, Environment, Experiment
     20 from azureml.data.data_reference import DataReference
---> 21 from azureml.pipeline.core import PipelineData, PipelineRun, PipelineStep
     22 from azureml.pipeline.steps import ParallelRunConfig, ParallelRunStep
     23 

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/pipeline/core/__init__.py in <module>
     22 service?](https://docs.microsoft.com/azure/machine-learning/concept-ml-pipelines)
     23 """
---> 24 from .builder import PipelineStep, PipelineData, StepSequence
     25 from .pipeline import Pipeline
     26 from .graph import PublishedPipeline, PortDataReference, OutputPortBinding, InputPortBinding, TrainingOutput

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/pipeline/core/builder.py in <module>
     12 from azureml.data._dataset import _Dataset
     13 from azureml.data.abstract_dataset import AbstractDataset
---> 14 from azureml.data.constants import _DATASET_ARGUMENT_TEMPLATE, _DATASET_OUTPUT_ARGUMENT_TEMPLATE
     15 from azureml.data.dataset_consumption_config import DatasetConsumptionConfig
     16 from azureml.data.output_dataset_config import OutputDatasetConfig

ImportError: cannot import name '_DATASET_OUTPUT_ARGUMENT_TEMPLATE'

!pip freeze

absl-py==0.11.0
adal==1.2.6
aiohttp==3.7.3
aiohttp-cors==0.7.0
aioredis==1.3.1
alembic==1.4.1
ansiwrap==0.8.4
antlr4-python3-runtime==4.7.2
applicationinsights==0.11.9
argcomplete==1.12.2
argon2-cffi==20.1.0
astor==0.8.1
astroid==2.4.2
astunparse==1.6.3
async-timeout==3.0.1
atari-py==0.2.6
attrs==20.3.0
autokeras==1.0.12
autopep8==1.5.5
azure-appconfiguration==1.1.1
azure-batch==10.0.0
azure-cli==2.19.1
azure-cli-core==2.19.1
azure-cli-telemetry==1.0.6
azure-common==1.1.26
azure-core==1.10.0
azure-cosmos==3.2.0
azure-datalake-store==0.0.51
azure-functions-devops-build==0.0.22
azure-graphrbac==0.61.1
azure-identity==1.2.0
azure-keyvault==1.1.0
azure-keyvault-administration==4.0.0b1
azure-loganalytics==0.1.0
azure-mgmt-advisor==2.0.1
azure-mgmt-apimanagement==0.2.0
azure-mgmt-appconfiguration==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.61.0
azure-mgmt-batch==9.0.0
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==1.0.0
azure-mgmt-botservice==0.3.0
azure-mgmt-cdn==5.2.0
azure-mgmt-cognitiveservices==6.3.0
azure-mgmt-compute==18.2.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-containerservice==9.4.0
azure-mgmt-core==1.2.2
azure-mgmt-cosmosdb==1.0.0
azure-mgmt-databoxedge==0.2.0
azure-mgmt-datalake-analytics==0.2.1
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==4.1.0
azure-mgmt-deploymentmanager==0.2.0
azure-mgmt-devtestlabs==4.0.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==3.0.0rc7
azure-mgmt-eventhub==4.1.0
azure-mgmt-hdinsight==2.2.0
azure-mgmt-imagebuilder==0.4.0
azure-mgmt-iotcentral==3.0.0
azure-mgmt-iothub==0.12.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==2.2.0
azure-mgmt-kusto==0.3.0
azure-mgmt-loganalytics==8.0.0
azure-mgmt-managedservices==1.0.0
azure-mgmt-managementgroups==0.2.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.2.1
azure-mgmt-media==2.2.0
azure-mgmt-monitor==2.0.0
azure-mgmt-msi==0.2.0
azure-mgmt-netapp==0.15.0
azure-mgmt-network==17.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.5.0
azure-mgmt-privatedns==0.1.0
azure-mgmt-rdbms==3.1.0rc1
azure-mgmt-recoveryservices==0.4.0
azure-mgmt-recoveryservicesbackup==0.11.0
azure-mgmt-redhatopenshift==0.1.0
azure-mgmt-redis==7.0.0rc2
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.6.0
azure-mgmt-resource==10.2.0
azure-mgmt-search==8.0.0
azure-mgmt-security==0.6.0
azure-mgmt-servicebus==0.6.0
azure-mgmt-servicefabric==0.5.0
azure-mgmt-signalr==0.4.0
azure-mgmt-sql==0.26.0
azure-mgmt-sqlvirtualmachine==0.5.0
azure-mgmt-storage==11.2.0
azure-mgmt-synapse==0.6.0
azure-mgmt-trafficmanager==0.51.0
azure-mgmt-web==0.48.0
azure-multiapi-storage==0.5.2
azure-nspkg==3.0.2
azure-storage-blob==12.7.1
azure-storage-common==1.4.2
azure-storage-queue==12.1.5
azure-synapse-accesscontrol==0.2.0
azure-synapse-artifacts==0.3.0
azure-synapse-spark==0.2.0
azureml-accel-models==1.22.0
azureml-automl-core==1.23.0
azureml-automl-runtime==1.23.0
azureml-cli-common==1.22.0
azureml-contrib-automl-pipeline-steps==1.23.0
azureml-contrib-dataset==1.22.0
azureml-contrib-fairness==1.22.0
azureml-contrib-gbdt==1.22.0
azureml-contrib-interpret==1.22.0
azureml-contrib-notebook==1.22.0
azureml-contrib-pipeline-steps==1.22.0
azureml-contrib-reinforcementlearning==1.22.0
azureml-contrib-server==1.22.0
azureml-contrib-services==1.22.0
azureml-core==1.23.0
azureml-datadrift==1.22.0
azureml-dataprep==2.10.1
azureml-dataprep-native==30.0.0
azureml-dataprep-rslex==1.8.0
azureml-dataset-runtime==1.23.0
azureml-defaults==1.23.0
azureml-explain-model==1.22.0
azureml-interpret==1.23.0
azureml-mlflow==1.22.0
azureml-model-management-sdk==1.0.1b6.post1
azureml-opendatasets==1.10.0
azureml-pipeline==1.23.0
azureml-pipeline-core==1.23.0
azureml-pipeline-steps==1.23.0
azureml-samples @ file:///mnt/jupyter-azsamples
azureml-sdk==1.23.0
azureml-telemetry==1.23.0
azureml-tensorboard==1.22.0
azureml-train==1.23.0
azureml-train-automl==1.23.0
azureml-train-automl-client==1.23.0.post1
azureml-train-automl-runtime==1.23.0.post1
azureml-train-core==1.23.0
azureml-train-restclients-hyperdrive==1.23.0
azureml-widgets==1.22.0
backcall==0.2.0
backports.functools-lru-cache==1.6.1
backports.tempfile==1.0
backports.weakref==1.0.post1
bcrypt==3.2.0
beautifulsoup4==4.9.3
bleach==3.3.0
blessings==1.7
blis==0.2.4
bokeh==2.2.3
boto==2.49.0
boto3==1.15.18
botocore==1.18.18
Bottleneck==1.3.2
cached-property==1.5.2
cachetools==4.2.1
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.4
colorful==0.5.4
configparser==3.7.4
contextlib2==0.6.0.post1
contextvars==2.4
convertdate @ file:///home/conda/feedstock_root/build_artifacts/convertdate_1605102623033/work
coremltools @ git+https://github.com/apple/coremltools@13c064ed99ab1da7abea0196e4ddf663ede48aad
cryptography==3.2
cycler==0.10.0
cymem==2.0.5
Cython @ file:///tmp/build/80754af9/cython_1594831565616/work
databricks-cli==0.14.1
dataclasses==0.8
decorator==4.4.2
defusedxml==0.6.0
dill==0.3.3
distro==1.5.0
dm-tree==0.1.5
docker==4.4.1
dotnetcore2==2.1.20
en-core-web-sm @ https://aka.ms/automl-resources/packages/en_core_web_sm-2.1.0.tar.gz
encrypted-inference==0.9
entrypoints==0.3
enum34==1.1.10
fabric==2.6.0
fairlearn==0.4.6
fastai==1.0.61
fastprogress==1.0.0
fbprophet==0.5
filelock==3.0.12
fire==0.4.0
flake8==3.8.4
Flask==1.0.3
Flask-Cors==3.0.10
flatbuffers==1.12
fusepy==3.0.1
future==0.18.2
gast==0.2.2
gensim==3.8.3
gevent==21.1.2
gitdb==4.0.5
GitPython==3.1.13
google-api-core==1.26.0
google-auth==1.26.1
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
googleapis-common-protos==1.52.0
gpustat==0.6.0
greenlet==1.0.0
grpcio==1.35.0
gunicorn==19.9.0
gym==0.18.0
h5py==2.10.0
hiredis==1.1.0
holidays==0.9.11
horovod==0.19.1
humanfriendly==9.1
idna==2.10
idna-ssl==1.1.0
imageio==2.9.0
immutables==0.15
importlib-metadata==3.4.0
interpret-community==0.16.0
interpret-core==0.2.1
invoke==1.5.0
ipykernel==5.4.3
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
isodate==0.6.0
isort==5.7.0
itsdangerous==1.1.0
javaproperties==0.5.1
jedi==0.18.0
jeepney==0.6.0
Jinja2==2.11.2
jmespath==0.10.0
joblib==0.14.1
jsmin==2.2.2
json-logging-py==0.2
json5==0.9.5
jsondiff==1.2.0
jsonpickle==1.5.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.11
jupyter-console==6.2.0
jupyter-core==4.7.1
jupyter-server-proxy==1.6.0
jupyterlab==2.1.4
jupyterlab-nvdashboard==0.4.0
jupyterlab-server==1.2.0
jupyterlab-widgets==1.0.0
jupytext==1.6.0
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
keras-tuner==1.0.2
keras2onnx==1.6.0
kiwisolver==1.3.1
knack==0.8.0rc2
lazy-object-proxy==1.4.3
liac-arff==2.5.0
lightgbm==2.3.0
lunardate==0.2.0
lz4==3.1.3
Mako==1.1.4
Markdown==3.3.3
markdown-it-py==0.5.8
MarkupSafe==1.1.1
matplotlib==3.2.1
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.2.0
mkl-random==1.1.0
mkl-service==2.3.0
mlflow==1.13.1
mock==4.0.3
msal==1.8.0
msal-extensions==0.1.3
msgpack==1.0.2
msrest==0.6.21
msrestazure==0.6.4
multidict==5.1.0
murmurhash==1.0.5
nbconvert==5.6.1
nbformat==5.1.2
ndg-httpsclient==0.5.1
networkx==2.5
nimbusml==1.8.0
notebook==6.2.0
numexpr==2.7.2
numpy==1.18.5
nvidia-ml-py3==7.352.0
oauthlib==3.1.0
olefile==0.46
onnx==1.7.0
onnxconverter-common==1.6.0
onnxmltools==1.4.1
onnxruntime==1.3.0
opencensus==0.7.12
opencensus-context==0.1.2
opencv-python==4.5.1.48
opencv-python-headless==4.3.0.36
opt-einsum==3.3.0
packaging==20.9
pandas==0.25.3
pandas-ml==0.6.1
pandocfilters==1.4.3
papermill==1.2.1
paramiko==2.7.2
parso==0.8.1
pathlib2==2.3.5
pathspec==0.8.1
patsy==0.5.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.1.0
pkginfo==1.7.0
plac==0.9.6
pluggy==0.13.1
pmdarima==1.1.1
portalocker==1.7.1
preshed==2.0.1
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
prompt-toolkit==3.0.14
protobuf==3.14.0
psutil==5.8.0
psycopg2==2.8.4
ptyprocess==0.7.0
py-cpuinfo==5.0.0
py-spy==0.3.4
py4j==0.10.9
pyarrow==1.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.0
pycodestyle==2.6.0
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pydocstyle==5.1.1
pyflakes==2.2.0
pyglet==1.5.0
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1610742651773/work
PyJWT==1.7.1
pylint==2.6.0
PyMeeus @ file:///home/conda/feedstock_root/build_artifacts/pymeeus_1589222711601/work
PyNaCl==1.4.0
pynvml==8.0.4
pyodbc===4.0.0-unsupported
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyrsistent==0.17.3
pyspark==3.0.1
pystan==2.19.0.0
python-dateutil==2.8.1
python-editor==1.0.4
python-jsonrpc-server==0.4.0
python-language-server==0.35.0
pytorch-transformers==1.0.0
pytz==2021.1
PyWavelets==1.1.1
PyYAML==5.4.1
pyzmq==22.0.2
qtconsole==5.0.2
QtPy==1.9.0
QuantLib==1.21
querystring-parser==1.2.4
ray==1.2.0
redis==3.5.3
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
rope==0.18.0
rsa==4.7
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
s3transfer==0.3.4
sacremoses==0.0.43
scikit-image==0.17.2
scikit-learn==0.22.2.post1
scipy==1.4.1
scp==0.13.3
scrapbook==0.5.0
SecretStorage==3.3.0
Send2Trash==1.5.0
sentencepiece==0.1.95
setuptools-git==1.2
shap==0.34.0
simpervisor==0.4
sip==4.19.24
six==1.15.0
skl2onnx==1.4.9
sklearn==0.0
sklearn-pandas==1.7.0
smart-open==1.9.0
smmap==3.0.5
snowballstemmer==2.1.0
soupsieve==2.2
spacy==2.1.8
SQLAlchemy==1.3.23
sqlparse==0.4.1
srsly==1.0.5
sshtunnel==0.1.5
statsmodels==0.10.2
style==1.1.0
tabulate==0.8.7
tenacity==6.3.1
tensorboard==2.1.1
tensorboard-plugin-wit==1.8.0
tensorboardX==2.1
tensorflow==2.1.0
tensorflow-estimator==2.1.0
tensorflow-gpu==2.1.0
termcolor==1.1.0
terminado==0.9.2
terminaltables==3.1.0
testpath==0.4.4
textwrap3==0.9.2
thinc==7.0.8
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tifffile==2020.9.3
tokenizers==0.10.1
toml==0.10.2
torch==1.6.0
torchvision==0.7.0
tornado==6.1
tqdm==4.56.0
traitlets==4.3.3
transformers==4.3.2
typed-ast==1.4.2
typing-extensions==3.7.4.3
ujson==4.0.2
update==0.0.1
urllib3==1.25.11
vsts==0.1.25
vsts-cd-manager==1.0.2
waitress==1.4.4
wasabi==0.8.2
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
webencodings==0.5.1
websocket-client==0.57.0
websockets==8.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
xgboost==1.3.3
xmltodict==0.12.0
yapf==0.30.0
yarl==1.6.3
zipp==3.4.0
zope.event==4.5.0
zope.interface==5.2.0

[cartacioS]

Hi @saleung - we've published an improved Many Models notebook, can you please try this and see if you can overcome the issue? We will move away from the existing notebook experience soon.

New Notebook; https://github.com/Azure/azureml-examples/tree/main/python-sdk/tutorials/automl-with-azureml/forecasting-many-models