AssertionError: assert self.num_slices > 0

[yapolyak]

Dear cuQuantum devs,

My turn to report a strange crash of a circuit contraction :)

Basically I am doing a super-simple thing in serial - try to find the contraction path for some circuit tensor network representation:

with cq.Network(*ovl) as network:
    path, info = network.contract_path({'samples': 50})

where ovl is my circuit tensor-network interleaved representation.

When I run above, I get a strange assertion error, saying that I have less than 1 slice. I tried using Logger level 6, and got enormous amount of printouts, with the final n lines being not very informative (at least to me):

3245=[type=5 modes=[760,795,778,1071,1073,1038,994,1022,1351,1431,1336,1409,814,812,1034,1033,1018,1002,774,991,976,756,740,636,] extents=[2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2] strides=[1,2,4,8,16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608] op=1 alignment=256] [isConstant=0][requiresGrad=0][requiresCache=0]
3246=[type=5 modes=[] extents=[] strides=[] op=1 alignment=256] [isConstant=0][requiresGrad=0][requiresCache=0]
 ]

[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetContractionOptimizerInfoGetAttribute] handle=0X55BAE907B690 optimizerInfo=0X55BAE9016440 attr=10 buf=0X55BB2312A400 sizeInBytes=8
[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetDestroyContractionOptimizerConfig] optimizerConfig=0X55BAF6F64D10
[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetDestroyContractionOptimizerInfo] optimizerInfo=0X55BAE9016440
[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetDestroyWorkspaceDescriptor] workDesc=0X55BAF548D360
[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetDestroyNetworkDescriptor] desc=0X55BB0F3EB240
[2023-07-27 05:32:09][cuTensorNet][169906][Api][cutensornetDestroy] handle=0X55BAE907B690
[2023-07-27 05:32:09][cuTensorNet][170133][Info][contractionOptimize] thread pool receives a signal 2, interrupting...
Traceback (most recent call last):
  File "/global/u1/i/ipolyak/Henriks_circuits/test.py", line 16, in <module>
    path, info = network.contract_path({'samples': 50})
  File "/global/homes/i/ipolyak/.conda/envs/py-cuquantum-23.06.0-mypich-py3.9/lib/python3.9/site-packages/cuquantum/cutensornet/_internal/utils.py", line 474, in inner
    result = wrapped_function(*args, **kwargs)
  File "/global/homes/i/ipolyak/.conda/envs/py-cuquantum-23.06.0-mypich-py3.9/lib/python3.9/site-packages/cuquantum/cutensornet/_internal/utils.py", line 443, in inner
    raise e
  File "/global/homes/i/ipolyak/.conda/envs/py-cuquantum-23.06.0-mypich-py3.9/lib/python3.9/site-packages/cuquantum/cutensornet/_internal/utils.py", line 435, in inner
    result = wrapped_function(*args, **kwargs)
  File "/global/homes/i/ipolyak/.conda/envs/py-cuquantum-23.06.0-mypich-py3.9/lib/python3.9/site-packages/cuquantum/cutensornet/tensor_network.py", line 555, in contract_path
    assert self.num_slices > 0
AssertionError

Since this are commands directly from your manual/example, I can only suggest there is something wrong with my tensor network, although it Should be fine. I am converting other circuits in the same way and they work fine. I am currently trying to figure out if I can share the TN representation of this (unfortunately massive) circuit with you.

Thanks!!

[yapolyak]

Interesting.. If I perform some optimisation (modification) of this circuit, I instead get this:

[2023-07-27 06:49:54][cuTensorNet][1900374][Error][cutensornetContractionOptimize] No valid path could be found with specified constraints.
[2023-07-27 06:51:07][cuTensorNet][1900374][Api][cutensornetGetErrorString] error=24: CUTENSORNET_STATUS_ALL_HYPER_SAMPLES_FAILED

No valid path... What are the constraints the message is telling me? Number of samples?

[yapolyak]

Double-checked: number of requested samples doesn't matter, it obviously fails on the first sample.

[yapolyak]

I tried to revv up the reconfiguration settings:

path, info = network.contract_path({'samples': 50, 'reconfiguration': {'num_iterations': 1000, 'num_leaves': 15}})

it took way longer, but, alas, same failure ("No valid path could be found with specified constraints.") and (seemingly randomly) this num_slices > 0 assertion error...

[yangcal]

Hello,

Can you share your network? You could just pickle the shape and the indices you have using the interleaved format. BTW, what GPU do you have for the calculation?

[yapolyak]

Hi!

I got permission to share the full interleaved representation (ovl in my above scripts), please find attached!
I am using the 40GB A100 on Perlmutter!

Many thanks for looking into it!
troubledTN.tar.gz

[yangcal]

Sorry for dropping the ball here. I can reproduce the issue and we're looking at it. Can you tell us a bit about the tensor network? Is it a "raw" network directly converted from a standard quantum circuit or did you do any preprocessing (likely decomposition, simplification) on top of it? Is bitstring amplitude the target compute quantity?

[yapolyak]

No problem at all and thank you very much for looking into this!

It is a raw circuit without any optimisation done (although when I tried some tket optimisation on it, resulting circuit would also not have its contraction path calculated): 56 qubits, 728 gates, out of which there are 280 Rx gates, and 448 ZZPhase gates. If I first compile it with tket, I get more gates, including TK1 and CX. What I try to contract is an overlap, i.e. a scalar and a 2x longer circuit.

One thing I noticed is that we first convert it from circ to tket. Perhaps I should try your own circ->TN converter?

If it helps, I could probably share the circuit itself?

[yapolyak]

We also plan to implement building RDMs using your new API and check if this is more efficient than direct conversion we do now.

[mtjrider]

@yapolyak do you still have any questions related to this topic?

[haidarazzam]

Dear @yapolyak,
Thank you very much for your patience and report of the behavior.
We were able to reproduce the issue and working on it to find a possible fix.
Currently I can propose a workaround for it but it will slowdown the pathfinder (3x-4x).
Please set the environment variable export CUTENSORNET_LEGACY_OPTIMAL=1

One trick to speed it up a little bit the pathfinder is to set CUTENSORNET_CONTRACTION_OPTIMIZER_CONFIG_RECONFIG_NUM_LEAVES=6.
The later could affect the quality of the path, but for QAOA or simple path, i didn't see any effect.
Thanks
Azzam

[haidarazzam]

@yapolyak
did the answer above "Please set the environment variable export CUTENSORNET_LEGACY_OPTIMAL=1" helped you resolve the issue?
Thanks