Unable to connect to ignite with persistence enabled

[siddharth-u]

Hi
I am trying to run ignite in a cluster on kubernetes. My service tries to connect to the running ignite pod. But with ignite 2.16, the connection fails when persistence is enabled. The same operation works with persistence disabled with version 2.16 and also works with the previous versions irrespective of whether persistence is enabled or disabled.
I use thinClient to connect to ignite from my service

logsof my service -

org.apache.ignite.client.ClientException: Ignite failed to process request [24]: 50000: Failed to execute map query on remote node [nodeId=514c2e13-805e-4b75-8811-9b2e8343b44b, errMsg=General error: "class org.apache.ignite.binary.BinaryObjectException: Cannot find metadata for object with compact footer (Ignite work directory might have been cleared after restart. Make sure that IGNITE_HOME does not point to a temp folder or any other folder that is destroyed/cleared on restarts) [typeId=396738638, IGNITE_HOME='/opt/ignite/apache-ignite']"; SQL statement:
SELECT
PUBLIC.__Z0._KEY __C0_0,
PUBLIC.__Z0._VAL __C0_1,
__Z0.LASTUPDATED __C0_2
FROM PUBLIC.ABC __Z0
WHERE __Z0.ABCDEF = ?1
ORDER BY 3 LIMIT ?2 [50000-197]] (server status code [1])
	at org.apache.ignite.internal.client.thin.TcpClientChannel.convertException(TcpClientChannel.java:475) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:421) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:311) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ReliableChannel.lambda$service$0(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:835) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:788) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:205) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:186) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.GenericQueryPager.next(GenericQueryPager.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.nextPage(ClientQueryCursor.java:93) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at org.apache.ignite.internal.client.thin.ClientQueryCursor$1.hasNext(ClientQueryCursor.java:76) ~[ignite-core-2.16.0.jar!/:2.16.0]
	at java.base/java.lang.Iterable.forEach(Unknown Source) ~[na:na]
	at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery$QueryCursorWrapper.getAll(IgniteRepositoryQuery.java:963) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
	at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.transformQueryCursor(IgniteRepositoryQuery.java:616) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
	at org.apache.ignite.springdata.repository.query.IgniteRepositoryQuery.execute(IgniteRepositoryQuery.java:360) ~[ignite-spring-data-ext-2.0.0.jar!/:2.0.0]
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:136) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:120) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:164) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:143) ~[spring-data-commons-3.0.5.jar!/:3.0.5]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244) ~[spring-aop-6.0.11.jar!/:6.0.11]
	at jdk.proxy2/jdk.proxy2.$Proxy129.getGoodABCs(Unknown Source) ~[na:na]
	at  ~[classes!/:na]
	at reactor.core.publisher.MonoCallable.call(MonoCallable.java:72) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.publisher.MonoFlatMapMany.subscribeOrReturn(MonoFlatMapMany.java:49) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.9.jar!/:3.5.9]
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

[siddharth-u]

To summarise:

Ignite version < 2.16 - works with persistence enabled
Ignite version 2.16 - fails with above error for persistence enabled