Crash when trying to set xlim on datetime axis

[TheoMathurin]

HoloViews 1.17.0
Bokeh 3.2.1

The condition here introduced in #5621 yields an error for a datetime axis:
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''.

Replacing util.cftime_types with util.datetime_types makes it work. I can make a PR if it's the right fix.

Minimal example:

import pandas as pd
import datetime as dt
import holoviews as hv

df = pd.DataFrame({'time': [pd.Timestamp('2023-07-27T09:00:00Z'), pd.Timestamp('2023-07-27T15:00:00Z')], 'value': [5, 6]})
xlim = (dt.datetime(2023, 7, 27, 0), dt.datetime(2023, 7, 28, 0))

hv.Curve(df, 'time', 'value').opts(xlim=xlim)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.