Missing support for numpy.record dtype

[JonathanAnderson]

I have a npy file that I cannot read. I keep getting an error saying parsing header failed: expected character ''', found '['.

The file can be created as follows...

df = pd.DataFrame({"i": [1, 2, 3],"j": [4, 5, 6],"k":[3.14, 2.72, 1.62]})
arr = np.ascontiguousarray(df.to_records(index=False)).view()
np.save("/tmp/temp.npy", arr)

or alternatively just writing the bytes of the file from julia

bytes = UInt8[0x93, 0x4e, 0x55, 0x4d, 0x50, 0x59, 0x01, 0x00, 0x66, 0x00, 0x7b, 0x27, 0x64, 0x65, 0x73, 0x63, 0x72, 0x27, 0x3a, 0x20, 0x5b, 0x28, 0x27, 0x69, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x69, 0x38, 0x27, 0x29, 0x2c, 0x20, 0x28, 0x27, 0x6a, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x69, 0x38, 0x27, 0x29, 0x2c, 0x20, 0x28, 0x27, 0x6b, 0x27, 0x2c, 0x20, 0x27, 0x3c, 0x66, 0x38, 0x27, 0x29, 0x5d, 0x2c, 0x20, 0x27, 0x66, 0x6f, 0x72, 0x74, 0x72, 0x61, 0x6e, 0x5f, 0x6f, 0x72, 0x64, 0x65, 0x72, 0x27, 0x3a, 0x20, 0x46, 0x61, 0x6c, 0x73, 0x65, 0x2c, 0x20, 0x27, 0x73, 0x68, 0x61, 0x70, 0x65, 0x27, 0x3a, 0x20, 0x28, 0x33, 0x2c, 0x29, 0x2c, 0x20, 0x7d, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x0a, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x1f, 0x85, 0xeb, 0x51, 0xb8, 0x1e, 0x09, 0x40, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc3, 0xf5, 0x28, 0x5c, 0x8f, 0xc2, 0x05, 0x40, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xec, 0x51, 0xb8, 0x1e, 0x85, 0xeb, 0xf9, 0x3f]
open("/tmp/temp.npy", "w") do f
    write(f, bytes)
end

Then i read the file the usual way...

npzread("/tmp/temp.npy")

and this is my stacktrace

parsing header failed: expected character ''', found '['

Stacktrace:
 [1] parsechar(::String, ::Char) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:70
 [2] parsestring(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:76
 [3] parsedtype(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:123
 [4] parseheader(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:157
 [5] npzreadarray(::IOStream) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:191
 [6] npzread(::String) at /home/janders/.julia/v0.6/NPZ/src/NPZ.jl:218

versioninfo():

Julia Version 0.6.1-pre.0
Commit dcf39a1 (2017-06-19 13:06 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
  WORD_SIZE: 64
  BLAS: libmkl_rt
  LAPACK: libmkl_rt
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

And my NPZ package has head at

* 60c4bc9 2017-02-01 (HEAD, tag: v0.2.0, origin/master, cache/heads/master, master) REQUIRE: downgrade Compat because we don't use Compat.view [Fazlul Shahriar]

[fhs]

numpy.record dtype is not yet supported. You can convert it to a normal array and save it:

df = pd.DataFrame({"i": [1, 2, 3],"j": [4, 5, 6],"k":[3.14, 2.72, 1.62]})
np.save("/tmp/temp.npy", df.values)