-
Notifications
You must be signed in to change notification settings - Fork 134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RangeError: Invalid array length #208
Comments
Thanks for reporting. I've never really created such huge arrays in JS. Could you test if you can create an array with the same size (82 million elements) w/o using binary-parser? var dataView = new DataView(buffer.buffer, buffer.byteOffset, buffer.length);
var offset = 0;
var vars = {};
vars.data = [];
for (var $tmp0 = 82272642; $tmp0 > 0; $tmp0--) {
var $tmp1 = dataView.getFloat32(offset, true);
offset += 4;
vars.data.push($tmp1);
}
vars.dataLength = offset
return vars; |
This too fails, on the Code that works: data.data = new Float32Array(82272642);
for (var $tmp0 = 0; $tmp0 < 82272642; $tmp0++) {
var $tmp1 = dataView.getFloat32(offset, true);
offset += 4;
data.data[$tmp0] = $tmp1
}
data.dataLength = offset If I initialize the necessary float32 array, that it is fine. Also, never needs to realloc. This method btw, is 8 times faster, for an array of size 27,424,214, compared to the regular parsing. |
Ok, that makes sense. Reallocs are definitely an overhead, and I guess typed arrays are more compact than normal arrays. But this approach would only work for fixed-length arrays of primitive types. Is that what you are parsing? |
Yes, the largest arrays that I parse are indeed of fixed sizes (as in, I specify |
It turns out you can directly create a Float32Array from an ArrayBuffer (zero copy). Can you try if the following works? const dataParser = new Parser()
.buffer("data", {
length: 82272642 * 4, // length in bytes
formatter: (buf) => new Float32Array(buf.buffer) // buf is a DataView
})
.saveOffset('dataLength'); |
Hi @keichi I tried to write a test for it, but it passes, so I think it's out of my league to contribute here describe('Large arrays', () => {
it('should parse large array without error', () => {
const length = 80_000_000;
const array = Buffer.from(new Float32Array(length).fill(0).buffer);
const parser = new Parser()
.array("data", {
type: "floatle",
length
});
const buffer = factory(array);
doesNotThrow(() => parser.parse(buffer));
})
}) |
I am trying to parse an a float array.
Normally, this code works, however, I now have one huge file (400MB of file), and I want to start reading it.
As you can see, I am trying to parse an array with 82 million entries, which is less than the 2147483647 limit in javascript, however, I am getting the following error:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length
Additional information
Manual experimentation finds that the limit is somewhere between
50,100,000
and50,500,000
.(related to sign/translate#44)
The text was updated successfully, but these errors were encountered: