File size / file name limitation?

[AutomationD]

Is there any known file size or file name limitation that would prevent loading certain files?

Thank you

Cross-issue xlfe/spiffy#7

[raz123]

File name is limited to 32 chars, IIRC.

[AutomationD]

Thank you for your reply, @raz123
Our file names are less than 32 chars. See spiffy log output:

Creating rom spiff_rom.bin of size 196608 bytes
Adding files in directory files
Unable to read file .
Unable to read file ..
bootstrap.css.gz added to spiffs (15615 bytes)
index.html added to spiffs (2876 bytes)
jquery.js.gz added to spiffs (30153 bytes)

[pellepl]

The default config for filename size is 32 chars, true. I do not think it is changed on the nodemcu project. File sizes - well, the max length depends on pretty many things. The length field is 32 bits, so that's not it.

Then, I think the span field config on nodemcu is 16 bits with pages of 256 bytes, giving roughly 250*65536 bytes max, so that's cool also.

And then I think, filesystem has roughly 40k of spi flash to play with. The spiff_rom.bin of 196608 bytes seems large in this sense.

I'll have to check what spiffy actually does.

Edit:
The files you try to add, do they sum up to 15615 + 2876 + 30153 bytes? Unless you haven't changed the spiffs/nodemcu config, that is too much.

Edit2:
Quickly glanced over spiffy. It seems you've tinkered with the spiffy.c MAX_FILE define, yes? Some nodemcu knowledgeble guy should confirm this, but as far as I know this can maximum be 4096*20.

[pellepl]

Ping?

[raz123]

Pong? :)

[pellepl]

Sorry for undirected ping, raz :) It was intended to @kireevco

[alonewolfx2]

we are using spiffs on sming. not nodemcu. spiffy using "cfg.log_block_size = SPI_FLASH_SEC_SIZE" main problem is this. if we change "cfg.log_block_size = SPI_FLASH_SEC_SIZE*2" finally we can use ~142kb on filesystem. also spiffs failing if total files bigger than ~142kb. (we have free 192kb on filesystem)
btw: here is sming source. https://github.com/anakod/Sming

[pellepl]

Ah I see. What errorcodes do you get? I'll see if I can browse your code
this evening.

2015-04-17 9:04 GMT+02:00 alonewolfx2 [email protected]:

we are using spiffs on sming. not nodemcu. spiffy using
"cfg.log_block_size = SPI_FLASH_SEC_SIZE" main problem is this. if we
change "cfg.log_block_size = SPI_FLASH_SEC_SIZE*2" finally we can use
~142kb on filesystem. also spiffs failing if total files bigger than
~142kb. (we have free 192kb on filesystem)
btw: here is sming source. https://github.com/anakod/Sming

—
Reply to this email directly or view it on GitHub
#12 (comment).

[alonewolfx2]

"write errno -10001" its appearing after ~141kb. if you can take a look, we will be happy

[anakod]

Take this opportunity to say thank you for the excellent work, pellepl!

[alonewolfx2]

@anakod agree :) @pellepl thanks for excellent work

[pellepl]

blush Thanks for the kind words! Perhaps you should de-emphasize the praise until I've found the problem ;) It occurs to me that this might not be a bug, but what you see is a replica of nodemcu/nodemcu-firmware#309 (see lengthy comment by me somewhere in the middle)

Anyhow, in SmingFramework/Services/SpifFS/spiffs.c you have a spiffs_mount() function. I need to know exactly what is in you cfg struct before you send it to SPIFFS_mount().

Also, are you running the latest and greatest spiffs version? Seems to be some parts missing in the config for the new stuff.

My hopes are that spiffs could be tuned a wee bit better for the ESP8266 altogether, for both nodemcu and sming by simply configuring it differently.

[anakod]

@pellepl I have a question:
https://github.com/anakod/Sming/blob/master/Sming/Services/SpifFS/spiffs.c#L56
Thisout that change (* 2) I can't write to the end of filesystem (it's crashed). Why? Is it right fix, or we should change something else?

[pellepl]

Hmm, it should definitely not crash. Did you get a dump or a trace? If there is a bug you've found, any info would help.

In my testbench it works filling the FS with what I presume are the same params - I still need to know the exact contents of the config struct Sming sends into SPIFFS_mount().

Other than that, it is fully legal to have a logical block size that is two times the size of a physical sector. It's all in the https://github.com/pellepl/spiffs/blob/master/docs/TECH_SPEC. In short, it works like this:

The SPI flash has sector boundaries - say 4096 bytes as in the ESP8266 case. Spiffs groups the SPI flash into logical blocks, which must be on sector size boundary, in our case e.g. 4096, 8192, etc. Each block is then divided into pages, in your config 256 bytes. So, having a sector size of 4096, a block size of 8192, a page size of 256 gives 4096*2/256 = 32 pages.

Also, each logical block contains a meta data look up table, stealing at least one page per block. In the case of having a block size of 4096 and a page size of 256, 1/16th or 6.25% of the SPI flash is used for LUTs. Having a block size of 8192 instead, 1/32th or 3.125% if the SPI flash is used for LUTs. The former case is not a very optimal configuration, because only 25% of the LUT will be used as there are only 15 actual file data pages per block to index. The latter case is a better solution.

Still it must not crash. It is confusing, because the nodemcu project uses this configuration (4096 bytes block, 256 bytes pages) without problems (?). Is sming using the latest version? Any details you could provide would be great.

[alonewolfx2]

here is cfg config. https://github.com/anakod/Sming/blob/master/Sming/Services/SpifFS/spiffs_config.h#L168

[pellepl]

Yes, the static config I've found. The runtime I need from someone with a board.
At https://github.com/anakod/Sming/blob/master/Sming/Services/SpifFS/spiffs.c#L108, I need the contents of the cfg struct. Preferable in a build that crashes.

[alonewolfx2]

@pellepl if you preffer any change i can compile and test on the board

[pellepl]

Ok great, before SPIFFS_mount at https://github.com/anakod/Sming/blob/master/Sming/Services/SpifFS/spiffs.c#L108, insert
debugf("spiffs.phys_size: %d\n", cfg.phys_size);
debugf("spiffs.phys_addr: 0x%08x\n", cfg.phys_addr);
debugf("spiffs.phys_era_blk: %d\n", cfg.phys_erase_block);
debugf("spiffs.log_blk: %d\n", cfg.log_block_size);
debugf("spiffs.log_pg: %d\n", cfg.log_page_size);

and attach the output here I can start digging for real.

[alonewolfx2]

here we go

fs.start: size:192 Kb, offset:4024C000

spiffs.phys_size: 196608

spiffs.phys_addr: 0x4024c000

spiffs.phys_era_blk: 4096

spiffs.log_blk: 8192

spi

[pellepl]

Thanks! But what happened to the log_pg output?

[alonewolfx2]

it seems module trying to connect network before that line. i will disable network and try again. wait a minute please

[alonewolfx2]

here is it.

fs.start: size:192 Kb, offset:4024C000
spiffs.phys_size: 196608
spiffs.phys_addr: 0x4024c000
spiffs.phys_era_blk: 4096
spiffs.log_blk: 8192
spiffs.log_pg: 256

[pellepl]

Cheers! Will dig into it asap.

[dobryj]

I don't know if it independent problem or same, therefore I can add this into this thread.
I would like to test SPIFFS with small erase sector. It is test for Micron NOR flash 4kB subsectors. But SPIFFS tests didn't pass.
I change ONLY params_test.h and only this lines:
#define SECTOR_SIZE 8192
#define LOG_BLOCK 8192
#define LOG_PAGE 256

After "make test" I see to message about failed "long_run_config_many_medium"
And with real 4kB sector it is total lost. Let's have this config:
#define SECTOR_SIZE 4096
#define LOG_BLOCK 131072
#define LOG_PAGE 256

and result:
TEST 33/56 : running test file_uniqueness
creating 3072 files
checking 3072 files
removing 1536 files
creating 1536 files
make: *** [test] Segmentation fault

What I doing wrong?

[pellepl]

@dobryj In the first case with 8k blocks, the test fail since the fs runs out of space. The testcase is one of the most complicated ones, and was written to totally fill the system using 64k blocks. Having 8k blocks will add more metadata which steals more actual data, and presumably I was too lazy to cover all possible configs in the test case when writing it.
I would say you can safely ignore this failure.

As for the 128k block size with 4k sectors, that must be a bug in the test setup. I'll have to check it. It works with 128k block size with 64k sectors, which makes no sense, as the sector size should not affect in this way.

Edit: as suspected, a bug in the testbench. Fixed and done with: #13 - thanks for finding this

[dobryj]

Segmentation fault problem, from my comment can be solved by change one line in test_spiffs.c to
static int erases[PHYS_FLASH_SIZE / SECTOR_SIZE];

Problem was fixed size of array to 256. I it was enought only for default test configuration.

[pellepl]

@anakod Regarding the max file size of 141kB when using 192kB of space: this is not a bug.

Parts of the "lost" space is eaten by metadata. Other parts are reserved as free buffer in order to do garbage collecting. And the remaining data that cannot be written is due to the same reason as is mentioned in nodemcu/nodemcu-firmware#309. I was fooled a bit there, I thought the free size of the spiffs system was 192kB, not the physical free space.

[pellepl]

@anakod Regarding the crash you mentioned: was this in the testbench (same as @dobryj found) or was it on target? If in testbench, then it is fixed and pushed to master.

If it is on target, things are worse. I've been testing it quite hard here, both on another target and in the testbench - but I cannot reproduce it. Without more info it's hard to straighten out what could be wrong. I would at least need a stack trace or something to begin with.

In any case, that is another issue - if there was a target crash, please open another issue and provide all info you can give me. I get the feeling the ESP8266 is not very transparent, can anyone jtag it or are you guys bound to printf-debugging only?