You are viewing a static version of the site. Enable JavaScript to benefit from the
full functionality of this site and enable such things as forum posting.
File Uploading into SPIFFS for ESP32
ullix:
I wanted to upload a small file (html, 4kB) but found no tool under uecide. So I switched to my installation of Arduino IDE, installed the "ESP32 Sketch data upload", and uploaded the file. Easy.
Except that this bombed both my EEPROM area and my SPIFFS filesystem - all data gone! Why does this even happen, I thought the EEPROM area is separated from the SPIFFS area?
Then SPIFFS did show the uploaded file with correct name and size. I could open the file, and read it - except that it did not read anything, though it continued well beyond 100kB until I stopped the sketch.
So, not only had it not worked, but ruined EEPROM and SPIFFS at the same time, great.
Now I really need a tool working under uecide in order to upload files to my SPIFFS filesystem. What options do I have?
matt:
It is something that I have been working on. However, due to the way ESP32 and ESP8266 use flexible partitioning it's not a simple thing to do. I have already implemented building of a filesystem in UECIDE - both SPIFFS and FAT filesystems - but nothing yet for uploading of those filesystem images to the flash memory. It's that which is the hardest part.
You can, of course, use mkspiffs manually to build your filesystem, then using information from your selected partition format (look for the .csv files in the core) use esptool to upload them to the chip. I'm sure there's some tutorials for at least part of that online...
ullix:
Well, I am afraid this is beyond my pay grade :-/
My workaround was to implement within my sketches uploading via the "simple webserver". That at least saves the files to where they are supposed to go. If you have WiFi, WebServer, and SPIFFS anyway, that is rather easy to do. Though a tool in uecide would be nice.
Speaking of wishes: after having worked for a while with uecide, I have a couple of complaints, quibbles and wishes. Are you interested to hear, and where should they be posted?
matt:
Of course I'm interested to hear. I'm in the process of a complete rewrite of the GUI and some of the internals, so anything that can help steer it all in the direction that makes life easier for you users is always welcome. Posting here is as good a place as any...
ullix:
As you mentioned an alternative to the SPIFFS filesystem: I just experienced a major setback for my project due to Spiffs, and now I need to look for alternatives to store data on the flash.
I read about LittleFS but found no straight forward way to install it, neither on uecide, nor on the Arduino IDE. Did I miss an option?
Anything else already supported on uecide?
My problem: when I start with a freshly formatted flash, spiffs takes 3-5 ms to store a record (of some 50 bytes). With increasing fill level, the times for storing a record increases significantly. When the FS is near 50% full, the time increases to the 1000 ms range. At 60% it gets to 2000ms, and at 70% nothing gets stored anymore. It just continues to fail, all data are lost! That's no way to go on.Any help?
matt:
I haven't delved too deeply into the inner workings of SPIFFS, but I think it's a COW filesystem (Copy On Write). An existing file entry is invalidated by writing a bit low in the header, and a new file entry is created. For larger files I think this can start getting slower.
I think it was more designed for storing files intended for driving web pages, where the files are written just the once and then read many times.
Another part of the issue is that the SPIFFS is in the same flash as the code, using the same data channel, so it either has to be interleaved with the flow of code, or code has to be executed from elsewhere (RAM) to perform the write operations - and I guess that means the code needs copying out of flash and placing into RAM at some point (not idea when that would happen - maybe at startup, maybe on demand, I have no idea with those chips).
Still, all in all, SPIFFS on the same flash chip as the code is not really a great solution whatever the filesystem you use. You would probably be better off using an SD card or an extra flash or EEPROM chip (depending on data quantities).
ullix:
The SPIFFS belongs to the class of logging filesystems, at least according to the author(s) of LittleFS (https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md ). Every write is appended, so the wear level mitigation is perfect. The trouble comes when the flash is full and garbage collection is needed.
I had big trouble with spiffs and decided to investigate. I made my sketch write some 50 bytes every second to spiffs with opening, writing, flushing, and closing, and measured disk space usage and timings. Results in the attached picture.
The magenta line is the disk space usage. The light brown is the total saving time for each record of ~50 bytes. As you can see, this explodes at under 50% fs usage and reaches 2.5 sec to write 50 bytes! The Romans were faster carving letters into stone ;-).
Furthermore, the magenta line flattens in this area, meaning nothing is saved any more! You can't fill spiffs beyond <65% despite the fs telling me that 490kilo bytes are still available, and my records are only 50 bytes. spiffs does not even tell me that it could not write; it just does nothing and continues despite flushing and closing. The last 10+ hours the ESP has continued working but not a single byte has been written.
The bottom graph shows the read time for the first 100 bytes in the file (in red), and for the last 100 bytes (in black). Red is only a few ms, black is a sawtooth from 50 ms down to near zero, with occasional peaks at ~100ms. Not perfect, but reading would be ok.
But overall spiffs is just plain awful, at least for me.
I want to try LittleFS (https://github.com/ARMmbed/littlefs ). As they seem to have c-files only, can I even use it under uecide, and where would I have to put it?
ullix:
What a difference a filesystem makes: attached the results like above for SPIFFS now for FFat. A whole different univers; this fs can even be used!
I'd like to also test LittleFS, but it does not seem to be available for ESP32, only for ESP8266. You have any plans on this for uecide?
matt:
I have no idea what LittleFS is. If it's a library for ESP8266 then it's down to the library maintainer to port it to the ESP32.
ullix:
LittleFS seems to be truly bleeding edge, so I am surprised you haven't swallowed it ;-)
All sorts of good things are being claimed by its author, which I surely would like to verify. What I am certainly impressed about is how the author explains his design decisions based on on mathematical analysis. That is an approach I truly appreciated. So far I am going with FFat, which is at least way ahead of SPIFFS, that much I have already seen.
Read here: https://github.com/ARMmbed/littlefs/blob/master/DESIGN.md
matt:
The problem is, LittleFS is written for MBED not Arduino - so there would be massive amounts of work needed to port it to the Arduino framework.
ullix:
Too bad. But thanks for checking and explaining.
