No announcement yet.

Optimize file_Open()

  • Filter
  • Time
  • Show
Clear All
new posts

  • Optimize file_Open()


    I am working on a project were a lot of text files are stored on the SD card of my 4D Display. So far, ~200 files, but count will increase over time.

    On a user action, I need to :
    - Open a specific file
    - Get some data
    - Close file
    > Repeat these operations up to 40 times, with different files
    - Update my UI

    After some tests, I found that the function file_Open() was rather slow and unpredictable. The execution of this function can take anything from 5ms to 20ms.
    While this can seem reasonable, it is very long in specific case since I have to call it 40 times : 40 x 200ms = 800ms, creating a laggy experience for the user.

    I guess the delay, and the fact the duration is not always the same, can be caused by some of the followings:
    - SD card speed : I use a 4GB Class 4 SD. Would a higher class made a big difference ?
    - SD cart format : I use FAT
    - The number of files on the SD : so far, between my program, some user data and text files, I have ~250 files
    - The date of creation of the file : it seems that the opening duration can be linked to the date of creation of the file. Maybe because of the way the files are sorted ?

    All that being said, is there a way to increase the efficiency of file_Open() ?
    If my last point is correct (date of creation), is there a way to fake the creation of one file, so that it gets first in the list and opened quicker ?


  • #2

    That's a fairly large number of files to open once a user triggers something. Would it be possible to combine those into a lesser number? And may I know why do you need to separate those data into 40 different files?

    Perhaps, you could store data that are often used into a buffer then retrieve as needed. You could also try to store the text files into Flashbanks for faster retrieval. Here's an appnote which explains this in detail:
    AN-00149 ViSi – Read and Write Data on Diablo16 Flashbanks

    Regarding the reason why you're getting various execution time, I believe it has something to do with the file system(FAT16) rather than the file_Open() function per se. Please follow this link which describes how a file is stored and organized in FAT16 file system. It would also depend on the size of the file being read. For comparison, a leaflet would take less time to read versus a book with a hundred page.

    If my last point is correct (date of creation), is there a way to fake the creation of one file, so that it gets first in the list and opened quicker?
    Although this wouldn't change the arrangement of datablocks inside the memory storage(sdCard), there's a way to change/modify the date of creation of the file. You could use file_SetDate() to modify the date and time on an open file handle. For more details on this function, please see DIABLO16 Internal Functions Manual page 340.

    I hope this information helps.

    Kind regards,


    • #3
      Hi Sherwin,

      Thanks for your message.
      As I expected, it seems that there is not much that I can do.
      The flashbank will not provide enough space to copy my files, as I have ~6MB of data.

      There are a few points that I can optimize on my code, but not much : I already made a big optimization pass.

      I added Class 10 SD card on my shopping list in order to run some test. But I recon those are supposed to be faster at writing, but not at reading. I will let you know if I get some major improvements.

      I can not explain publicly the application, but I can take the analogy with a library and book.
      My SD card is a library, and it contains 200 books (txt files).
      I need the ability to add more books to my library, which is why I store the files on the SD card.
      When the user is pressing a button, he wants the program to create a random story, with some sentences from up to 40 different books.


      • James_4DSystems
        James_4DSystems commented
        Editing a comment
        Hi - Just to make a comment on the SD card. Class 10 or any other class for that matter will not make any difference when used in our displays. The Class system is when the cards are used in SD mode, as is the Read/Write speed of the card. Our displays talk to the cards in SPI mode, which is not dictated by these systems, so the importance is getting a card that is SPI compatible, not one that is Class 10 or x number of MB/s.

    • #4
      The class of the SD card will make almost no difference.

      The cluster size will make some difference, if you use RMPET to partition and format your card it will use the largest cluster size which will give the best performance.

      There are two components that will contribute to slowing file open times.

      One is the number of files in the directory and how far in the file you are opening is from the start. The files are in creation order (assuming nothing has been deleted), so newer files will take longer to open. Long filenames take up directory space as well, so it helps to ensure your filenames are all short and obey short rules, so that no 'shadow' long name entries are created.

      The other is the size of the used part of the FAT and the number of FAT entries for each file. This can be minimized by using the largest cluster size possible, but a large file will still have a lot of FAT entries.

      Windows and modern computers with lots of memory hides most of this as both the root directory and FAT are generally cached, on a small system we don't have that luxury.

      What you might like to try is creating your files as RAW rather that using FAT. Place each file at a fixed starting sector on the card with a gap in between to ensure that the files cannot possibly overlap. Maybe the files can be further 'split' by starting 'logical records' on (multiple) sector boundaries.

      This will potentially slow down the creation process, but the retrieval process should be much faster.