 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : How many files can MM use on SD card?
Page 1 of 2   |
|||||
| Author | Message | ||||
Grogster Admin Group  Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Assuming a 2GB card or so and directories, how many files can the MM access? I assume that it is only limited by the FAT16 or FAT32 restrictions, which are pretty generous with respect to what the MM wants. However, assuming hundreds of text messages, each one in a separate file, all stored in one common directory, how many files can the MM have access to in any one folder? Smoke makes things work. When the smoke gets out, it stops! |
||||
MicroBlocks Guru  Joined: 12/05/2012 Location: ThailandPosts: 2209 |
Until storage space runs out. :) However storing everything in one directory will make things slower. A few hundred is a good compromise, but every situation has its own requirements. I don't know the exact implementation that is used by MM but 'walking' through a directory with a large number of files can use up a lot of ram and cpu power. What is also important to realize is how much space a file occupies. If you have a file that contains "Hello World!" it does not take up 12 bytes on the sd card. First the entry in the File Allocation Table will be at least 32 bytes, then depending on the cluster size used it can take between 512 and 65536 bytes. Having a lot of small files will waste large amounts of space. A hundred files with only "Hello World!" can use as much as 6.5 MB! The real data stored is only 1200 bytes. That is a waste of 99.9%. The most efficient is a file size that is a multiple of the cluster size. You can find out the cluster size of an sd card by using the chkdsk command on your computer. My own best practice is not more then a few hundred files in one directory. When i have log files i often use this: A directory for the year. In that directory a directory for every month. In every month directory a directory for every day. And in those directories files for every hour. Only when those hourly files are getting very big i use 10 minute files. When you use a timestamp it is easy to split. YYYY/MM/DD for the directory structure and files use the HH part of the timestamp. If 10 minute files are use i use the first three characters from the HHMM part of the timestamp. Microblocks. Build with logic. |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
I hear what you are saying, but I have never heard of a simple text file taking up 64KB of space... 
Having said that, pretty much all editing and creating of files will be within the MM program - does that change things - to take Windoze out of the equation? I ran chkdsk on a spare SD card, and there was no reported cluster size. Do you need to use a command-line switch? All it told me was that there were 16,384 bytes in each allocation unit - unless that is what you are referring to... At any rate, programs I have been playing with, saved to the same card on the MM are only a few hundred bytes each, certainly not 16K or more. 
EDIT: I just reformatted the SD card in Windoze, and now the allocation unit size is 1,024 bytes - not sure if this is relevant, but I mention it in passing. I will see if the MM can see this newly formatted card... Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Ahhhhh - I think I see what is going on here. MM saves the file as it actually is byte-size-wise, but to make the card compatible with Windoze, Bill Gates pads the file out to the allocation(cluster?) size. Since the new format, a test file saved to the reformatted card in MM is 253 bytes, but it shows up in Windoze as 1K, or 1,024 bytes. Right-click-PROPERTIES on the file shows that the actual size is 253 bytes, but the "Size on disk" is 1,024 bytes. Smoke makes things work. When the smoke gets out, it stops! |
||||
| MicroBlocks Guru Joined: 12/05/2012 Location: ThailandPosts: 2209 |
Allocation unit or cluster. Same thing :) I should correct myself. The 64KB is only possible on NTFS systems, FAT only allows a max of 32KB clusters. Otherwise it is not compatible with many systems. A cluster is made up of sectors, the smallest unit on a disk. there are 512 bytes in a sector. Good that the cluster size is 'only' 1024 bytes (2 sectors). It will be more efficient with smaller files. FAT is a standard that existed before windows and is one of the most used file systems, FAT and FAT12 were used on floppies. The new exFat system Microsoft wanted to patent (didn't follow it up to see if they actually did) is very similar to TRSDOS used in 1978, so not much progress on that front. :) NTFS is a much better file system but it also has the cluster size problem as most other file systems have. The cluster size is one of those parts that stays hidden. A FAT16 has only 65535 entries, multiply that with a cluster size and sector size to get the maximum addressable space on a disk. Microblocks. Build with logic. |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
At this stage, I expect there could be UP TO 300 files in one directory, but not more then that. I assume this should be OK? All files are simply text messages, so even assuming 1024 byte clusters, and ABOUT 50-75% wasted space per cluster, that should still be fine as a starting point - correct? What I might do is a little test: I will fill a directory on the card with several hundred files and a few test ones that I actually want, then run a few tests to see if hundreds and hundreds of files in the directory actually slow the MM up by that much. I would not actually expect it to be that bad - I will return with the results...  Smoke makes things work. When the smoke gets out, it stops! |
||||
James_From_Canb Senior Member  Joined: 19/06/2011 Location: AustraliaPosts: 265 |
Under FAT16 you could have 512 files in the root directory, (including . and ..) so you could have 510 files in the root. I don't believe there was any practical limit to the number of files in a directory. Microsoft has a good description of the differences between FAT16 and FAT32. Link James My mind is aglow with whirling, transient nodes of thought careening through a cosmic vapor of invention. Hedley Lamarr, Blazing Saddles (1974) |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Cheers for the link and extra info. 
I think I am OK on my foundation of 300 or so files in a directory - please others, let me know if you don't think so... Smoke makes things work. When the smoke gets out, it stops! |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 3802 |
It should work but may be SLOW. Try it :) John |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Hokey pokey, I made two directories, one called LOGS which has 299 files in it, and another called ZARA which has 441 files in it. I can change into either directory fine, but FILES results in the message: ERROR: Too many files to list. So, out of interest - what is the maximum that MM can handle? Geoff? In the LOGS directory with 299 files, access time to find and display the contents of the file is approximately half a second from the time the search for the file starts, to when it starts showing up on the screen - that is perfectly acceptable. 
In the ZARA directory with 441 files, the time is about 1 second. I'm not going to have 400 files in one directory anyway, and if I DID need this many files, I would find a way to split them up in the multiple directories, but I don't think I will need to worry about this anyway... Smoke makes things work. When the smoke gets out, it stops! |
||||
Geoffg Guru Joined: 06/06/2011 Location: AustraliaPosts: 3194 |
The short answer is... a large number. The limit is set by the structure of the particular file system (FAT16 and FAT32) and I have not tested them to come up with a precise number. The FILES command is a different proposition. It must sort the file list and that requires memory which in turn limits the number of files that can be listed to about 200. Geoff Graham - http://geoffg.net |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Acknowledged.
It does not really matter, so long as I can access the files. Here is my test code: Dim A$(100) Dim Q$(1) Timer = 0 Open "B:\ZARA\blah.log" For input As #1 T=Timer start: Cls Print "ACCESS TIME IN ms WAS: "; T For x=1 To 25 Line Input #1, A$ Print A$ Next x Print "":Print "" Input "SHOW MORE"; Q$ If Q$="Y" Or Q$="y" Then GoTo start Close #1 End This code measures exactly how long it takes to open the file, and in the case of the ZARA folder with 441 files in it  the time delay to open the file is 743ms.
This is still only about three quarters of a second to find the file in 400+ files, so this is still and excellent result, as far as I am concerned.
The result for the LOGS directory with 299 files is about 212ms. Very good results. Anything under 1 second is fine with me. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Just found an old 16MB(yes, MB!) SD card, and when I tried to format that in Win7, it set the cluster size to 4096 which I think is a little odd, when Win7 formatted a 64MB SD card with 1024 cluster size...
I would have thought that 1024 would be the norm for small capacity cards? Perhaps it is just Win7 being bloody-minded. I would not put it past Bill... Might crank up a copy of Linux or Puppy and see if I can set the cluster size there. Any comments on this would be great, also if anyone knows of an SD card formatter app for Win7(that lets you set the cluster size), that also would be great. Smoke makes things work. When the smoke gets out, it stops! |
||||
TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6099 |
In Windows 7 (or any other Windows for that matter), from a command prompt type: format /?
and see what you can do if you really want to control things. Jim VK7JH MMedit MMBasic Help |
||||
| paceman Guru Joined: 07/10/2011 Location: AustraliaPosts: 1329 |
Happened to have an MS Support link on this: Formatting cluster size Greg |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
EXCELLENT - Thank you!
That worked a treat, and at the Windoze command prompt at C:\>, I could then issue the command: format k: /A:512 and the SD card was formatted as FAT16 with 512 byte clusters. Perfect.
@ paceman - just missed your post - will check that now. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
Still something odd going on here with the maths. Formatted as FAT16 with 512 byte clusters - that went fine. Copied a 621 byte file to the card - that went fine. Right-click/Properties of the file on the SD card reports that it's size is 621 bytes, but size-on-disk is still 4096 bytes, so the cluster size still seems to be 4096.
Lost... EDIT: Forget it - looks like a Win7 bug perhaps.
While it is true that in explorer, it shows as 4096, when I reopen the command prompt and ask for a chkdsk k:, it reports that there is 1 file taking up 1024 bytes, which is what I would expect with 512 byte clusters and a file bigger then 512 but less then 1024... Odd - but lets put this one down to Windoze explorer being wrong. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
This is a bit of an old thread, but I need to bring it back. On the CMM, the access time is now 15.8 seconds for the same code shown on page 1. This really IS to long. I am guessing at this point, that the slow access is due to the CMM vs the B/W MM - the CMM needs much more time to generate colour output, which is holding everything up. Would that be correct? I will try some experiments with the different colour modes. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9308 |
ADDITIONAL - Might be the SD card. I have tried 16MB and 128MB on BOTH the MM and the CMM, and the times are about the same. I must have been using a faster card when I did the initial tests. I will keep you all posted. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Geoffg Guru Joined: 06/06/2011 Location: AustraliaPosts: 3194 |
No, that does not sound right. Maybe 15% slower but nothing like this. Something else must be wrong: corrupt SD card, different versions of MMBasic, different SD cards, ??? Geoff Graham - http://geoffg.net |
||||
| Page 1 of 2 |
|||||
 Print this page |
