Dear all,
I would like to ask you if there is any maximum allowed number of files per folder in linux (without risking it to lose everything).
I am using openuse 11.4 with latest kde (4.6?).
I am trying something fast and dirty and it might be that one folder will contain like 10^6 files.
Is there is anything I should be warned about that? Of course filesystem security is still the most important.
Best Regards
Alex
2.3. Sub directory scalability
Right now the maximum possible number of sub directories contained in a single directory in Ext3 is 32000. Ext4 breaks that limit and allows a unlimited number of sub directories.
Edit: I assume that means the maximum number of files per directory is unlimited too.
Actually you don’t risk data loss. The mkdir syscall will just fail when the limit is reached. Unless you’re ignoring the error return from the call.
And it may of course depend on the type of the file system. The others assume Ext4, which might be what you have there, but you fail to tell (and there are allways people who think that non Linux file systems behave exectly the same as Linux native ones).
On 04/28/2011 02:36 PM, alaios wrote:
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
there is no risk if you have all of your data backed up to an off
machine location, right?
how many? discover google:http://tinyurl.com/3wzbgdd
Have a lot of fun!
–
CAVEAT: http://is.gd/bpoMD
[openSUSE 11.3 + KDE4.5.5 + Thunderbird3.1.8 via NNTP]
HACK Everything → http://www.youtube.com/watch?v=j5b4CCe9pS8&NR=1
On 2011-04-28 14:36, alaios wrote:
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
It depends on what filesystem you use.
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
No problem.
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
Some filesystem types are faster than others in that situation. Reiserfs is
the best, xfs follows - IMO. But R is in the decline.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
ext4 ofc 
Then just go ahead and have a lot of fun.
When you have been longer on the forums, you would know nothing is “of course” 
I have seen here people with problems who only after 10 or more posts revealed that they where having this on a FAT file system, which explained everything. >:(
of course you are right
Cheers
Alex
alaios wrote:
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
> I am using openuse 11.4 with latest kde (4.6?).
>
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
>
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
One pragmatic observation: there is a real issue if you ever try to view an
directory with many. many files with a file manager (dolphin or konqueror).
Once you select that directory (folder) it may be a long, long time before
you can do anything as it builds the display for all those files.
I did some builds that generated all sorts of intermediate files I used for
debug and I had to be careful to NEVER open a view of that temp directory in
the filemanager. Note that the same effect is seen with a bunch of
subdirectories as well. The problem went away as I got smarter and stored
the stuff in deeper trees but it does create a fairly important practical
limit on the number of items stored at each level of the directory tree.
No real problem if you access the files from the command line or a script
but the GUI view response time is problematic.
–
Will Honea
On 04/28/2011 08:14 PM, Will Honea wrote:
> No real problem if you access the files from the command line or a script
> but the GUI view response time is problematic.
yep, you can make it long enough that even in mc you might think
something is broken…
–
CAVEAT: http://is.gd/bpoMD
[openSUSE 11.3 + KDE4.5.5 + Thunderbird3.1.8 via NNTP]
HACK Everything → http://www.youtube.com/watch?v=j5b4CCe9pS8&NR=1
On 2011-04-28 20:14, Will Honea wrote:
> No real problem if you access the files from the command line or a script
> but the GUI view response time is problematic.
Even the command line can be problematic. If you try “ls /thatdirectory/*”,
because the asterisk is expanded to the entire directory list before the
command ls or whatever is called, and that list can be so large that it is
more than 64 KB and overflow the buffer. You have to be careful of that in
scripts.
I would do a test of the application with different filesystems. Some work
but are much slower than others.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
On Thu, 28 Apr 2011 13:06:02 GMT, gropiuskalle
<gropiuskalle@no-mx.forums.opensuse.org> wrote:
>
>‘Ext4 - Linux Kernel Newbies’ (http://tinyurl.com/9y2tx7)
>
>> 2.3. SUB DIRECTORY SCALABILITY
>> Right now the maximum possible number of sub directories contained in a
>> single directory in Ext3 is 32000. Ext4 breaks that limit and allows a
>> unlimited number of sub directories.
>
>Edit: I assume that means the maximum number of files per directory
>is unlimited too.
As a matter of personal experience i have had problems when a directory
(in an ext3 fs) got to over 100,000 files. And it got really ugly with
138,000 files. I have also experienced issues when trash go to being
over 11 GB worth, of course that was driven by filling the volume up
(99%).
YMMV
Well, the maximum of subdirectories (and, as I assume, of files) per directory is 32000 within an ext3 filesystem, so it’s not really a surprise to have some trouble when creating three or four times as many.
On Fri, 29 Apr 2011 12:36:02 GMT, gropiuskalle
<gropiuskalle@no-mx.forums.opensuse.org> wrote:
>
>Well, the maximum of subdirectories (and, as I assume, of files) per
>directory is 32000 within an ext3 filesystem, so it’s not really a
>surprise to have some trouble when creating three or four times as many.
Thewse were files, not directories. As per OP question.
On 2011-04-29 17:44, josephkk wrote:
> Thewse were files, not directories. As per OP question.
I tested Reiserfs with way more than that.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
Thewse were files, not directories. As per OP question.
Again: I assume the maximum number of files has the same limit as the number of folders - and the OP is using ext4, not ext3.
On 2011-04-28, alaios <alaios@no-mx.forums.opensuse.org> wrote:
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
> I am using openuse 11.4 with latest kde (4.6?).
>
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
>
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
Beyond the file system limits, you’ll soon run into the problem of speed.
Too many files in 1 dir, and you’re slowin down to a crawl.
Even your application will slow down, if it has to add a file to a dir with
10^6 files.
Don’t know about your file’s names. But assuming they have enough chars,
make a dir struct based on the first 3-4 chars. One char per dir level, of
course.
–
When in doubt, use brute force.
– Ken Thompson
This sounds like what database management systems are made for.