Dear all,
I would like to ask you if there is any maximum allowed number of files per folder in linux (without risking it to lose everything).
I am using openuse 11.4 with latest kde (4.6?).
I am trying something fast and dirty and it might be that one folder will contain like 10^6 files.
Is there is anything I should be warned about that? Of course filesystem security is still the most important.
Right now the maximum possible number of sub directories contained in a single directory in Ext3 is 32000. Ext4 breaks that limit and allows a unlimited number of sub directories.
Edit: I assume that means the maximum number of files per directory is unlimited too.
And it may of course depend on the type of the file system. The others assume Ext4, which might be what you have there, but you fail to tell (and there are allways people who think that non Linux file systems behave exectly the same as Linux native ones).
On 04/28/2011 02:36 PM, alaios wrote:
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
there is no risk if you have all of your data backed up to an off
machine location, right?
On 2011-04-28 14:36, alaios wrote:
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
It depends on what filesystem you use.
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
No problem.
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
Some filesystem types are faster than others in that situation. Reiserfs is
the best, xfs follows - IMO. But R is in the decline.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
When you have been longer on the forums, you would know nothing is “of course”
I have seen here people with problems who only after 10 or more posts revealed that they where having this on a FAT file system, which explained everything. >:(
>
> Dear all,
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
> I am using openuse 11.4 with latest kde (4.6?).
>
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
>
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
One pragmatic observation: there is a real issue if you ever try to view an
directory with many. many files with a file manager (dolphin or konqueror).
Once you select that directory (folder) it may be a long, long time before
you can do anything as it builds the display for all those files.
I did some builds that generated all sorts of intermediate files I used for
debug and I had to be careful to NEVER open a view of that temp directory in
the filemanager. Note that the same effect is seen with a bunch of
subdirectories as well. The problem went away as I got smarter and stored
the stuff in deeper trees but it does create a fairly important practical
limit on the number of items stored at each level of the directory tree.
No real problem if you access the files from the command line or a script
but the GUI view response time is problematic.
On 04/28/2011 08:14 PM, Will Honea wrote:
> No real problem if you access the files from the command line or a script
> but the GUI view response time is problematic.
yep, you can make it long enough that even in mc you might think
something is broken…
On 2011-04-28 20:14, Will Honea wrote:
> No real problem if you access the files from the command line or a script
> but the GUI view response time is problematic.
Even the command line can be problematic. If you try “ls /thatdirectory/*”,
because the asterisk is expanded to the entire directory list before the
command ls or whatever is called, and that list can be so large that it is
more than 64 KB and overflow the buffer. You have to be careful of that in
scripts.
I would do a test of the application with different filesystems. Some work
but are much slower than others.
–
Cheers / Saludos,
Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)
>
>‘Ext4 - Linux Kernel Newbies’ (http://tinyurl.com/9y2tx7)
>
>> 2.3. SUB DIRECTORY SCALABILITY
>> Right now the maximum possible number of sub directories contained in a
>> single directory in Ext3 is 32000. Ext4 breaks that limit and allows a
>> unlimited number of sub directories.
>
>Edit: I assume that means the maximum number of files per directory
>is unlimited too.
As a matter of personal experience i have had problems when a directory
(in an ext3 fs) got to over 100,000 files. And it got really ugly with
138,000 files. I have also experienced issues when trash go to being
over 11 GB worth, of course that was driven by filling the volume up
(99%).
Well, the maximum of subdirectories (and, as I assume, of files) per directory is 32000 within an ext3 filesystem, so it’s not really a surprise to have some trouble when creating three or four times as many.
>
>Well, the maximum of subdirectories (and, as I assume, of files) per
>directory is 32000 within an ext3 filesystem, so it’s not really a
>surprise to have some trouble when creating three or four times as many.
Thewse were files, not directories. As per OP question.
On 2011-04-28, alaios <alaios@no-mx.forums.opensuse.org> wrote:
> I would like to ask you if there is any maximum allowed number of files
> per folder in linux (without risking it to lose everything).
> I am using openuse 11.4 with latest kde (4.6?).
>
> I am trying something fast and dirty and it might be that one folder
> will contain like 10^6 files.
>
> Is there is anything I should be warned about that? Of course
> filesystem security is still the most important.
Beyond the file system limits, you’ll soon run into the problem of speed.
Too many files in 1 dir, and you’re slowin down to a crawl.
Even your application will slow down, if it has to add a file to a dir with
10^6 files.
Don’t know about your file’s names. But assuming they have enough chars,
make a dir struct based on the first 3-4 chars. One char per dir level, of
course.