Trnasfer Files from EXT4 to NTFS not entirely working

English Hardware
1

Hi there,

I’m trying to transfer 22.2GB worth of music from my ~/Music folder (Absolute Path: /tb3/Music) to my F: Drive on my Windows partition (Absolute Path: /windows/F/)

I’ve tried both rsync -a, and cp -R but certain things get lost in the transfer, for example ID3 tags on many of my songs are suddenly gone. (Ex. 65daysofstatic which has all the relevant ID3s on Linux, has lost them ALL when viewed from Windows)

I’m only transferring the songs to be able to put them in iTunes to transfer to my iPhone, but I don’t want to have to go through and re-tag them all. :frowning:

Any ideas on what’s going wrong? How to fix?

Thanks!

2

I meant to edit my post, but it doesn’t seem like I can?

Anyway, I also wanted to mention that some folders will show up in Windows (eg. M.I.A. or Does It Offend You Yeah) but when clicked it gives me an error to the ffect of “This folder no longer exists in the location” or something of the sort. :frowning:

3

As you do not provide much details, the following is a bit of a guess, but it might start strains of thought for you :wink:

NTFS and other Windows fs types are very different from Linux native file systems. Let the fact that Linux has software that make thes file system types usable by Linux not lure you into thinking thaty features available in Linux are by magic available in those file systems whenn used by Linux. The whole concept of owning user, group and access bits is e.g. not available on NTFS and thus lost on a transfer. I am not a Windows expert, not even a Windows user, but by rumour I understand that file names are case independent in NTFS and thus putting the files AAP and aap both on such a file ystem will result in some clash I guess. Also a feature of Windows file names, the so called extension, is not a feature of Linux. I guess that Linux file name having at the end a . (dot) and three other characters wiil have these chopped of from the filename and the three characters used for the extension on NTFS (this not being illogical to implement). But what happens with Linux filenames having a . (dot) and more then three non-dot charachter after it, I do not know.

This may all be documented and/or known exactly by others, who may post here.

HTH

4

file names are case independent in NTFS and thus putting the files AAP and aap both on such a file ystem will result in some clash I guess.

From what I understand, NTFS files are stored case dependent but much functionality is backwards compatible with earlier Windows systems ignoring case. Newer functionality in Windows is generally case sensitive making Windows more like *NIX.

Also a feature of Windows file names, the so called extension, is not a feature of Linux. I guess that Linux file name having at the end a . (dot) and three other characters wiil have these chopped of from the filename and the three characters used for the extension on NTFS (this not being illogical to implement).

I was curious about this, too but I’m finding that the filename with extension is preserved when transferred between Linux and Windows. Apparently Linux just ignores the file extension but supports the naming convention.

But what happens with Linux filenames having a . (dot) and more then three non-dot charachter after it, I do not know.

This causes the directory/filename and everything within the subdirectory tree to be visible but unnavigable and files cannot be opened within Windows.

Bottom line is, until someone creates an Ext4NTFS utility for Windows like what has existed for earlier versions of Linux disk format/file systems, you’ll need to mount your disks using Linux, ie. using a LiveCD if you don’t want to install Linux to disk.

Tony

5

Problems with NTFS partitions can also arise because of the options setup in your fstab file. Here is ONE line from my fstab file where I have changed the default options on a NTFS partition to read just defaults, giving full read/write access to the Windows partition, something you might consider doing also.

/dev/disk/by-id/usb-ST315005_41AS_0123456789ABCDEF-0:0-part1 /Software             ntfs-3g    **defaults** 0 0

To edit your fstab file if you use KDE is the menu Run Command:

kdesu kwrite /etc/fstab

Please ask more questions if there is anything that is not clear here. I have put the defaults option command in bold, which will not be the case in your fstab file, but used for clarity here.

Thank You,

6

On the danger of going off topic a bit I like to comment on this a bit more. Probably showing my ignorance in Windows.
I for example do not know if all characters allowed in Linux file names are also allowed in Windows and vv. Especialy, what about the dot in Windows? This may also be of interest to the OP. Same for the length of a filename (IIRC the maximum of 14 characters for an MS-DOS filename is lifted, but I do not know the current Windows maximum).

Your first remark about the case sensitivity certainly explains more on how it is done and where problems could surface. If I understand you correct, AAP and aap wil be both stored, but Windows programs may be confused by there very existence.

On the extension. In MS-DOS (and I assume that there is a hangoiver from there to nowadays Windows fs types) the extension is just a three character field in the administration of a file inside the directory. It is normaly show to the user seperated by a dot from the filename. This leads to the idea that the dot is part of the whole which it is not. It is just a separator between two fields.

Linux does not have such a field. So what to do in software which makes not only copying in Linux to and from Windows file systems possible , but also tries to keep the names of those files as much the same as possible?

From Windows to Linux the obvious thing is: make a Linux filename coonsisting of the Windows filename, a dot and the Windows extension. That looks very much as itt is shown on Windows (but mark that information is lost on this transfer).

From Linux to Windows it is different. It looks to me (and you) that when the Linux filename ends in a . and three character, the assumption is made that the three character are an extension (which may be utterly wrong).
The problems which might arise when the Linux file name is e.g.* This.is.a.Linux.filename* ar allready touched above.

Interesting subject, even if I never plan to use Windows :wink:

7

Especialy, what about the dot in Windows? This may also be of interest to the OP. Same for the length of a filename (IIRC the maximum of 14 characters for an MS-DOS filename is lifted, but I do not know the current Windows maximum).

The length supported now is so long I don’t even look it up unless a system balks, and with each new NTFS version it gets longer. More likely the path including filename might get too long, earlier it was 255 characters and now is likely another order longer.

Your first remark about the case sensitivity certainly explains more on how it is done and where problems could surface. If I understand you correct, AAP and aap wil be both stored, but Windows programs may be confused by there very existence.

Yes, they’re both stored, and usually for backwards compatiblility the DOS 8.3 filename is also stored. Haven’t yet found a Windows program confused but may balk if the case sensitive name doesn’t doesn’t exist (like *NIX fs).

On the extension. In MS-DOS (and I assume that there is a hangoiver from there to nowadays Windows fs types) the extension is just a three character field in the administration of a file inside the directory. It is normaly show to the user seperated by a dot from the filename. This leads to the idea that the dot is part of the whole which it is not. It is just a separator between two fields.

Not lately. Earlier Windows fs rejected a period in any part of a directory or filename but today the only restriction is that you can’t lead or trail periods. And, remember the 8.3 filename is deprecated and stored only for backwards compatibility so it’s not fundamental to how a modern fs works.

Linux does not have such a field. So what to do in software which makes not only copying in Linux to and from Windows file systems possible , but also tries to keep the names of those files as much the same as possible?

From my experience, nothing untoward happens as long as the fewer Windows fs rules are followed. So, for instance if a file was created in Windows, contents modified in Linux and then re-opened in Windows (using the same filename) I haven’t seen a problem.

From Windows to Linux the obvious thing is: make a Linux filename coonsisting of the Windows filename, a dot and the Windows extension. That looks very much as itt is shown on Windows (but mark that information is lost on this transfer).

So far, haven’t seen anything lost.

From Linux to Windows it is different. It looks to me (and you) that when the Linux filename ends in a . and three character, the assumption is made that the three character are an extension (which may be utterly wrong).
The problems which might arise when the Linux file name is e.g.* This.is.a.Linux.filename* ar allready touched above.

I haven’t seen a problem as long as Windows naming conventions aren’t violated. As I noted before, the most current NTFS naming rules support periods in the middle of the filename. The consequence of your example is simply that there is no “viewer” associated with a possible filetype “filename” so the User would have to manually select an application to open the file.

Interesting subject, even if I never plan to use Windows :wink:

Ah, living in a Windowless world? rotfl!

8

Thanks for all the answers.

I can understand that. Changing the contents is very different from what the OP does, creating new files with new names.

When the file file.ext is seen from a Linux system, you wouldn’t know if this the file file with extension ext or the file file.ext with an empty extention (or would that be shown as file.ext. ?)
But I think it is rather academic.

Is the supposed extension filename shortened?

O yes. There is one system here which could boot into Windows XP, but that is not done for at least three years (and thus rather vulnarable when I would boot it with the network cable attached rotfl! ). When we need disk space there, I will wipe it.

9

When the file file.ext is seen from a Linux system, you wouldn’t know if this the file file with extension ext or the file file.ext with an empty extention (or would that be shown as file.ext. ?)
But I think it is rather academic.

It seems that in a Windows fs the three character extension is meaningful, a *NIX fs it’s meaningless (just part of the name). But no changes are made by either fs.

Is the supposed extension filename shortened?

Nope. Just considered an ordinary filename.

Tony

10

On 2010-09-26 19:06, tsu2 wrote:
>
>> When the file file.ext is seen from a Linux system, you wouldn’t know if
>> this the file file with extension ext or the file file.ext with an empty
>> extention (or would that be shown as file.ext. ?)
>> But I think it is rather academic.
>
> It seems that in a Windows fs the three character extension is
> meaningful, a *NIX fs it’s meaningless (just part of the name). But no
> changes are made by either fs.
>
>> Is the supposed extension filename shortened?
>
> Nope. Just considered an ordinary filename.

You people are concentrating and giving importance to the extension in windows filesystem, when
there isn’t any, nowdays.

Historically, msdos stored 11 chars, 8 for the name and 3 for the extension. The dot was not stored,
it was just displayed on outputs. When they added long name support, the coded extension
disappeared. They store the dot in the filename, several dots if wanted. Haven’t you seen files with
“two extensions”? Files named like “very interesting document.txt.exe”? They are popular with mail
virus attaches (trojans, actually). The last three are not displayed, the user thinks it is a document.

But the extension does not really exist in the filename, and can be longer than three chars, I
believe (I’m not going to boot my windows to test that just now). We are not that different in this
respect.

What is different is what both systems do with the extension. Linux, nothing. Windows, program
association. However, some linux programs do give importance to extensions, windows style.

But nothing of this is related to the OP problem.

Possible issues are:

case.

charsets.

Some one commented that the wrong charset definition in fstab can cause files to be invisible or not
accessible. The post must be somewhere in the forum… Ah, that was swerdna in the instal-boot-login
forum, thread “recovering Windows files via Open Suse”.

All the above affects the names of the files, not the content.

But the poster is having problems with data inside the files, with ID tags in music files. No way
a file copy can cause that. It must be a problem with the program that created those files, or the
one that reads them. Probably linux is using a charset that the windows program doesn’t handle.

Possibly linux used UTF-8, no idea what windows uses/expects.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

11

You’re right, we got a bit off topic.

Replying the OP, the tags you’re describing are “file attributes” which AFAIK are supported only within the OS and application, not at the lower file system on the disk. Bottom line what this means is that you’ll likely need a special file transfer utility to preserve those attributes. Ordinary file transfer utilities as you’ve discovered won’t work (because attributes and attribute structures won’t be stored there). I also have not heard that there is any recognized standard for file attributes which further means that only special utilities can view and manage these kinds of files fully.

So, the most obvious question likely is, why aren’t you using the iTunes app to transfer files? – That should do the job.

Getting back to Carlos’ post (immediately before this one), I think that much of your post is consistent with what I posted before, but IMO you’re also leaving out probably the most common “problem” - not charactersets/encoding methods, but individual characters. A large number of characters still have special meaning in a Windows fs so are prohibited but are fully supported in *NIX fs, and that is besides the leading and trailing period issue I mentioned.

Tony

12

Carlos E. R. wrote:

> On 2010-09-26 19:06, tsu2 wrote:
>>
>>> When the file file.ext is seen from a Linux system, you wouldn’t know if
>>> this the file file with extension ext or the file file.ext with an empty
>>> extention (or would that be shown as file.ext. ?)
>>> But I think it is rather academic.
>>
>> It seems that in a Windows fs the three character extension is
>> meaningful, a *NIX fs it’s meaningless (just part of the name). But no
>> changes are made by either fs.
>>
>>> Is the supposed extension filename shortened?
>>
>> Nope. Just considered an ordinary filename.
>
> You people are concentrating and giving importance to the extension in
> windows filesystem, when there isn’t any, nowdays.
>
> Historically, msdos stored 11 chars, 8 for the name and 3 for the
> extension. The dot was not stored, it was just displayed on outputs. When
> they added long name support, the coded extension disappeared. They store
> the dot in the filename, several dots if wanted. Haven’t you seen files
> with “two extensions”? Files named like “very interesting
> document.txt.exe”? They are popular with mail virus attaches (trojans,
> actually). The last three are not displayed, the user thinks it is a
> document.
>
> But the extension does not really exist in the filename, and can be longer
> than three chars, I believe (I’m not going to boot my windows to test that
> just now). We are not that different in this respect.
>
> What is different is what both systems do with the extension. Linux,
> nothing. Windows, program association. However, some linux programs do
> give importance to extensions, windows style.
>
>
>
> But nothing of this is related to the OP problem.
>
> Possible issues are:
>
> case.
>
> charsets.
>
> Some one commented that the wrong charset definition in fstab can cause
> files to be invisible or not accessible. The post must be somewhere in the
> forum… Ah, that was swerdna in the instal-boot-login forum, thread
> “recovering Windows files via Open Suse”.
>
>
> All the above affects the names of the files, not the content.
>
> But the poster is having problems with data inside the files, with ID
> tags in music files. No way a file copy can cause that. It must be a
> problem with the program that created those files, or the one that reads
> them. Probably linux is using a charset that the windows program doesn’t
> handle.
>
> Possibly linux used UTF-8, no idea what windows uses/expects.

Slightly incomplete, Carlos. The extension IS stored - I’ve used non-system
extensions for years, even on Windows. The problem that bites most people
is that Winxx has a “convenience” setting that by default does not display
the extensions for known file types unless you specifically change the
setting. The command processor “assumes” a .exe or .com extension when
looking for an executable (but will attempt to run ANY fully qualified file
name). Hence file.txt.exe is DISPLAYED as file.txt but clicking one it
causes the command processor to find file.txt.exe and execute it. Piss poor
design feature that has caused problems since Win98 at least. My delivery
checklist includes turning that “feature” off - it’s a problem waiting to
happen. There have been all sorts of system coding problems with the hiding
feature over which ‘.’ triggered what actions with multiple .'s in a file
name.

All the issues over using a file name/namepart irks me in Linux as well in
that it uses a leading ‘.’ to signify a hidden file. That should be an
attribute of the file, not part of the name.

–
Will Honea

13

Thanks for all the replies guys, I’ve learnt a lot actually. :slight_smile: I don’t understand what you mean by “using the iTunes app” to transfer my files? I used Kid3 on linux to do my tags, and as far as I can tell, there is no iTunes for Linux :stuck_out_tongue:

I think much of my problem might be from Windows-illegal characters in the filenames ( ] / \ , . ? etc.) Because the ones I’m seeing that are missing their tags are like “Does it offend you yeah?” so maybe just the question mark is messing it up. I’ll try a few things, take some of the suggestions here into account and see what I can get. :slight_smile: Thanks!

14

You’re right, I haven’t actually supported any Linux users who use iTunes. Noobs I know all use Windows or Macs.

Hmmm… used KDE did you? That’s probably not good. I’m not sure there is a solution to your situation then unless you can somehow get those file attributes exported and into something common to both Linux and Windows (eg iTunes attributes). Note that since Mac is fundamentally Linux doing this would have a high chance of success.

According to a quick search which returns plenty of Ubuntu Forum posts, you’ll have to install WINE to run iTunes.

Tony

Tony

15

I’m wondering what the OP is using for ntfs-3g. I’m using the latest ntfs-3g library (2010.8.8) from OpenSUSE Factory and copying, moving, editing and deleting files without incident on my compressed NTFS partition (2010.8.8 introduces complete compressed NTFS support).

deltatux

16

The User isn’t likely using anything special, but the file attributes he’s trying to preserve aren’t likely stored at that level (disk sub-system). Since AFAIK there isn’t a common standard for file attributes between disk formats and secondarily the use of prohibited Windows characters, he’s losing that information transferring from one disk format to another and using a different OS and applications.

Tony

17

On 2010-09-28 07:03, Will Honea wrote:
> Carlos E. R. wrote:

> Slightly incomplete, Carlos. The extension IS stored - I’ve used non-system
> extensions for years, even on Windows. The problem that bites most people
> is that Winxx has a “convenience” setting that by default does not display
> the extensions for known file types unless you specifically change the
> setting.

I did not say the extension is not stored. I said, rather implied, that the extension is not
displayed (by default), and that it is not stored separately. Rather the full name, extension
included, is stored with the dot or dots in the middle as a string, same as in Linux. That is. There
is no “physical” extension as there was in msdos.

My whole point is that they are stored the same in linux as in windows, there is no “extension”.
Just names with dots somewhere, and the last part is called extension.

(of course, there are forbidden chars and such, that’s another thing)

> The command processor “assumes” a .exe or .com extension when
> looking for an executable (but will attempt to run ANY fully qualified file
> name). Hence file.txt.exe is DISPLAYED as file.txt but clicking one it
> causes the command processor to find file.txt.exe and execute it. Piss poor
> design feature that has caused problems since Win98 at least. My delivery
> checklist includes turning that “feature” off - it’s a problem waiting to
> happen. There have been all sorts of system coding problems with the hiding
> feature over which ‘.’ triggered what actions with multiple .'s in a file
> name.

Yes.

> All the issues over using a file name/namepart irks me in Linux as well in
> that it uses a leading ‘.’ to signify a hidden file. That should be an
> attribute of the file, not part of the name.

Different oses, different methods :slight_smile:

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

18

My whole point is that they are stored the same in linux as in windows, there is no “extension”.
Just names with dots somewhere, and the last part is called extension.

Question: Does this mean that Windows HTTP servers can call their HTML pages *.html instead of the silly *.htm?

19

On 2010-09-28 21:36, hcvv wrote:
>
>> My whole point is that they are stored the same in linux as in windows,
>> there is no “extension”.
>> Just names with dots somewhere, and the last part is called extension.
> Question: Does this mean that Windows HTTP servers can call their HTML
> pages *.html instead of the silly *.htm?

Well… Let me see, because the answer is not so simple.

I just plugged an usb stick, FAT formatted, into linux, and I created a “file.html” file. So, the
filesystem allows it. The next test is to see if Windows sees it, which means booting linux in my
laptop, which now is hibernated in linux.

But even if the filesystem allows it, even if I can create such files in windows (test later), the
particular software you use has to allow for a different extension than the one it thinks is the
proper one.

Meaning: the filesystem allows that, the software may not.

[testing on windows 7]

Booting windows once a month is a chore, it starts updating a zillion things (even though it is a
minimal install), and rebooting a few times. But I’m curious.

…]

Yes, W7 does allow me to name a file.html, and happily opens it with Firefox (hey! I’m not going to
use iexplorer in my windows ;-)) without a complain. And I created another one with a “long
name.texto”, no problem - except that this one it doesn’t know how to open it (and I’m not going to
teach it :wink: ).

And, if you try to change the extension, it warns you that the file might be not usable or
“openable”. That’s to be expected.

So, the answer is yes, you can use .html as extension - but it may happen that other software, or
older versions, do not recognize them. I assume that modern software is configurable and would have
no problems with that. It may be a simple question of the admins not wanting to, or wanting to keep
compatibility with older versions.

–
Cheers / Saludos,

Carlos E. R.
(from 11.2 x86_64 “Emerald” at Telcontar)

20

As Carlos restated my assertion, filename extensions of any length and any legal character are permitted and as I stated may be used by the OS to launch a default viewer.

This question about what sequences of characters are permitted is brought about by confusion over dividing the functionality of the disk format, operating system and the application. If you properly understand the layering and division of functionality things should become clear…

At the lowest level, the disk format will store files and their permissions.
At the next lowest level, the operating system reads, writes and executes depending on the permissions set at the lower level.
At the third level, an application will read, write and execute depending on what is permitted by the operating system.

So, when you’re talking about something like an .html file extension, because .html does not contain any illegal characters and length is not a consideration, given that permissions permit opening the file the operating system has “first right” to open the file and if possible then the application then has “second right” to open that file. The application can be a client application like Firefox or it can be a server application like Apache.

Where this becomes relevant to the OP in this thread is that the file attributes he’s configuring aren’t the filename or permissions, they’re a custom creation of whatever OS/application he’s using, in his case KDE. Since again AFAIK there hasn’t been a IEEE standard defined that everyone adheres to, his KDE creations can’t be guaranteed to be viewable on any other system than one with KDE installed, and that is also why I suggested using the iTunes application to create and manage those file attributes since it would be the cross platform, cross OS common denominator for any desktop and mobile device he wanted to transfer those files to.

Also, stating that older or newer (application) software is a non sequiter. At least in modern Linux and Windows OS, the OS is configured to select the viewer of a file, this is not done in the application itself. Technically speaking you should be able to configure an HTML file to be viewable automatically by FF even if the file extension was php, jsp, net, or mxylplix.

Expanding perhaps unnecessarily, note that on Windows fs the 8.3 filename may also be stored, but that is because of the way legacy method longer names were shortened to 8.3… so that legacy “DOS/Win3.x” applications can still run.

hth,
Tony