Page 1 of 2 12 LastLast
Results 1 to 10 of 17

Thread: Looking for advise on distributed filesystem choice for opensuse

  1. #1
    Join Date
    Sep 2008
    Location
    Porto Alegre, RS, Brazil
    Posts
    72

    Lightbulb Looking for advise on distributed filesystem choice for opensuse

    Hi all.


    First of all, sorry if this is the wrong place in the forums to put this question: I was unsure where it would be mostly appropriate, specially after it was refuse in the mail lists.


    I'm looking for some advise for a test implementation of a distributed filesystem at a computational chemistry students lab. If it is successful, the same solution will be considered for the (small) research lab clusters available.



    At the research lab we are used to NFS usage for quite some time (since the 90's) under suse, even for our clusters. The students lab also usually used similar solutions, but for reasons that are unimportant here a single NFS server won't be an option now. The students lab is also being upgraded at this moment, and will have new 6 i3 computers with 500gb disks (that will still share room with 5 old Pentium Ds with 200gb disks).



    The idea that we are considering is the following:


    1) The old Pentium Ds will mount an NFS /home from the i3s (with "failure fallback" between them);

    2) The i3s will serve the NFS /home for the Pentium Ds;

    4) In order to have all computers accessing all files coherently, a distributed filesystem among them for this /home must be used. This will also increase the available disk space due to data striping;

    5) Reliability is however important, and as such we are considering that instead of 3tb (6*500gb) of total space we have only 2tb (4*500gb + 2) and be able to withstand up to two i3 computers failing;

    6) Reliability must go farther than just files, but even necessary metadata servers and others must be redundant: optimally, replicated on every i3 computer;

    7) They must be able to also deal with all 11 computers accessing, both reads and writes, the whole/home . From molecular Dynamics to quantum mechanics calculations and also molecular docking, they must be able to deal with different demands and some big (several gb) file sizes, all simultaneously;

    8) Also we would not like to have to have to make special instructions to the users, like "you can't rename a file, but copy and delete it later" for example, as some (old?) gluster instructions indicate to be necessary;

    9) The simpler the implementation and the maintenance, the better it will be for both actual students lab and the future production clusters use.



    From what I can gather, the most usual suggestions are ceph, gluster or beegfs. Which ones are available on opensuse leap 15? Any other options worth considering? Which ones are recommended for the proposed application?



    Any help or direction on this will be really useful, and we would be very grateful for it.



    Thanks a lot in advance.

  2. #2
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,130
    Blog Entries
    2

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    Let's first consider one of your proposed ideas which is to mount /home as a shared directory used by all.


    In principle, that's not ordinarily advisable.
    The /home partition is typically associated with a User profile which is tied to the logged in User. A "better" decision is for every person on your network to log in with their own personal, unshared credentials so that person's activities are unique. It's the only way that you can have accountability. Once Users start sharing login credentials and resources on the network using same credentials, you've lost any way of identifying who did what and when. Even if everyone on your network is your most trusted family, things can get sticky when odd things happen and people point fingers at each other and you have no way to identify the cause of such and such was at such time by the User who used particular credentials. This should all sound somewhat familiar if you've been running your lab for years, it'd be hard to avoid situations when you'd want to know who did what.


    So,
    /home shouuld be unique to the logged in User.
    And, you can mouunt network resources separately and make access to those resources easy to find... Like a shortcut on the Desktop. Or, a website with links that automatically connect to, or launch those mount points. Or, web pages with instructions. Or, whatever else which can be imagined.


    Now,
    As for installing ceph, you can if you wish
    https://en.opensuse.org/openSUSE:Ceph
    And, you can install Glusterfs if you wish
    https://software.opensuse.org/package/glusterfs


    But, you'd have to really consider whether either of those addresses any need. Do Users really need to share their results? Do your lab exercises require intensive writing to a shared resource?


    Typically most lab exercises I'm familiar with do nothing of the above, every exercise only reads and doesn't write to a shared resource. Users/students run exercieses individually or as individual groups where the results are stored as "own" files which even if uploaded somewhere would still be an "own" location, not used by others to do additional work.


    Note that if you're setting up an application to run in a lab (or otherwise), that would be different. Any time you <run an application>. the specific requirements for that stand alone.


    The requirements to support what I just described are relatively modest... You might cluster as one strategy to support fault tolerance, but you shouldn't have any real issues with performance or contention of any type even today before considering any changes. If your requirements haven't changed, then I'd ask why you might want to change what has worked for you, and what you're familiar with.


    HTH and IMO,
    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  3. #3

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    On 07/08/2018 08:56 PM, tsu2 wrote:

    What was originally proposed, and then what you are writing here at the
    top of your post, I think were different things entirely.

    > Let's first consider one of your proposed ideas which is to mount /home
    > as a shared directory used by all.


    In a normal system, /home is a shared directory used by all, but just not
    directly; it is shared and used by 'joe' to provide /home/joe, and the
    'susie' to provide /home/susie, etc. Maybe the OP meant it your way, but
    that just sounds crazy, as you pointed out below.

    > In principle, that's not ordinarily advisable.
    > The /home partition is typically associated with a User profile which is
    > tied to the logged in User. A "better" decision is for every person on
    > your network to log in with their own personal, unshared credentials so
    > that person's activities are unique. It's the only way that you can have
    > accountability. Once Users start sharing login credentials and resources
    > on the network using same credentials, you've lost any way of
    > identifying who did what and when. Even if everyone on your network is
    > your most trusted family, things can get sticky when odd things happen
    > and people point fingers at each other and you have no way to identify
    > the cause of such and such was at such time by the User who used
    > particular credentials. This should all sound somewhat familiar if
    > you've been running your lab for years, it'd be hard to avoid situations
    > when you'd want to know who did what.


    It did not sound, to me, like the proposal was to simply have everybody
    write directly into a shared /home directory, but to mount the /home to
    all systems and then still have separate directories within there for each
    user, with appropriate/normal permissions.

    > But, you'd have to really consider whether either of those addresses any
    > need. Do Users really need to share their results? Do your lab exercises
    > require intensive writing to a shared resource?
    >
    > Typically most lab exercises I'm familiar with do nothing of the above,
    > every exercise only reads and doesn't write to a shared resource.
    > Users/students run exercieses individually or as individual groups where
    > the results are stored as "own" files which even if uploaded somewhere
    > would still be an "own" location, not used by others to do additional
    > work.


    I guess my experience here is more like the OP's; while in many academic
    settings each user should do their own assignment sans collaboration, I've
    also seen a lot of work in both academia and the commercial word where
    sharing is the norm among teams. Maybe sharing within /home is not the
    norm, but having /home available (with subdirectories) for each user, and
    then having another shared location (maybe /home/finance-group or
    /var/finance-group or whatever) is also necessary for projects. Sure,
    users can share files other ways, but depending on the task at hand,
    sharing directly with a group is much more efficient, especially as the
    groups get beyond a few individuals.

    Just a few more opinion, worth what anybody gets from them.

    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below.

    If you want to send me a private message, please let me know in the
    forum as I do not use the web interface often.

  4. #4
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,130
    Blog Entries
    2

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    @ab,
    Agree with everything you posted.

    But,
    If collaboration on a project is the desired organization,
    Then I'd suggest various common solutions...
    - Network security like LDAP or AD support "single sign-on" to grant group level permissions to resources.
    - A web based solution, eg a Wiki is commonly used as a gathering point. Users are granted permissions to contribute at various levels. A CMS might also be used.

    Either of the above can then be applied to services and resources like File Shares, specific applications, more.

    And, perhaps I was too quick to assume that a shared /home would mean shared User settings, file storage, etc.
    But, if the goal is collaboration, nothing comes to my mind at the moment that would be simply deployed in /home although it's certainly possible to store <settings> that point to collaborative resources.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  5. #5
    Join Date
    Oct 2008
    Location
    Glasgow, Scotland
    Posts
    1,131

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    I have probably got this wrong, but I read the OP differently to both tsu2 and ab. I took the critical phrases to be “distributed file system”, “reliability/redundancy”, “accessibility”. Mounting, NFS etc. just seemed means to an end.

    The use of a striped filesystem across multiple machines and resiliency are contradictory. Whether a suitable compromise is available would depend on the numbers of machines (old and new that were available. Hard drive (spinning) space is cheap. I would abandon striping and consider fitting the older machines with larger hard drives to avoid them being dependent on a cluster or partnered new machine.

    Assuming that the application data files are stored (student read-only) in “/home/chemlabdata/” or somesuch, and that we have only a small number (less than 100?) of active students.
    Use rsync to synchronise the each local “/home” files with the most recent versions available in the lab at the start of the day and/or at user login. At logout use an rsync script to update “$HOME” on the other machines to the local version. If a student/user makes a mess of a session, they can simply crash out (<ctrl+alt+backspace>) of the session without logging out so as to start again. Combined with an out-of-lab backup this gives a lot of resiliency with minimum effort and is easy to administer.
    I have used this approach with rooms of 30 machines and 90+ users in the past -- BSD Unix and SuSE/openSUSE Linux.
    ~Thank you for sharing an interesting problem.
    --
    slàinte mhath,
    rayH

  6. #6
    Join Date
    Sep 2008
    Location
    Porto Alegre, RS, Brazil
    Posts
    72

    Talking Re: Looking for advise on distributed filesystem choice for opensuse

    Hi guys.

    First, thank you all for the prompt replies! I wasn't expecting it to be so fast.

    Second, let me clarify the misunderstandings pointed out mostly by tsu: I really meant that /home was shared over the network, but each user (student in this case) only have access to its own /home/$USERNAME directory (when in research clusters where there is need for different users to access the same files we deal with the XFS capabilities on demand).

    In this way, every student had its own files readily available on every single computer: except for network or server failures, they would be resilient to single seat failures of any kind.

    Also: there is no way on using NFS now because each computer we got has a singles 500gb disk that, also, both bureaucratic and physically cannot be put together into a single NFS file server machine. That is why we are considering a distributed file system to be in place. Widely well-known Brazilian economic crisis accounts for the impossibility to upgrade the older computers (getting new ones in place of the totally broken ones is already enough of a miracle nowadays).

    As such, our main problem is: what would be the recommended approach to make a "some sort of over the network raid among different computers", that would allow for both the striping of data that improves space usage and redundancy that allows for its resilience?

    I see it like the following: Let's suppose I have a data "ABCDEF", that spread across 6 computers would be:
    1: ABC
    2: BCD
    3: CDE
    4: DEF
    5: EFA
    6: FAB

    Somewhat like raid 6 would be, but over the network here. I think a very well chosen distributed filesystem is the answer, but which one, and under what configurations?

    Thanks a lot for all the answers!

  7. #7
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,130
    Blog Entries
    2

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    For what you're describing you might also take a look at drbd.
    Keep in mind, there is a good chance this is overkill, but it can deliver the best solution supporting High Availability.
    I was going to set this up on a project years ago but the SUSE/openSUSE implementation was broken at that time so I had to pass(Didn't want to take the time to learn the whole thing on RHEL)...

    But, in response to a bug I submitted,
    DRBD is supposed to be fixed and it looks like besides the standard User-space management tools, it looks like there is even a YaST module (That's cool, isn't it?).

    The Wikipedia entry
    https://en.wikipedia.org/wiki/Distri...d_Block_Device
    SLES documentation
    https://www.suse.com/documentation/s...configure.html
    https://www.suse.com/documentation/s...configure.html

    Note that this is the kind of thing that you set up when you need really fast performance across a network and ordinarily is very expensive to set up... This is the kind of thing you might consider if something like rsync isn't suitable for some reason, like the files are too large, you need to minimize latencies, nearly real time mirroring, etc.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  8. #8
    Join Date
    Jun 2008
    Location
    Auckland, NZ
    Posts
    20,279
    Blog Entries
    1

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    I concur. Investigate using DRBD for a cost-effective open source solution.

    https://en.opensuse.org/openSUSE:High_Availability
    openSUSE Leap 15.0; KDE Plasma 5

  9. #9
    Join Date
    Sep 2008
    Location
    Porto Alegre, RS, Brazil
    Posts
    72

    Smile Re: Looking for advise on distributed filesystem choice for opensuse

    Thanks for the suggestion. However, it seems that DRBD is still not the right solution for the problem at hand.

    As you mentioned, rsync is not a solution due to both big files usage (100s gb easily) and the real time mirroring needed (say a computer fails during class, student must just jump to the next seat and continue working: undoable with rsync solutions).

    DRBD also have a big issue: while having a dedicated yast module is lovely, it only "kind of mimics" a RAID 1 configuration, and a striping+parity of something closer to a RAID 6 is a requirement (yes, I'm considering the possibility of a second failure before the request fix for a first failure arrives: it already happened more often than once! ).

    From this solution, however, I learned about the NBD, which while a very interesting idea for other problems, still doesn't "quite fill the gap" needed: unfortunately, as it only allows for a single read-write access, it can have only one single NFS server working on it, thereby creating again a "single point of failure".

    I spent a good chunk of the day studying both gluster and ceph. Gluster is almost out of the poll: while pretty easy to install, the impossibility of using a simple "mv" command for renaming together with the whole file storage approach it seems to use is a show stopper. I cannot, for example, have a situation where I have 4x400gb disks (not even considering parity here just for the sake of the argument) where I might have issues in writing 5x300gb files. Other smaller issues also seemed to "float around it".

    Ceph seems like a good option, however it makes me concern its metadata server approach: it now can have more than one metadata server, but I'm unsure how parity works among them (so, if I loose one or two of them, will the rest still be able to serve the whole system and the files accessible?), and if I can put a server on each computer together with the actual files storage, or if I have to have dedicated computers (which would be unacceptable and total show stopper once again).

    Does anybody now the answer for the above questions (tomorrow I'll post them also on ceph forums or alike)? Any alternative, both on distributed filesystems or any other completely different approach suggestion?

    Once again, thanks for all the help up to now.

  10. #10
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,130
    Blog Entries
    2

    Default Re: Looking for advise on distributed filesystem choice for opensuse

    The thing that makes DRBD lightning fast for mirroring large files is that you're talking about <block> replication, ie only the changes at the disk block level are replicated.

    Take for example a 4GB data file, it can be an enormous spreadsheet (?!) or a RDBMS file, and only one tiny part of the file is changed, like a single sentence or a single data point. If you used rsync, the entire 4GB file would have to be replicated to maintain the mirror's integrity. With DRBD, only the changed data in 256k or 512k blocks would be replicated, perhaps totaling a transfer of only 4MB. That's a big difference in network transfer, 4GB vs 4MB.

    Replication is also a one-way transfer, replication is not the same as synchronization, so it's not the same as RAID 1 when both members of the set are considered both master and slave.

    Think of DRBD this way... it can be the backbone of High Availability in that you can have at least one and possibly many nearly exact clones of an original machine(or part of machine) every second... Without DRBD, alternatives for setting up high performance replication or something similar to RAID 1 are usually limited to some specific application and may not perform as well.

    Because of the limitations of replication, DRBD cannot be considered the ultimate and perfect solution for fault tolerance, but it can uniquely make some things possible that wouldn't be possible or practical any other way.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •