Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: bash script - parameter max size lengthg

  1. #1
    Join Date
    Jun 2008
    Location
    South-West France
    Posts
    1,024

    Default bash script - parameter max size lengthg

    Hello.

    The question is :
    Is there a size limits when passing argument to a program ( a binary program or a bash script )
    I start a script from desktop.
    Code:
    Comment[en_US]=my install edit
    Comment=my install edit
    Exec=/bin/bash -c '/home/some_user/bin/200_install/some_script_1_launcher'
    GenericName[en_US]=my install edit
    GenericName=my install edit
    Icon=apparmor_edit_profile
    MimeType=
    Name[en_US]=my install edit
    Name=my install edit
    Path=/home/some_user/bin/200_install
    StartupNotify=true
    Terminal=true
    TerminalOptions=\s--noclose
    Type=Application
    X-DBUS-ServiceName=
    X-DBUS-StartupType=none
    X-KDE-SubstituteUID=false
    X-KDE-Username=
    This launcher script set some variables and then start the main script
    /home/some_user/bin/200_install/some_script_2_main :

    /home/some_user/bin/200_install/some_script_1_launcher :
    Code:
    MY_PATH="/home/some_user/bin/200_install"
    MY_USR="/backup_sys/000_COMMON/Bin"
    MY_SU_USR="/backup_sys/000_COMMON/SBin"
    MY_SU_USR2="/backup_sys/000_COMMON/SBin2"
    MY_ETC="/etc"
    MY_SYSTEMD="/etc/systemd/system"
    MY_LIB_SYSTEMD="/usr/lib/systemd/system"
    #
    MY_FILE1="$MY_SU_USR2/user_opensuse_create_standard_user  $MY_SU_USR2/user_opensuse_put_public_standard_user    $MY_SU_USR2/zypper_search_for_repos_from_lib_name_grep_lancher   $MY_SU_USR2/zypper_search_for_repos_from_lib_name_lancher  $MY_PATH/yast_00_1_1_A_swap_make_swap_on_file  $MY_PATH/yast_00_1_2_A_create_public_folder  $MY_PATH/yast_00_1_3_A_create_local_folder  $MY_PATH/yast_00_1_4_A_create_zypper_local  $MY_PATH/yast_00_1_5_A_rename_yast_install_repo  $MY_PATH/yast_00_1_6_A_disable_yast_debug_repo  $MY_PATH/yast_00_1_7_A_adjust_locale_LC  $MY_PATH/yast_00_1_8_A_adjust_etc_files  $MY_PATH/yast_00_1_9_A_disable_services  $MY_PATH/yast_00_0_reset_flag_install   $MY_PATH/yast_00_0_test_yast_first_pass_done  $MY_PATH/yast_00_1_my_install_edit  $MY_PATH/yast_00_1_my_install_run  $MY_PATH/yast_00_2_my_app_file  $MY_PATH/yast_00_2_my_function  $MY_PATH/yast_00_2_my_app_version  $MY_PATH/yast_01_set_initial_ALL  $MY_PATH/yast_02_set_initial_repos  $MY_PATH/yast_03e_install_ALL_libraries  $MY_PATH/yast_04c_install_basic_libraries_leap  $MY_PATH/yast_05c_install_some_files  $MY_PATH/yast_06_install_NOTEBOOK_aplication_files  $MY_PATH/yast_10_create_main_repos  $MY_PATH/yast_11_create_my_repos  $MY_PATH/yast_11a_create_my_package_repo  $MY_PATH/yast_13_list_repos_enabled  $MY_PATH/yast_14_remove_main_repos  $MY_PATH/yast_15_remove_my_repos  $MY_PATH/yast_16_activate_main_repos_only  $MY_PATH/yast_17_activate_my_repos  $MY_PATH/yast_18_deactivate_main_repos  $MY_PATH/yast_19_deactivate_my_repos   $MY_PATH/yast_20_reinstall_brother_printer  $MY_PATH/yast_21_reinstall_samsung_printer  $MY_PATH/yast_22_reinstall_epson_scanner   $MY_PATH/yast_26_more_install_software   $MY_PATH/yast_27_download_individual_file    $MY_PATH/yast_29_complementary_task   $MY_PATH/yast_30_install_systemd_files  $MY_PATH/yast_31_remove_systemd_files  $MY_PATH/yast_32_add_printers_and_scanners  $MY_PATH/yast_96_make_public_directories    $MY_PATH/yast_97_create_standard_user  $MY_PATH/yast_98_copy_my_etc_files  $MY_PATH/yast_99_copy_initial_zypper_files  $MY_PATH/yast_100_download_current_kernel  $MY_PATH/yast_101_download_new_kernel  $MY_PATH/yast_102_install_A_kernel  $MY_PATH/yast_103_install_freenx  $MY_PATH/yast_106_install_virtualbox  $MY_PATH/yast_109_install_skype  $MY_PATH/yast_150_install_mysql_mariadb  $MY_PATH/yast_151_de-install_mysql_mariadb $MY_PATH/yast_152_install_bacula  $MY_PATH/yast_153_de-install_bacula $MY_PATH/yast_154_install_postfix  $MY_PATH/yast_155_de-install_postfix  "
    #
    export MY_FILE1
    
    /home/some_user/bin/200_install/some_script_2_main "$MY_FILE1"
    /home/some_user/bin/200_install/some_script_2_main :
    Code:
    kdesu --noignorebutton -c "kate $MY_FILE "
    MY_FILE1 contains 4044 characters.
    There is no problem with MY_FILE1 set as this.

    Now if I add a few more characters in MY_FILE1
    Code:
      $MY_PATH/yast_160_reinstall_player
    that does not work any more. Kate trunc the data and the last file to be edited is
    Code:
    yast_160_re
    The full path name in kate is
    Code:
    /home/some_user/bin/200_install/yast_160_re
    MY_FILE1 then contains 4105 characters.
    There is a problem with MY_FILE1 set as this, not all of the necessary files are edited.

    Any help is welcome.
    Thanks for helping. JCD
    __________

    server leap 15-- ASUS g75vw KDE leap 42.3 -- ASUS g750JZ KDE leap 42.3 -- acer aspire s13 win 10 home -- HP Omen win 10 home - scan EPSON V500 - Brother HL2250DN - Samsung CLP-325W

  2. #2

    Default Re: bash script - parameter max size lengthg

    Write the long string to a file and then iterate over that file's contents
    rather than trying to use that as one big list of parameters.
    Alternatively, echo the big long list and capture it in some script using
    the something like the 'read' command so that you can iterate over the
    list that way.


    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

  3. #3
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    10,927
    Blog Entries
    2

    Default Re: bash script - parameter max size lengthg

    Although I have no formal instruction/education in BASH data size limits, I've encountered what you describe a few times...
    A simple Google search suggests that "ulimit" will display set limits that are relevant

    The BASH online manual
    ftp://ftp.gnu.org/old-gnu/Manuals/ba...href_5.es.html

    A likely relevant stackexchange post
    http://stackoverflow.com/questions/1...-lines-at-4095

    In my own words,
    What I found was that if the data is processed through an interactive shell, or in your case the equivalent using another app like Kate which processes the data completely before inputting the entire data set into your main script, the data is processed as a batch job.

    If this is actually what is happening to you, then the above (particularly the second) link describes how to change the buffer size limits, but in my case I decided instead to feed the data directly to the script without going through the intermediate step of an interactive shell so that the data is streamed and processed FIFO instead of as a batch job. So, if for instance you were to do what I opted to do, feed your data directly into your processing script without first opening in Kate... Or, at least if I understand what your script is trying to do, design your script to more efficiently store patterns before dedupping... Maybe even use a distributed dedup script instead of a home made script.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  4. #4
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    10,927
    Blog Entries
    2

    Default Re: bash script - parameter max size lengthg

    Quote Originally Posted by ab View Post
    Write the long string to a file and then iterate over that file's contents
    rather than trying to use that as one big list of parameters.
    Alternatively, echo the big long list and capture it in some script using
    the something like the 'read' command so that you can iterate over the
    list that way.


    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...
    Agreed, my thought was similar...

    Hash the first entry and write to an array,
    Then loop the following...
    Hash the next entry, if it doesn't match anything in the array then append the new hash to the array.
    If it matches, then you know you have a dupe, so then you can do what you want(create a new list? immediately remove?)

    Compared to storing a list of the original full text string in your comparison list, my process would require more processing initially but as the enormous list grows, I'd expect that comparing hashes should be much faster at some point.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  5. #5
    Join Date
    Jun 2008
    Location
    South-West France
    Posts
    1,024

    Default Re: bash script - parameter max size lengthg

    Quote Originally Posted by ab View Post
    Write the long string to a file and then iterate over that file's contents
    rather than trying to use that as one big list of parameters.
    Alternatively, echo the big long list and capture it in some script using
    the something like the 'read' command so that you can iterate over the
    list that way.


    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...
    I am not iterating a process.
    Just starting kate with a long list of files as a root.
    Thanks for helping. JCD
    __________

    server leap 15-- ASUS g75vw KDE leap 42.3 -- ASUS g750JZ KDE leap 42.3 -- acer aspire s13 win 10 home -- HP Omen win 10 home - scan EPSON V500 - Brother HL2250DN - Samsung CLP-325W

  6. #6
    Join Date
    Jun 2008
    Location
    South-West France
    Posts
    1,024

    Default Re: bash script - parameter max size lengthg

    Quote Originally Posted by tsu2 View Post
    Although I have no formal instruction/education in BASH data size limits, I've encountered what you describe a few times...
    A simple Google search suggests that "ulimit" will display set limits that are relevant

    The BASH online manual
    ftp://ftp.gnu.org/old-gnu/Manuals/ba...href_5.es.html

    A likely relevant stackexchange post
    http://stackoverflow.com/questions/1...-lines-at-4095

    In my own words,
    What I found was that if the data is processed through an interactive shell, or in your case the equivalent using another app like Kate which processes the data completely before inputting the entire data set into your main script, the data is processed as a batch job.

    If this is actually what is happening to you, then the above (particularly the second) link describes how to change the buffer size limits, but in my case I decided instead to feed the data directly to the script without going through the intermediate step of an interactive shell so that the data is streamed and processed FIFO instead of as a batch job. So, if for instance you were to do what I opted to do, feed your data directly into your processing script without first opening in Kate... Or, at least if I understand what your script is trying to do, design your script to more efficiently store patterns before dedupping... Maybe even use a distributed dedup script instead of a home made script.

    TSU
    from https://www.in-ulm.de/~mascheck/various/argmax/

    Number of arguments and maximum length of one argument

    At least on Linux 2.6, there's also a limit on the maximum number of arguments in argv[].
    On Linux 2.6.14 the function do_execve() in fs/exec.c tests if the number exceeds PAGE_SIZE*MAX_ARG_PAGES-sizeof(void *) / sizeof(void *) On a 32-bit Linux, this is ARGMAX/4-1 (32767). This becomes relevant if the average length of arguments is smaller than 4. Since Linux 2.6.23, this function tests if the number exceeds MAX_ARG_STRINGS in <linux/binfmts.h> (2^32-1 = 4294967296-1).
    And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072).
    This might become relevant if you generate a long call like "sh -c 'automatically generated with many arguments'".
    (pointed out by Xan Lopez and Ralf Wildenhues)

    Code:
    superjc@MY-SERVER-LINUX:~> getconf ARG_MAX 
    2097152
    superjc@MY-SERVER-LINUX:~>

    Code:
    VAR=`perl -e 'print "a"x131071'` ; bash /backup_sys/000_COMMON/Bin/test.sh $VAR
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.........
    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    Print 131071 "a" without problem.
    Thanks for helping. JCD
    __________

    server leap 15-- ASUS g75vw KDE leap 42.3 -- ASUS g750JZ KDE leap 42.3 -- acer aspire s13 win 10 home -- HP Omen win 10 home - scan EPSON V500 - Brother HL2250DN - Samsung CLP-325W

  7. #7
    Join Date
    Jun 2008
    Location
    South-West France
    Posts
    1,024

    Default Re: bash script - parameter max size lengthg

    The problem comes fron kdesu ( see /home/some_user/bin/200_install/some_script_2_main in thread #1 )

    Removing kdesu, kate start normally and all files are present.

    Any help is welcome
    Thanks for helping. JCD
    __________

    server leap 15-- ASUS g75vw KDE leap 42.3 -- ASUS g750JZ KDE leap 42.3 -- acer aspire s13 win 10 home -- HP Omen win 10 home - scan EPSON V500 - Brother HL2250DN - Samsung CLP-325W

  8. #8
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    10,927
    Blog Entries
    2

    Default Re: bash script - parameter max size lengthg

    What ab and I were suggesting is to not invoke kate at all, bypass that step entirely.
    Is there some reason why you're processing your data through kate?

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  9. #9

    Default Re: bash script - parameter max size lengthg

    Quote Originally Posted by jcdole View Post
    I am not iterating a process.
    Just starting kate with a long list of files as a root.
    Hi,

    Right then iterate through those files?
    "Unfortunately time is always against us" -- [Morpheus]

    .:https://github.com/Jetchisel:.

  10. #10
    Join Date
    Jun 2008
    Location
    South-West France
    Posts
    1,024

    Default Re: bash script - parameter max size lengthg

    Quote Originally Posted by tsu2 View Post
    What ab and I were suggesting is to not invoke kate at all, bypass that step entirely.
    kate?TSU
    Noted. I shall make a try next time.


    Quote Originally Posted by tsu2 View Post
    Is there some reason why you're processing your data through kate?TSU
    Yes .
    I use a script to help me when I install/reinstall linux on a computer. So I load all the scripts and all the config files at ounce. So I have everything at hand if things are going wrong.
    Thanks for helping. JCD
    __________

    server leap 15-- ASUS g75vw KDE leap 42.3 -- ASUS g750JZ KDE leap 42.3 -- acer aspire s13 win 10 home -- HP Omen win 10 home - scan EPSON V500 - Brother HL2250DN - Samsung CLP-325W

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •