Page 1 of 2 12 LastLast
Results 1 to 10 of 17

Thread: Bash script, strings limited to 128 characters?

  1. #1

    Default Bash script, strings limited to 128 characters?

    Code:
    #!/bin/bash #: Title       : string limits
    IFS=''
    
    #read line output of 'clamscan -r' 
    while read command_output; do
    
         printf "%s\n" ${command_output}
    
    done < <(clamscan -r /)
    
    unset IFS

    With this, I found that strings were being cut off at 128 characters. Using IFS field splitting makes my code far more complex. With IFS, string length can still be exceeded.
    I've found nothing on internet searches to make strings bigger. string_size=128 to string_size=2000.

    Can you make string length maximum larger? Through syntax or bash system variables.

  2. #2

    Default Re: Bash script, strings limited to 128 characters?

    It'd be helpful to see the output you get, as well as the output you expect.

    Using the following I easily get lines longer than 128 characters. I
    would guess something about your clamav output is breaking the line.
    Maybe try 'read -r' instead of just 'read':

    Code:
    #!/bin/bash
    IFS='';
    while read command_output; do
    printf "%s\n" ${command_output}
    done < <(openssl rand -hex 256)
    --
    Good luck.

    If you find this post helpful and are logged into the web interface,
    show your appreciation and click on the star below...

  3. #3
    Join Date
    Sep 2012
    Posts
    5,230

    Default Re: Bash script, strings limited to 128 characters?

    I cannot reproduce it either. In quick testing bash happily read single line over 240K in length.

  4. #4
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,474
    Blog Entries
    2

    Default Re: Bash script, strings limited to 128 characters?

    This is likely the interactive input buffer which I've run into in the past.

    I ran into this awhile back following the original SDB Java (Oracle SE JDK 7) instructions,
    If you simply copied the original code comprising well over a hundred individual commands into a console, you'd run into that error (limiting number of characters in a command).
    The alternative at that time was to paste only a single or at most a few commands at a time into a console.

    My guess is that your output is being sent to a command for input in a new command.

    Solution which I implemented in the scripts I wrote...
    Change the commands from interactive to non-interactive.
    Then, all the commands can be run at once.

    I would guess that your problem is "printing" the string is the problem and might be resolved by simply piping or redirecting the text stream directly to file.

    In another thread (IIRC it's also likely authored by you) I suggested writing all your code as non-interactive instead of invoking interactive commands. I would repeat that recommendation, I would guess there should also be performance benefits.

    TSU

  5. #5

    Default Re: Bash script, strings limited to 128 characters?

    I'll try piping the output and see if that fixed the problem.

    I'll post the complete code section, same basic idea. This has the output bug. It prints only if 'found' or 'moved'. This cuts the string short and outputs.
    See below for the piped version.

    Code:
            #!/bin/bash
    
             IFS=':'
             
             while read -ra command_output_filescan
             do             
                 #delete line
                 echo -en "\E[2K\r"
                 
                 Max_Words=${#command_output_filescan[*]}
                 Last_Word=${command_output_filescan[Max_Words]}
    
                 let START=0
                 let END=Max_Words-1
                 
                 #print string
                 Index=0
                 for index_word in "${command_output_filescan[@]}"
                 do
                   printf "%s" $index_word
                   if [ $Index != $END ];then
                     printf ":"
                   fi
                   ((Index +=1))
                 done
     
                 Scanfile_Result=${command_output_filescan[END]}
    
                 if [[ "$Scanfile_Result" == *"FOUND"* ]] || [[ "$Scanfile_Result" == *"moved to"* ]];then
                   printf "\n"
                 fi
    
             done < <(clamscan -r / --exclude-dir=/sys --exclude-dir=/proc --exclude-dir=/dev --exclude-dir=/.snapshots --follow-dir-symlinks=0 --follow-file-symlinks=0)
    ... end
    /home/username/Downloads/How to Enable or Disable_Turn off Fast boot or Fast Startup in Windows 10 and Windows 8.1 « SillyCodes_files/postmessageRelay.ht
    /home/username/Downloads/How to Enable or Disable_Turn off Fast boot or Fast Startup in Windows 10 and Windows 8.1 « SillyCodes_files/comment-iframe.html
    /home/username/Downloads/How to Enable or Disable_Turn off Fast boot or Fast Startup in Windows 10 and Windows 8.1 « SillyCodes_files/tweet_button.html:
    /home/username/Downloads/How to Enable or Disable_Turn off Fast boot or Fast Startup in Windows 10 and Windows 8.1 « SillyCodes_files/TlA_zCeMkxl_002.htm
    /home/username/Downloads/How to Enable or Disable_Turn off Fast boot or Fast Startup in Windows 10 and Windows 8.1 « SillyCodes_files/TlA_zCeMkxl.html: O
    /home/username/.mozilla/firefox/0ywv6puy.default/datareporting/archived/2015-12/1450591995453.a420be3a-5bb9-4866-9b88-24f13bfebe0d.main.jsonlz4: Empty fi
    /home/username/bin/scanvirus: line 242: command_output_filescan: bad array subscript



    (cont)

  6. #6

    Default Re: Bash script, strings limited to 128 characters?

    Code:
             IFS=':'
             
             clamscan -r / --exclude-dir=/sys --exclude-dir=/proc --exclude-dir=/dev --exclude-dir=/.snapshots --follow-dir-symlinks=0 --follow-file-symlinks=0 | while read -ra command_output_filescan
             do
                 
                 #delete line
                 echo -en "\E[2K\r"
                 
                 Max_Words=${#command_output_filescan[*]}
                 Last_Word=${command_output_filescan[Max_Words]}
    
                 let START=0
                 let END=Max_Words-1
                 
                 Index=0
                 for index_word in "${command_output_filescan[@]}"
                 do
                   printf "%s" $index_word
                   if [ $Index != $END ];then
                     printf ":"
                   fi
                   ((Index +=1))
                 done
     
                 Scanfile_Result=${command_output_filescan[END]}       #bad array subscript
                 #printf "%s" $Scanfile_Result
                 if [[ "$Scanfile_Result" == *"FOUND"* ]] || [[ "$Scanfile_Result" == *"moved to"* ]];then
                   printf "\n"
                 fi
    
                 #Scanfile_Result=${command_output_filescan:Index}
     
                 #break
             done
                
             unset IFS

    Code:
    /run/media/username/KINGSTON/Oblivion/oblivion tech/oblivion CS WIKI tutorials/File DialogTute-New_topics.jpg_files/120px-DialogTute-New_topics_002.jpg: 
    /run/media/username/KINGSTON/Oblivion/oblivion tech/oblivion CS WIKI tutorials/File DialogTute-Quest_Stages.jpg_files/120px-DialogTute-Quest_Stages.jpg: 
    /run/media/username/KINGSTON/Oblivion/oblivion tech/oblivion CS WIKI tutorials/File DialogTute-Quest_Stages.jpg_files/800px-DialogTute-Quest_Stages.jpg: 
    /run/media/username/KINGSTON/Oblivion/oblivion tech/oblivion CS WIKI tutorials/From_3dsmax_to_Oblivion_CS_files/Max9ToCSAlphaBlendingWithoutTesting.jpg: 
    /run/media/username/KINGSTON/Oblivion/oblivion tech/OMOD script info/How_to_convert_an_OMOD_to_an_OMOD-ready_archive_files/poweredby_mediawiki_88x31.png:
    /run/media/username/KINGSTON/Oblivion/oblivion tech/Script Tutorials/A_beginner's_guide,_lesson_1_-_The_Construction_Set_Primer_files/poweredby_mediawiki
    /run/media/username/KINGSTON/Oblivion/oblivion tech/Script Tutorials/A_beginner's_guide,_lesson_2_-_Creating_a_room_files/poweredby_mediawiki_88x31.png: 
    /run/media/username/KINGSTON/Oblivion/oblivion tech/Script Tutorials/A_beginner's_guide,_lesson_4_-_Anatomy_of_a_quest,_part_1_files/poweredby_mediawiki_
    /run/media/username/KINGSTON/Oblivion/oblivion tech/Script Tutorials/A_beginner's_guide,_lesson_5_-_Anatomy_of_a_quest,_part_2_files/poweredby_mediawiki_
    /run/media/username/KINGSTON/Oblivion/oblivion tech/Script Tutorials/A_beginner's_guide,_lesson_6_-_Quest_Dialogue_files/poweredby_mediawiki_88x31.png: O
    line 242: command_output_filescan: bad array subscript
    It's still cutting strings at 128 characters.

  7. #7
    Join Date
    Sep 2012
    Posts
    5,230

    Default Re: Bash script, strings limited to 128 characters?

    Code:
    cn1:~ # { for i in {1..128}; do echo -n $i " "; done; echo :FOUND; }  | wc -lc
          1     539
    cn1:~ # { for i in {1..128}; do echo -n $i " "; done; echo :FOUND; }  | /tmp/foo.sh
    1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99  100  101  102  103  104  105  106  107  108  109  110  111  112  113  114  115  116  117  118  119  120  121  122  123  124  125  126  127  128  :FOUND
    Where /tmp/foo.sh is copy-paste of your code.

  8. #8
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,474
    Blog Entries
    2

    Default Re: Bash script, strings limited to 128 characters?

    Am interested, but unable to understand <exactly> what you are trying to accomplish.

    Looks like you're running a clamscan scan, and you've generated an output file.

    But, after that I'm uncertain what you are trying to do,
    You've decided you want to modify the IFS to a full colon (
    But can't see what you're trying to do...
    You suggest you are deleting a line (unknown why)
    Then you seem to be trying to identify a "last word" which is probably the last field determined by your custom IFS instead of the usual space character.

    I don't know what the following is supposed to mean, do you believe this is the line that causes your error?
    Code:
    Scanfile_Result=${command_output_filescan[END]}       #bad array subscript
    Also, there seems to be a lot of functionality that's not described... particularly "command_output_filescan."
    Don't know if any of that functionality is relevant to your error.

    In any case, there seems to be a large number of hits if you Google "bad array subscript" (there can be many causes) but in your case at least I don't see enough information to troubleshoot.

    TSU

  9. #9

    Default Re: Bash script, strings limited to 128 characters?

    Quote Originally Posted by tsu2 View Post
    Am interested, but unable to understand <exactly> what you are trying to accomplish.

    Looks like you're running a clamscan scan, and you've generated an output file.

    But, after that I'm uncertain what you are trying to do,
    You've decided you want to modify the IFS to a full colon (
    But can't see what you're trying to do...
    You suggest you are deleting a line (unknown why)
    Then you seem to be trying to identify a "last word" which is probably the last field determined by your custom IFS instead of the usual space character.

    I don't know what the following is supposed to mean, do you believe this is the line that causes your error?
    Code:
    Scanfile_Result=${command_output_filescan[END]}       #bad array subscript
    Also, there seems to be a lot of functionality that's not described... particularly "command_output_filescan."
    Don't know if any of that functionality is relevant to your error.

    In any case, there seems to be a large number of hits if you Google "bad array subscript" (there can be many causes) but in your case at least I don't see enough information to troubleshoot.

    TSU
    The bad array subscript is because of blank lines in clamscan output. Before the scan results at the end, there shouldn't be any blank lines. So, I check for them. I managed to find some programing errors and fix them. This generally what it does.

    send clamscan output to while loop

    delete line: printline

    As each line is printed it erases the previous line. If 'found' is in the line it prints a newline so it doesn't get deleted. So, the printout only shows if a virus is found. scanvirus - linux scan

    It works now, but I'v not had a chance to test it more with test it more with long lines and fake virus lines. It's usable by itself, but scan results don't show up. The last array field holds the scan results, after the last ':'. I just added a check for blank lines and this.

    Code:
             clamscan -r / --exclude-dir=/sys --exclude-dir=/proc --exclude-dir=/dev --exclude-dir=/.snapshots --follow-dir-symlinks=0 --follow-file-symlinks=0 | while read -r -a command_output_filescan -n 120
    This should do a double field split read up to 120 characters at a time and split that field based on IFS=":".

    I'll post again when I have more time.

  10. #10

    Default Re: Bash script, strings limited to 128 characters?

    I think I'v figured it out. The bash 'read' command has buffer limit of 128 characters.


    IFS=':'


    clamscan -r / | while read -a output_line
    do
    end


    read characters until ':' then split into next array element. If at the max character limit of 128, split the array.

    I'v have to do some coding to take this into account.

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •