Reading raw data using a shell script?

English Programming/Scripting
1

Is there a way to read plain old raw data from a file using a shell script?
Preferably with a way to get the data from a specific point in a file?

2

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yes… cat.

man cat

For a specific point in a file it may be slightly more interesting but
should still be possible with various commands (dd, sed, etc.) depending
on what you mean exactly by “raw data” and “specific point”.

Good luck.

On 06/28/2010 02:16 PM, ns89 wrote:
>
> Is there a way to read plain old raw data from a file using a shell
> script?
> Preferably with a way to get the data from a specific point in a file?
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJMKQSEAAoJEF+XTK08PnB5h6wP/R48dbwf0RTgGFMucsgRZSIo
vQx903CedCbBKTBck07XnAogBKQuF2Z8pFxBahhEXL+BSbL9rY9s2vzB9uKd/kSd
cO+IMx5O/Ys7r2rypmX8ywdVYRs/b7TWr1PLqDx6WhpNK9utbHjnfNdeuSeeJn85
9Sz551US7648MjN2SH/muC3ALSbhNyPIQcwmNJsSqX9y5dmlths+Lol/C/ThVHIf
eRHGQL4dybkTnwp7yRTaA7sKAuCL9xWEyxc1FzC6E5jK1FrMbIbpk6bi6fz5pUju
MO/BXHrLk/aQ/jpFOdElGFDcgHmmyDi5CHbyFqfYuMUxteO0+/wn+k/oW/4QcqoR
k/wl/U3mB2BPjrFlLk3YqIRK38f2q8richxvWGZ9QsVJg7EZQhvt5vOj9hnZfzjI
tlXtwAI05PChLxfwfU7LRBkDYQGduBSzpUc7k9lFTmIUbgAJuxlSuwffokAikaQC
EWNIvdr71fiPQJ2IcHDtPEiQqWaqz/TaX55sFAetTNTSad7Tj6krHxCYt0ZlUGVV
/TdbfNFjx3y5AnYOBrfqmrWHhe1ddyJMM/7HgpGG9yX7GaSktyPDZYpDO1GUPBiB
SJeLOmciSlY8w8P3GEpfWqXgySYycNxkd+ZioA1AR/mAWbVIQXYqkAoAbx3lLapU
O/ABNu5L96x+gXvus17p
=JjsB
-----END PGP SIGNATURE-----

3

Hi,

You might consider AWK

Regards,
Barry.

4

On Mon, 28 Jun 2010 20:16:01 +0000, ns89 wrote:

> Is there a way to read plain old raw data from a file using a shell
> script?
> Preferably with a way to get the data from a specific point in a file?

How you’d do that depends on what you mean by “raw data” - are you
talking about binary blobs or text data?

Jim

–
Jim Henderson
openSUSE Forums Administrator

5

On Mon, 2010-06-28 at 20:16 +0000, ns89 wrote:
> Is there a way to read plain old raw data from a file using a shell
> script?
> Preferably with a way to get the data from a specific point in a file?
>
>

man dd

However, even after reading the “raw” data, how you manipulate in the
shell could prove problematic. Can you say exactly what you plan to do
with the raw data? If it’s just get it and write it, then I think
you’ll have no problem.

6

By “raw data” I meant binary data.
I would use od, but I can’t figure out how/if you can turn off the address lines.
I’m trying to make a shell script that reads EDID data from an EDID dump, which is in binary form.

7

Assuming that you want to remove the first word of od output:

#!/bin/bash
od ${1} | while read LINE
do      echo "${LINE#* }"
done

You can of course change the od call by adding options to your wish.

8

The only problem with that is od still prints the last address.

> sh temp.sh edid.bin
ff00 ffff ffff 00ff 6904 22f2 0101 0101
1230 0301 3080 781b c4ee a3f6 4a57 239c
4f11 bf54 00ef 4f71 8081 4081 0095 40a9
00b3 c0d1 0101 3a02 1880 3871 402d 2c58
0045 0cdd 0011 1e00 0000 fd00 3200 1f4c
1153 0a00 2020 2020 2020 0000 fc00 5600
3248 3632 200a 2020 2020 2020 0000 ff00
3800 4c42 514d 3053 3331 3631 0a34 0500
0000200

Any ideas on how to get rid of it?

9 
#!/bin/bash
od ${1} | head -n -1 | while read LINE
do      echo "${LINE#* }"
done

And be so nice to give my wonderfull script a nice name, set the x-bit for at least the owner and call it by name, not by calling* sh* (it being a bash and not and* sh *script).

10

On Mon, 2010-06-28 at 22:06 +0000, ns89 wrote:
> By “raw data” I meant binary data.
> I would use od, but I can’t figure out how/if you can turn off the
> address lines.
> I’m trying to make a shell script that reads EDID data from an EDID
> dump, which is in binary form.
>
>

I’d use dd instead of od… just me…

With that said, capturing the data inside of a variable and echoing out
is a problem with things like the NULL (0) byte apparently.

So… it all depends on what you are trying to get and manipulate really.

Alternatively, convert it do something usable using xxd (or similar) and
translate back out.

Consider:

  1. cat /dev/urandom >random.bin
    (interrupt this when you think you have enough random binary data)

  2. dd ibs=1024 seek=20 obs=1024 count=1 if=random.bin
    of=random-part1.bin

Using dd, I’m seeking 20 x 1024 into the binary file (make sure you let
it run long enough above) and extracting 1024 bytes to a file.

  1. data=dd ibs=1024 skip=20 obs=1024 count=1 if=random.bin | xxd

Now, using xxd, I extracted the same bytes in a hexdump like fashion
(text) into a shell variable.

  1. echo “$data” | xxd -r >random-part1-fromshell.bin

I then echoed the string through xxd to remake it a binary and output it
to a file.

  1. cmp random-part1.bin random-part1-fromshell.bin

Comparing the two files shows they are identical.

Also, I noticed that xxd has offset and length capability, so might not
even need something like dd.

HTH,
Chris

11

What is it you want to do with the data? Print out certain fields?

It’s not hard to do with a language like Perl, Python or Ruby. They have builtins or libraries to handle binary data. E.g. for Perl see the pack and unpack functions.

12

I have just decided to rewrite the thing as Perl script. I ran into some big problems when converting different bases.