Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: Minicom Loosing Data

  1. #1

    Default Minicom Loosing Data

    You know sometimes Mr. Murphy just won't give it a rest!

    I have had many "challenges" with this latest project. The most recent one I posted on these forums about how I can create a daily log file and someone suggested logrotate. Excellent, and now I have that working.

    Now I am at the next and hopefully last problem.

    First a little history, I have built a microprocessor board which allows me to log my hydro usage. It involves a photo diode on my hydro meter which counts flashes.

    Every 10 minutes I send the kWh out the serial port. Being power conscious I want to connect this to my servers serial port which is on anyway.

    The problem is that I am loosing data. I know this because part of the data I send is a counter and there are counts missing. Sometimes 3 or 4.

    Fortunately I have a second server (one that I don't normally leave on) which is identical in both hardware and software (suse 10.1). If I use that server it logs the data perfectly.

    The server in question has been running 24/7 for many years (since 10.1 was released) and has been "tweaked" over the years so I suspect one of my "tweaks" has caused this problem but I have no idea what or how to investigate this.

    Thank you for any ideas you may have as I really don't want another server running just to capture this data.

  2. #2
    Join Date
    Jun 2008
    Location
    Auckland, NZ
    Posts
    20,713
    Blog Entries
    1

    Default Re: Minicom Loosing Data

    Even though you report this works ok on another machine with identical hardware, I wonder if the signal is marginal somehow.... shielding>>noise.... What baud rate are you using? Have you tried reducing this (if practical)?

  3. #3
    Join Date
    Jun 2008
    Location
    Auckland, NZ
    Posts
    20,713
    Blog Entries
    1

    Default Re: Minicom Loosing Data

    Another possibility is that another process or app is reading the buffer occasionally.

    Serial HOWTO

    In particular:

    In order to use a device, you (or the program you run if you have "set user id") needs to have permission to read and write the device "file" in the /dev directory. So a logical way to prevent others from using a device is to make yourself the temporary owner of the device and set permissions so that no one else can use it. A program may do this for you. A similar method can be used with the group of the device file.

    While lock files prevent other process from using the device, changing device file owners/permissions restricts other users (or the group) from using it. One case is where the group is permitted to write to the port, but not to read from it. Writing to the port might just mean a message sent to a text-terminal while reading means destructive reading. The original process that needs to read the data may find data missing if another process has already read that data. Thus a read can do more harm that a write since a read causes loss of data while a write only adds extra data. That's a reason to allow writes but not reads. This is exactly the opposite of the case for ordinary files where you allow others to read the file but not write (modify) it. Use of a port normally requires both read and write permissions.

  4. #4
    Join Date
    Mar 2010
    Location
    Austin - Texas
    Posts
    10,140
    Blog Entries
    48

    Smile Re: Minicom Loosing Data

    You know sometimes Mr. Murphy just won't give it a rest!

    I have had many "challenges" with this latest project. The most recent one I posted on these forums about how I can create a daily log file and someone suggested logrotate. Excellent, and now I have that working.

    Now I am at the next and hopefully last problem.

    First a little history, I have built a microprocessor board which allows me to log my hydro usage. It involves a photo diode on my hydro meter which counts flashes.

    Every 10 minutes I send the kWh out the serial port. Being power conscious I want to connect this to my servers serial port which is on anyway.

    The problem is that I am loosing data. I know this because part of the data I send is a counter and there are counts missing. Sometimes 3 or 4.

    Fortunately I have a second server (one that I don't normally leave on) which is identical in both hardware and software (suse 10.1). If I use that server it logs the data perfectly.

    The server in question has been running 24/7 for many years (since 10.1 was released) and has been "tweaked" over the years so I suspect one of my "tweaks" has caused this problem but I have no idea what or how to investigate this.

    Thank you for any ideas you may have as I really don't want another server running just to capture this data.
    Hello sd_read. I read with interest your problem, but it is not clear to me who is losing this data. The board that records it or the computer you are sending it to? I happen to work in the HVAC industry and recording KW, KWH & GPM's for instance is very normal. The controllers that record this data are proprietary and utilize what would be equal to a Real Time kernel in Linux I suppose. While most PC's work in a batch mode of sorts, real time computers are not allowed to miss a thing. Its imperative to keep a computer under it maximum ability or it will simply miss events. While this kind of holds true for any PC in some ways, a real time system is different from anything you have worked with. In the case of recoding data, it needs to have very few unnecessary duties running. When sending data to some where else, the frequency the data is sent needs to be around 10 times more often that the ability of the recording computer to save recorded data. Transmissions are cumulative, but over a short time frame. I can miss several transmissions without missing usage today or over several days, depending on the recording computers ability to retain data. Now, after that mish mash of statements, my normal response would be to use what works and not try to get everything to work. Or, you need to restore what works on one PC to another if you really need two PC's to work identically. There are lots of controllers that could communicate to a PC running Linux. I work with a lot of LON (As in Echelon) based controllers and lots of controllers that use BACnet. Industrial controllers are not inexpensive, but a whole lot of them cost less than an iPhone or other such device if you are really into this. And in any event, good luck with your experiments.

    Thank You,
    My Blog: https://forums.opensuse.org/blogs/jdmcdaniel3/

    Software efficiency halves every 18 months, thus compensating for Moore's Law

    Its James again from Austin, Texas

  5. #5
    Join Date
    Jun 2008
    Location
    Auckland, NZ
    Posts
    20,713
    Blog Entries
    1

    Default Re: Minicom Loosing Data

    Fortunately I have a second server (one that I don't normally leave on) which is identical in both hardware and software (suse 10.1). If I use that server it logs the data perfectly.
    I think you need to compare the processes that are running between servers.
    Code:
    ps -e

  6. #6

    Default Re: Minicom Loosing Data

    Thank you for all the replies.

    I wonder if the signal is marginal somehow.... shielding>>noise.... What baud rate are you using?
    I am running 57600 baud and I have connected this board to many computers and looked at the output on an oscilloscope and observed healthy signal swings so I am fairly confident that it is not a hardware or signal issue.

    The board that records it or the computer you are sending it to?
    Excellent question and I believe I can say it is the computer because the other server never misses data. If you recall I send an incrementing count which is how I am able to determine I am missing data. On the good server I never miss a count.

    Another possibility is that another process or app is reading the buffer occasionally.
    On my server that looses data I write the file to a directory which I share over samba (which is another reason why I want to run this on my main server) with the following permissions:

    Code:
     0 -rw-r--r-- 1 root root     0 2011-09-03 00:00 hydro.csv
    While the "good" server is in the root directory as:

    Code:
    4 -rw-r--r-- 1 root root  476 2011-09-05 10:14 hydro.csv
    Thus a read can do more harm that a write since a read causes loss of data while a write only adds extra data.
    If I understand this I think I want to set the permissions to:

    Code:
    0 -rwx------ 1 root root     0 2011-09-03 00:00 hydro.csv
    I will try this when I get a chance as well as comparing the the processes running on them.


    Thank you - Steve

  7. #7

    Default Re: Minicom Loosing Data

    On the server that looses data I set the file permissions to write only by the owner:

    Code:
    4 --w------- 1 root root  124 2011-09-05 19:12 hydro.csv
    Also, I am writing to the exact same location as the "good" server which is:

    Code:
    /root/HydroMeter/hydro.csv
    Unfortunately I am still loosing data.

    I did a:

    Code:
    ps -e
    for both servers but am not sure how to interpret the results so I am posting them.

    First the bad server:

    Code:
    SambaServer:~ # ps -e
      PID TTY          TIME CMD
        1 ?        00:00:00 init
        2 ?        00:01:28 ksoftirqd/0
        3 ?        00:00:00 events/0
        4 ?        00:00:00 khelper
        5 ?        00:00:00 kthread
        7 ?        00:00:02 kblockd/0
        8 ?        00:00:00 kacpid
      120 ?        00:00:00 aio/0
      119 ?        00:01:32 kswapd0
      325 ?        00:00:00 cqueue/0
      326 ?        00:00:00 kseriod
      362 ?        00:00:00 kpsmoused
      759 ?        00:00:00 reiserfs/0
      840 ?        00:00:00 udevd
     1228 ?        00:00:00 khubd
     1327 ?        00:00:00 shpchpd_event
     1595 ?        00:00:00 scsi_eh_0
     1596 ?        00:06:18 usb-storage
     2059 ?        00:00:00 upsmon
     2060 ?        00:00:01 upsmon
     2141 ?        00:00:03 syslog-ng
     2145 ?        00:00:00 klogd
     2157 ?        00:00:00 acpid
     2173 ?        00:00:12 dbus-daemon
     2176 ?        00:00:00 resmgrd
     2291 ?        00:00:08 hald
     2501 ?        00:00:00 hald-addon-acpi
     2540 ?        00:00:00 mdnsd
     2688 ?        00:00:00 portmap
     2825 ?        00:00:00 auditd
     2830 ?        00:00:00 kauditd
     2833 ?        00:00:01 snmpd
     2936 ?        00:00:00 nfsd4
     2937 ?        00:00:00 nfsd
     2938 ?        00:00:00 nfsd
     2939 ?        00:00:00 nfsd
     2940 ?        00:00:00 nfsd
     2942 ?        00:00:00 lockd
     2943 ?        00:00:00 rpciod/0
     2951 ?        00:00:00 rpc.mountd
     2952 ?        00:00:00 ntpd
     2960 ?        00:00:04 nscd
     3027 ?        00:00:03 powersaved
     3067 ?        00:00:15 sshd
     3114 ?        00:00:01 cron
     3122 ?        00:00:01 smbd
     3123 ?        00:00:00 smbd
     3130 ?        00:00:00 kdm
     3325 tty1     00:00:00 mingetty
     3326 tty2     00:00:00 mingetty
     3327 tty3     00:00:00 mingetty
     3328 tty4     00:00:00 mingetty
     3329 tty5     00:00:00 mingetty
     3330 tty6     00:00:00 mingetty
     3401 tty7     00:04:46 X
     3402 ?        00:00:00 kdm
     3456 ?        00:00:00 startkde
     3494 ?        00:00:00 gpg-agent
     3495 ?        00:00:00 ssh-agent
     3499 ?        00:00:00 dbus-launch
     3500 ?        00:00:00 dbus-daemon
     3531 ?        00:00:00 kdeinit
     3534 ?        00:00:01 dcopserver
     3536 ?        00:00:00 klauncher
     3538 ?        00:00:06 kded
     3543 ?        00:00:00 kwrapper
     3545 ?        00:00:00 ksmserver
     3546 ?        00:00:19 kwin
     3548 ?        00:00:09 kdesktop
     3559 ?        00:00:00 kmix
     3560 ?        00:42:30 amarokapp
     3564 ?        00:00:01 knotify
     3567 ?        00:00:06 zen-updater
     3570 ?        00:00:00 kpowersave
     3584 ?        00:00:00 amarokapp
     3585 ?        00:00:00 amarokapp
     4594 ?        00:00:01 kio_uiserver
     9818 ?        00:00:07 konsole
     9897 pts/2    00:00:00 bash
    10020 ?        00:00:21 kicker
    24718 ?        00:00:03 pdflush
    26742 ?        00:00:02 nmbd
    31635 ?        00:00:00 master
    11774 ?        00:00:00 smbd
    24909 ?        00:00:00 screen
    24910 pts/5    00:00:00 bash
    24964 pts/5    00:00:00 minicom
    31113 ?        00:00:28 konqueror
     3417 ?        00:00:00 qmgr
     4180 ?        00:00:00 kio_file
     4576 ?        00:00:04 konqueror
     5006 ?        00:00:00 smbd
    24029 ?        00:00:02 pdflush
     2558 ?        00:02:56 zmd
    27384 pts/2    00:00:00 minicom
    28287 pts/3    00:00:00 bash
    28525 ?        00:00:00 pickup
    28530 ?        00:00:00 smbd
    28535 ?        00:00:00 smbd
    28536 ?        00:00:00 smbd
    28537 ?        00:00:00 smbd
    28538 ?        00:00:00 smbd
    28539 ?        00:00:00 smbd
    28540 ?        00:00:00 smbd
    28541 ?        00:00:00 smbd
    28542 ?        00:00:00 smbd
    28543 ?        00:00:00 smbd
    28611 ?        00:00:00 sshd
    28614 pts/1    00:00:00 bash
    28645 pts/1    00:00:00 ps
    SambaServer:~ #
    And the server which works correctly:
    Code:
    (none):~ # ps -e
      PID TTY          TIME CMD
        1 ?        00:00:00 init
        2 ?        00:00:00 ksoftirqd/0
        3 ?        00:00:00 events/0
        4 ?        00:00:00 khelper
        5 ?        00:00:00 kthread
        7 ?        00:00:00 kblockd/0
        8 ?        00:00:00 kacpid
      117 ?        00:00:00 pdflush
      118 ?        00:00:00 pdflush
      120 ?        00:00:00 aio/0
      119 ?        00:00:00 kswapd0
      325 ?        00:00:00 cqueue/0
      326 ?        00:00:00 kseriod
      362 ?        00:00:00 kpsmoused
      759 ?        00:00:00 reiserfs/0
      840 ?        00:00:00 udevd
     1224 ?        00:00:00 khubd
     1409 ?        00:00:00 shpchpd_event
     1634 ?        00:00:00 boot.localfs
     1853 ?        00:00:00 sulogin
     1856 ?        00:00:00 upsmon
     1859 ?        00:00:00 upsmon
     1925 ?        00:00:01 dbus-daemon
     1944 ?        00:00:00 acpid
     2003 ?        00:00:00 syslog-ng
     2006 ?        00:00:00 klogd
     2007 ?        00:00:00 resmgrd
     2056 ?        00:00:01 hald
     2176 ?        00:00:00 hald-addon-acpi
     2363 ?        00:00:00 mdnsd
     2377 ?        00:00:00 auditd
     2379 ?        00:00:00 kauditd
     2384 ?        00:00:00 portmap
     2456 ?        00:00:00 kdm
     2538 tty7     00:00:08 X
     2544 ?        00:00:00 kdm
     2579 ?        00:00:00 ntpd
     2581 ?        00:00:00 nscd
     2594 ?        00:00:00 cupsd
     2624 ?        00:00:00 cifsoplockd
     2625 ?        00:00:00 cifsdnotifyd
     2634 ?        00:00:00 cifsd
     2655 ?        00:00:00 powersaved
     2669 ?        00:00:00 smbd
     2671 ?        00:00:00 xinetd
     2696 ?        00:00:00 smbd
     2700 ?        00:00:01 sshd
     2739 ?        00:00:00 master
     2779 ?        00:00:00 cron
     2981 tty1     00:00:00 mingetty
     2982 tty2     00:00:00 mingetty
     2983 tty3     00:00:00 mingetty
     2984 tty4     00:00:00 mingetty
     2985 tty5     00:00:00 mingetty
     2986 tty6     00:00:00 mingetty
     3044 ?        00:00:00 kde
     3085 ?        00:00:00 gpg-agent
     3086 ?        00:00:00 ssh-agent
     3090 ?        00:00:00 dbus-daemon
     3091 ?        00:00:00 dbus-launch
     3123 ?        00:00:00 kdeinit
     3126 ?        00:00:00 dcopserver
     3128 ?        00:00:00 klauncher
     3130 ?        00:00:00 kded
     3135 ?        00:00:00 kwrapper
     3137 ?        00:00:00 ksmserver
     3138 ?        00:00:00 kwin
     3140 ?        00:00:00 kdesktop
     3142 ?        00:00:01 kicker
     3151 ?        00:00:00 kmix
     3153 ?        00:00:00 knotes
     3154 ?        00:00:04 konqueror
     3156 ?        00:00:00 knotify
     3159 ?        00:00:01 zen-updater
     3161 ?        00:00:00 kpowersave
     3169 ?        00:00:00 klipper
     3900 ?        00:00:00 qmgr
    14594 ?        00:00:00 konsole
    14595 pts/1    00:00:00 bash
    14641 ?        00:00:00 kio_file
    14644 ?        00:00:00 kio_uiserver
     2386 ?        00:00:18 zmd
    21990 ?        00:00:00 pickup
    22029 pts/1    00:00:00 minicom
    22106 ?        00:00:00 sshd
    22109 pts/2    00:00:00 bash
    22136 pts/2    00:00:00 ps
    (none):~ #
    I appreciate any help or suggestions- Steve

  8. #8
    Join Date
    Mar 2010
    Location
    Austin - Texas
    Posts
    10,140
    Blog Entries
    48

    Smile Re: Minicom Loosing Data

    Something that might be interesting is the kernel modules loaded. You can list manually of course (lsmod) but I have a script that can be helpful as well:

    S.K.I.M. - SuSE Kernel Installed Modules - A lsmod replacement- Creates Alphabetized Module Listing - Blogs - openSUSE Forums

    A comparison of modules loaded could be interesting between the two computers.

    Thank You,
    My Blog: https://forums.opensuse.org/blogs/jdmcdaniel3/

    Software efficiency halves every 18 months, thus compensating for Moore's Law

    Its James again from Austin, Texas

  9. #9
    Join Date
    Aug 2008
    Location
    Temporarily, planet Earth
    Posts
    104

    Default Re: Minicom Loosing Data

    I use minicom extensively and have never found issue with it. All I can do is suggest a few things to examine

    Go into the minicom settings and ensure the following are identical:
    • Number of bits, same parity settings, same number of stop bits
    • Is flow control identical on both?
    • What type of serial port? USB, integrated into motherboard?

  10. #10
    Join Date
    Jun 2008
    Location
    Auckland, NZ
    Posts
    20,713
    Blog Entries
    1

    Default Re: Minicom Loosing Data

    I note your 'problematic' server has two instances of minicom running

    Code:
    24964 pts/5    00:00:00 minicom
    31113 ?        00:00:28 konqueror
     3417 ?        00:00:00 qmgr
     4180 ?        00:00:00 kio_file
     4576 ?        00:00:04 konqueror
     5006 ?        00:00:00 smbd
    24029 ?        00:00:02 pdflush
     2558 ?        00:02:56 zmd
    27384 pts/2    00:00:00 minicom
    Might just be a red herring though....

Page 1 of 2 12 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •