bash: how to pipe stdout and stderr separately, line by line
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 11 of 11
  1. #1
    Join Date
    Jan 2005
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    bash: how to pipe stdout and stderr separately, line by line

    Hi,

    I need a way to pipe stdout and stderr of a program separately to scripts which read their input line by line. The scripts are basically "while read line;do logger {custom options} -- $line;done".

    With stdout or stdout and stderr combined this is simple. A simple pipe or "2>&1 |" for both will work.

    However if I want to have them separately it isn't as simple.

    One solution I found is process substitution "program > >(script1.sh) 2> >(script2.sh)", but with this script#.sh are fed all the output at once, not line by line as I need it.

    What I want would look something like "program | script1.sh 2| script2.sh", where "2|" is a pipe for stderr.

    Any ideas?

    PS: I would have posted this to a bash discussion mailing list, but didn't find one. There's only a bug and annouce list for bash on gnu.org. If you know a more appropriate place, please advise.

  2. #2
    aleph's Avatar
    aleph is offline Banned (for/from) behaving just like everybody else!
    Join Date
    Jul 2007
    Location
    Nanjing, China
    Posts
    1,332
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Perhaps the comp.unix.shell group is a good place to ask questions about bash.

    As for your question, sorry I can't figure it out...
    Code:
    from rlyeh import cthulhu
    cthulhu.fhtagn()

  3. #3
    aleph's Avatar
    aleph is offline Banned (for/from) behaving just like everybody else!
    Join Date
    Jul 2007
    Location
    Nanjing, China
    Posts
    1,332
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I tried a possible hack...

    First, create 2 named pipes:
    Code:
    $ mknod t1 p
    $ mknod t2 p
    Second, start the scripts "listening" to the pipes
    Code:
    $ ./script1.sh < t1 &
    $ ./script2.sh < t2 &
    Then, start the program like this:
    Code:
    $ program 1> t1 2> t2
    When done, remove the named pipes.

    Really ugly hack...
    Code:
    from rlyeh import cthulhu
    cthulhu.fhtagn()

  4. #4
    jose.carmona Guest
    nice hack!!

  5. #5
    Join Date
    Jan 2005
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks,

    The named pipe hack won't work for me as I want to use this in scripts and often. It could be done but that would be just too much overhead as every invocation would require it's own pipe (script.sh receives unique parameters).

    I'm trying to find a usenet server that would let me post messages. Any hints?

  6. #6
    aleph's Avatar
    aleph is offline Banned (for/from) behaving just like everybody else!
    Join Date
    Jul 2007
    Location
    Nanjing, China
    Posts
    1,332
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by kilgor
    I'm trying to find a usenet server that would let me post messages. Any hints?
    I'd suggest you set up a new free email account and use that account to post to the group via Google Groups: http://groups.google.com/group/comp.unix.shell/topics
    Code:
    from rlyeh import cthulhu
    cthulhu.fhtagn()

  7. #7
    Join Date
    Nov 2006
    Location
    Detroit
    Posts
    6,971
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    You may have an easier time doing this in Korn shell (ksh93) instead of bash. In ksh93 you can run your scripts as co-processes waiting for input from your program, one can wait for stdout and the other can wait for stderr. See the ksh93 man page for help.

  8. #8
    Join Date
    May 2007
    Location
    U.S.
    Posts
    4,851
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by kilgor
    I'm trying to find a usenet server that would let me post messages. Any hints?
    What's your ISP? These days some ISPs are dropping their news servers, but there's a chance your ISP is one of those that still provides an NNTP server that will allow you to hit the usenet.
    - Tom
    "What is freedom? To have the will to be responsible for one's self." - Stirner

  9. #9
    Join Date
    Sep 2006
    Posts
    52
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by kilgor
    Hi,

    I need a way to pipe stdout and stderr of a program separately to scripts which read their input line by line. The scripts are basically "while read line;do logger {custom options} -- $line;done".

    With stdout or stdout and stderr combined this is simple. A simple pipe or "2>&1 |" for both will work.

    However if I want to have them separately it isn't as simple.

    One solution I found is process substitution "program > >(script1.sh) 2> >(script2.sh)", but with this script#.sh are fed all the output at once, not line by line as I need it.

    What I want would look something like "program | script1.sh 2| script2.sh", where "2|" is a pipe for stderr.

    Any ideas?

    PS: I would have posted this to a bash discussion mailing list, but didn't find one. There's only a bug and annouce list for bash on gnu.org. If you know a more appropriate place, please advise.
    check the advance bash manual on i/o. you might get some inspiration.

  10. #10
    Join Date
    Jan 2005
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks for the replies.

    I got a few solutions from the news group.

    http://groups.google.com/group/comp....90230403b4a48a

    As a one time use or so they are OK, but for use in scripts as I wanted they are either not feasible like the named pipe solution or too complicated and not actually too reliable as redirecting via other fd's.

    So the best solution was to drop the idea. Pity though it's so tough to get STDERR piped to another programs STDIN separate from STDOUT properly.

    As for the news servers. It seems all the ISPs here have dropped their news servers too. Google to the rescue as usual.

    Thanks again.

  11. #11
    stevea Guest
    Actually Stephane CHAZELAS method (2nd post here)
    http://groups.google.com/group/comp....90230403b4a48a
    looks like a direct solution.

    Code:
    # cat s1
    while read;do
     echo S1: $REPLY >&2
    done
    # cat s2
    while read;do
     echo S2: $REPLY >&2
    done
    
    # # define a function to hide the fugly bits
    split() { { { $1 2>&3 | $2 ; } 3>&1; }; }
    
    # this cmd has stdout & stderr output
    # ls -l s1 s2 s3 s4 
    ls: cannot access s3: No such file or directory
    ls: cannot access s4: No such file or directory
    -rwxr-xr-x 1 root root 40 2008-10-25 16:57 s1
    -rwxr-xr-x 1 root root 40 2008-10-25 16:58 s2
    
    # # split the output to two scripts
    split "ls -l s1 s2 s3 s4" "./s1" | ./s2
    S2: ls: cannot access s3: No such file or directory
    S2: ls: cannot access s4: No such file or directory
    S1: -rwxr-xr-x 1 root root 40 2008-10-25 16:57 s1
    S1: -rwxr-xr-x 1 root root 40 2008-10-25 16:58 s2

Similar Threads

  1. Replies: 1
    Last Post: 9th March 2008, 10:18 PM
  2. Minimal BASH-like line...........
    By dewangpm in forum Using Fedora
    Replies: 6
    Last Post: 8th July 2005, 02:31 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •