Re: Reading a file as it fills up by another program
Re: Reading a file as it fills up by another program
- Subject: Re: Reading a file as it fills up by another program
- From: Harald E Brandt <email@hidden>
- Date: Sat, 13 Dec 2003 15:32:00 +0100
+--> Christopher Nebel wrote 03-12-13:
On Dec 12, 2003, at 6:04 AM, Harald E Brandt wrote:
It keeps the file open - it is a redirect (incl stderr) after a
pipe. I just talked to a Unix guru who says that this is a
relatively common problem caused by the fact that Unix usually
buffers everything before it is actually written to file. That
means I am stuck!
Well, stuck-ish. Your guru is at least partly correct -- the
answers you get depend on just how often you ask and how the output
from the original source is buffered. By default, stdout is block
buffered, where a "block" is usually a few kbytes, so if you're
watching stdout, you won't get new data that often. stderr is
normally line buffered, so you get new data every time there's a new
line. For example:
~/test:
#!/usr/bin/perl
for ( 1 .. 100 ) { print "$_\n"; sleep(1); }
do shell script "~/test > f 2>&1 &"
repeat
display dialog (do shell script "tail -n 1 f")
end repeat
You'll just get lots of blank dialogs for most of the script.
(Block buffering in action.) If you change the Perl script to say
'print STDERR "$_\n"', you'll get increasing numbers in the dialog.
I find it somewhat surprising that curl is writing its progress
information to anything other than stderr,
+-
Yes, that's a nice test. 'curl' DOES write to stderr, but since it is
redirected to stdout and piped throuh tr (or a similar thing) it is
"reverted" to stdout behavior. It is akin to doing the following
with your sample script:
for ( 1 .. 100 ) { print STDERR "$_\n"; sleep(1); }
do shell script "~/test 2>&1 | tr '\\n' '\\r' > f 2>&1 &"
In this case, the behaviour is that of stdout, despite we are writing
to stderr from perl. Maybe I should re-redirect it back to stderr and
then again redirect it to stdout?? That sounds absurd though! (In my
case, curl will output on both stderr and stdout, and I want both.)
(And can you really trust for sure that you really get all info using
stderr? I feel like going out on a limb now... but maybe I should
try to work only with stderr, i.e redirecting stdout to stderr? or...
what do you think?)
Deivy wanted to know more precisely what I am doing, but I don't want
to clutter this list with large complicated scripts, so I show the
following instead, which basically is the core of what I am doing:
curl --write-out '%{http_code}\\t' -o downloadedFile myURLs 2>&1 |
tr '\\n' '\\r' > progressfile 2> /dev/null &
----------
Now to a solution that I have found out myself:
The solution is a bit brute force - not very elegant, but it actually
works! (Maybe you will shudder when you see the code...., poor
Chris...:-))
The technique is to repeatedly in a loop open and close a file,
thereby forcing it to really write to the file. Like this:
set perlScript to "
$/ = qq(\\r);
open (fOut, qq(>" & tempfile4progress & ")); close fOut;
#just to zero the file in case it already exists
$pos = 0;
while ( <> ) {
open (fOut, qq(>>" & tempfile4progress & "));
seek fOut, $pos, 0;
tr/\\n/\\r/;
print (fOut $_);
$pos = tell;
close fOut;
}"
set curlString to curlString & " 2>&1 | perl -e " & quoted
form of perlScript & " > /dev/null 2> /dev/null &"
Then I will have to read the file with the same technique, but this
time directly from AppleScript:
--------
set fRef to open for access file "Scratch:tmp"
set LastPos to 0
repeat
try
set curlRecord to read fRef from LastPos until return
set LastPos to LastPos + (length of curlRecord)
on error errMsg number errNr
if errNr is -39 then --eof, but we are really waiting
for a tab as an eof I have at the end
set curlRecord to ""
close access fRef
set fRef to open for access file "Scratch:tmp"
else
close access fRef
error errMsg number errNr
end if
end try
--then do things with the curl record, such a looking at the first or
last characters etc - that is why I set the curlRecord to "" above so
as not to be fooled into thinking we got something new.
--------
Well, this amounts to an awful lot of 'open' and 'close' of files!
Shudder, shudder.... I simply could't make it work without closing
the file whenever eof was hit. (For test purposes, I actually use
GetMilliSec and assemble the curlRecords into a list, each stamped
with the time difference from start, so I can really check how it
proceeds.)
It would be slightly nicer to have a small delay here in the loop,
but I want this to be compatible from Jaguar and up, and I am not
aware of a friendly method of delaying things fractions of a second
on Jaguar (and 1 second is too much I think).
Yet another method I contemplated, is to pipe everything through
'cat-u' before writing to file. That would force it to be unbuffered,
but such things are considered dangerous - especially with disk
activity, and I don't even know if it would work.
I never thought of using the stderr descriptor only, as mentioned at
the beginning of this mail, and I am not really shure how I would go
about to really ensure it's going to work alright with streams that
actually originate from both stderr and stdout.
Most of all, of course, one would like to read the buffer instead,
not involving any disk activity at all! But is that really possible?
With the Best Regards,
--heb
_______________________________________________
applescript-users mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/applescript-users
Do not post admin requests to the list. They will be ignored.