background xargs/wget not adhering to -P and -n limits

Tag: background , applescript , wget , xargs Author: mutou1 Date: 2013-06-17

I'm having a problem with xargs and Wget when run as shell scripts in an Applescript app. I want Wget to run 4 parallel processes in the background. The problem: basically, when I try to run the process in the background with

cat urls.txt | xargs -P 4 -n 1 /usr/local/bin/wget -q -E -b 1> NUL 2> NUL

a Wget process is apparently started for each URL passed in from the .txt file. This is too burdensome on the user's memory. When I run it in the foreground, however, with something like:

cat urls.txt | xargs -P 4 -n 1 /usr/local/bin/wget -q -E

I seem to get the four parallel Wget processes I need. Does anybody know how to get this script to run in the background with only 4 processes? I'm a bit of a novice, and I'm afraid I can't figure out why backgrounding the process causes this change.

Other Answer1

You might run xargs on the background instead:

cat urls.txt | xargs -P4 -n1 wget -q &

Or if you want to return control to the AppleScript, disown the xargs process:

do shell script "cat urls.txt | xargs -P4 -n1 /usr/local/bin/wget -q & disown $!"

comments:

I have tried both without success. What am I missing? Basically, I want the Applescsript to continue so ASObjC Runner can update the user on the xargs/wget progress. Even after trying 'do shell script "cat urls.txt | xargs -P4 -n1 /usr/local/bin/wget -q -E & disown $!"', the Applescript waits until the process is completed before moving on. Any ideas? Thanks for the suggestions so far.

Other Answer2

As far as I can tell, I have solved the problem with

cat urls.txt| (xargs -P4 -n1 wget -q -E >/dev/null 2>&1) &

There may well be a better solution, though...