How to automate batch downloading of files

A video tour of how to automate batch downloading of multiple files from a web site. While Unix/Linux might be preferable for Perl scripts, Windows has a great option with URL2FILE. In a little over seven minutes, you will be exposed to several options, see different batch files, and then learn how to construct a batch file from scratch. What might normally take you an hour or more to download, you can use a batch file to get in a matter of minutes.

Category:

Science & Technology

Tags:

• tutorial
• batch download
• automatic download files

[GntlTch]

Although this video has useful XLS tips and links to ActivePerl and URL2FILE, this is NOT the way to do batch downloads from web sites. The Firefox (which he is using), extension Download Them All by Nils Mair et al (search addons.mozilla.com for downthemall), does everything he wanted and much, much more - without Perl, without batch files and without even leaving the browser. It not only scripts the DLs but is a fast full DL mngr with multiple DL streams and was first written in 2007.

[wdoerner]

@GntlTch good suggestion on "Download Them All". When I made the video, there didn't seem to be a lot of options for decent batch file downloads or webcrawling (or at least couldn't find much on forums or via Google searchers). Do you know of decent freeware for webcrawling instead of writing Perl scripts? I used Web Extractor to get data off webpages before (instead of download the entire HTML page) but it's not free. The open source/freeware I tried were pretty bad and not user friendly.

[GntlTch]

@wdoerner I don't do much crawling but the Firefox FoxySpider extension is a possibility (addons.mozilla.org/en-US/fire­fox/addon/foxyspider/). The author requires a $5 fee after a 30 day trial to remove size and link limitations. I support asking for donations but am very much opposed to required fees from authors taking advantage of open source resources to create and promote their products.

[GntlTch]

(Continuing - exceeded char limit):

cURL (curl.haxx.se), is a great freeware alternate for URL2FILE. It is available for over 50 OpSys - from AIX to x/OS. It is open source, has an active user and developer community,supports a huge list of protocols and offers a library of modules for inclusion in your own app.