cURL is a software package which consists of command line tool and a library for transferring data using URL syntax.
cURL supports various protocols like, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP.
Examples:
- curl http://www.centos.org
- curl -O URL1 -O URL2 // multiple file downloads
- curl -L http://www.google.com // follow HTTP location headers.
- curl -C – -O http://www.gnu.org/software/gettext/manual/gettext.html
// continue & resume download - curl -z 21-Dec-11 http://www.example.com/yy.html // download a file if modified after given date.
- curl -u username:password URL // pass http authentication in curl
- curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php // download files from ftp server
- curl dict://dict.org/d:bash // get definition of a word with DICT protocol
- curl -x proxysever.test.com:3128 http://google.co.in // use proxy for file download
- curl –mail-from blah@test.com –mail-rcpt foo@test.com smtp://mailserver.com // send mail with SMTP protocol
wget utility is the best option to download files from internet. wget can pretty much handle all complex download situations including large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc.
Examples:
- wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 // download file
- wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701 // download and store with different file name
- wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 // download in background
- wget –user-agent=”Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3″ URL-TO-DOWNLOAD // mask user agent and display it as browser
- wget –spider DOWNLOAD-URL // test if url to download exists.
- wget -i download-file-list.txt // download multiple files
- wget –mirror -p –convert-links -P ./LOCAL-DIR WEBSITE-URL // download a full website
- wget –reject=gif WEBSITE-TO-BE-DOWNLOADED // reject certain file types.
- wget -r -A.pdf http://url-to-webpage-with-pdfs/ //download only certain file types.
Two interesting blog posts that I have come across for cURL and wget. The definitions and examples in this post are taken from these.
cURL: http://www.thegeekstuff.com/2012/04/curl-examples/
wget: http://www.thegeekstuff.com/2009/09/the-ultimate-wget-download-guide-with-15-awesome-examples/
