The latest version of http-conduit-downloader is 1.1.5-0.
http-conduit-downloader
Version 1.0.4 revision 1 uploaded by Bodigrim.
Package meta
- Synopsis
- HTTP downloader tailored for web-crawler needs.
- Description
HTTP/HTTPS downloader built on top of
http-conduitand used in http://bazqux.com crawler.Handles all possible http-conduit exceptions and returns human readable error messages.
Handles some web server bugs (no persistent connections on HTTP/1.1, returning deflate data instead of gzip)
Ignores invalid SSL sertificates.
Receives data in 32k blocks internally to reduce memory fragmentation on many parallel downloads.
Download timeout.
Total download size limit.
Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.
Can be used with external DNS resolver (hsdns-cache for example).
- Author
- Vladimir Shabanov <vshabanoff@gmail.com>
- Bug reports
- n/a
- Category
- Web
- Copyright
- Vladimir Shabanov 2013
- Homepage
- https://github.com/bazqux/http-conduit-downloader
- Maintainer
- Vladimir Shabanov <vshabanoff@gmail.com>
- Package URL
- n/a
- Stability
- n/a