The latest version of http-conduit-downloader is 1.1.5-0.
http-conduit-downloader
Version 1.0.15 revision 0 uploaded by VladimirShabanov.
Package meta
- Synopsis
- HTTP downloader tailored for web-crawler needs.
- Description
HTTP/HTTPS downloader built on top of
http-conduitand used in https://bazqux.com crawler.Handles all possible http-conduit exceptions and returns human readable error messages.
Handles some web server bugs (returning deflate data instead of gzip, invalid gzip encoding).
Uses OpenSSL instead of tls package since it doesn't handle all sites.
Ignores invalid SSL sertificates.
Receives data in 32k blocks internally to reduce memory fragmentation on many parallel downloads.
Download timeout.
Total download size limit.
Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.
Can be used with external DNS resolver (hsdns-cache for example).
- Author
- Vladimir Shabanov <vshabanoff@gmail.com>
- Bug reports
- n/a
- Category
- Web
- Copyright
- n/a
- Homepage
- https://github.com/bazqux/http-conduit-downloader
- Maintainer
- Vladimir Shabanov <vshabanoff@gmail.com>
- Package URL
- n/a
- Stability
- n/a