streaming
Version 0.1.4.2 revision 1 uploaded by HerbertValerioRiedel.
Package meta
- Synopsis
- an elementary streaming prelude and general stream type.
- Description
Streaming.Prelude
exports an elementary streaming prelude focused on a simple "source" or "producer" type, namelyStream (Of a) m r
.Stream (Of a) m r
is a sort of effectful version of([a],r)
in which successive elements arise from some sort of monadic action. Everything is the library is organized to make programming with this type as simple as possible by making it as close toPrelude
andData.List
. Thus for example the trivial programS.sum (S.take 3 (S.readLn :: Stream (Of Int) IO ()))
sums the first three valid integers from user input. Similarly,
S.stdoutLn (S.map reverse (S.take 3 S.stdinLn))
reverses the first three lines from stdin as they arise, and sends them to stdout. And so on, with filtering, mapping, breaking, chunking and so forth. We program with streams of
Int
s orString
s directly as if they constituted something like a list rather than "extracting a list from IO", which is the origin of typical Haskell memory catastrophes. Basically any case where you are tempted to usemapM
,replicateM
,traverse
orsequence
with Haskell lists, you would do better to use something likeStream (Of a) m r
. The type signatures are a little fancier, but the programs themselves are mostly the same or simpler. Thus, the little demo program from this SO questionmain = mapM newIORef [1..10^8::Int] >>= mapM readIORef >>= mapM_ print
quickly exhausts memory; this of course has nothing to do with
IORefs
and is cured byimport qualified Streaming.Prelude as S main = S.print (S.mapM readIORef (S.mapM newIORef (S.each [1..10^8::Int])))
which uses no more memory than
hello-world
, and is simpler anyway, since it doesn't involve "extracting a list from IO". Almost every use of listmapM
,replicateM
,traverse
andsequence
produces this problem on a smaller scale. People get used to it, as if it were characteristic of Haskell programs to use a lot of memory, when "extracting a list or sequence from IO" is just bad practice pure and simple. ListmapM
,replicateM
,traverse
andsequence
make sense under certain conditions. Similarly,unsafePerformIO
makes sense under certain conditions.The
Streaming
module exports the general type,Stream f m r
, which can be used to stream successive distinct steps characterized by any functorf
, though we are mostly interested in organizing computations of the formStream (Of a) m r
. The streaming-IO libraries have various devices for dealing with effectful variants of[a]
or([a],r)
. But it is only with the general typeStream f m r
, or some equivalent, that one can envisage (for example) the connected streaming of their sorts of stream - as one makes lists of lists in the HaskellPrelude
andData.List
. One needs some such type if we are to express properly streaming equivalents of e.g.group :: Ord a => [a] -> [[a]] chunksOf :: Int -> [a] -> [[a]] lines :: [Char] -> [[Char]] -- but similarly with bytestring, etc.
to mention a few obviously desirable operations. (This is explained more elaborately in the readme below.) One could throw something like
Stream
on top of a prior stream concept: this is howpipes
andpipes-group
(which are very much our model here) useFreeT
. But once one grasps the iterable stream concept needed to express those functions - here given a somewhat optimized implementation asStream f m r
(following, as usual, models derived from thepipes
library) - then one will also see that, with it, one is already in possession of a complete elementary streaming library - since one possessesStream ((,) a) m r
or equivalentlyStream (Of a) m r
. This is the type of a 'generator' or 'producer' or whatever you call an effectful stream of items. The presentStreaming.Prelude
is thus the simplest streaming library that can replicate anything like the API of thePrelude
andData.List
.The emphasis of the library is on interoperation; for the rest its advantages are: extreme simplicity, re-use of intuitions the user has gathered from mastery of
Prelude
andData.List
, and a total and systematic rejection of type synonyms. The two conceptual pre-requisites are some comprehension of monad transformers and some familiarity with 'rank 2 types'. It is hoped that experimentation with this simple material, starting with the ghci examples inStreaming.Prelude
, will give people who are new to these concepts some intuition about their importance. The most fundamental purpose of the library is to express elementary streaming ideas without reliance on a complex framework, but in a way that integrates transparently with the rest of Haskell, using ideas - e.g. rank 2 types, which are here implicit or explicit in most mapping - that the user can carry elsewhere, rather than binding her intelligence to a so-called streaming IO framework (as necessary as that is for certain purposes.)See the readme below for further explanation, including the examples linked there. Elementary usage can be divined from the ghci examples in
Streaming.Prelude
and perhaps from this rough beginning of a tutorial. Note also the streaming bytestring and streaming utils packages. Questions about usage can be put raised on StackOverflow with the tag[haskell-streaming]
, or as an issue on Github, or on the pipes list (the package understands itself as part of the pipes 'ecosystem'.)The simplest form of interoperation with pipes is accomplished with this isomorphism:
Pipes.unfoldr Streaming.next :: Stream (Of a) m r -> Producer a m r Streaming.unfoldr Pipes.next :: Producer a m r -> Stream (Of a) m r
Interoperation with io-streams is thus:
Streaming.reread IOStreams.read :: InputStream a -> Stream (Of a) IO () IOStreams.unfoldM Streaming.uncons :: Stream (Of a) IO () -> IO (InputStream a)
With conduit one might use, e.g.:
Conduit.unfoldM Streaming.uncons :: Stream (Of a) m () -> Source m a Streaming.mapM_ Conduit.yield . hoist lift :: Stream (Of o) m r -> ConduitM i o m r ($$ Conduit.mapM_ Streaming.yield) . hoist lift :: Source m a -> Stream (Of a) m ()
These conversions should never be more expensive than a single
>->
or=$=
. The simplest interoperation with regular Haskell lists is provided by, sayStreaming.each :: [a] -> Stream (Of a) m () Streaming.toList_ :: Stream (Of a) m r -> m [a]
The latter of course accumulates the whole list in memory, and is mostly what we are trying to avoid. Every use of
Prelude.mapM f
should be reconceived as using the compositionStreaming.toList_ . Streaming.mapM f . Streaming.each
with a view to considering whether the accumulation required byStreaming.toList_
is really necessary.Here are the results of some microbenchmarks based on the benchmarks included in the machines package:
Because these are microbenchmarks for individual functions, they represent a sort of "worst case"; many other factors can influence the speed of a complex program.
- Author
- michaelt
- Bug reports
- https://github.com/michaelt/streaming/issues
- Category
- Data, Pipes, Streaming
- Copyright
- n/a
- Homepage
- https://github.com/michaelt/streaming
- Maintainer
- what_is_it_to_do_anything@yahoo.com
- Package URL
- n/a
- Stability
- Experimental