c# - Streaming a list of objects as a single response, with progress reporting -

my application has "export" feature. in terms of functionality, works this:

when user presses "export" button (after configuring options etc.), application first runs relatively quick query determines ids of objects need exported. then, each object, executes calculation can relatively long time finish (up 1s per object). while happening, user watching progress bar -- easy render, since know expected number of objects, how many objects have been processed far.

i move functionality webservice, usual reasons. however, 1 additional wrinkle in process our users have lot of network latency. thus, can't afford make 1000 requests if have 1000 rows process.

what i'd return custom stream service. can write row count first 4 bytes of stream. client read these 4 bytes, initialize progress bar, , proceed read stream , deserialize them on fly, updating progress bar deserializes each one. meanwhile, server write objects stream become available.

to make matters more interesting, since i'm sending long list of objects, use protobuf-net reduce overhead. hence, have several questions:

• is planning possible ? make sense, or there better way ?
• how can return custom stream servicestack service ?
• when deserializing stream of objects on client side, how can sort of notification each object deserialized ? need update progress bar.

i found answer, kind of want, doesn't address questions: lazy, stream driven object serialization protobuf-net

edit: should have mentioned client desktop c# application, uses servicestack , protobuf.net .

i recommend paging result set on multiple requests (i.e. using skip/take) instead of trying return stream of results require custom response, custom serialization , custom clients consume streaming response. more stateless approach more suitable on http each query can cached independently, better support retrying i.e. if there error 1 of requests can retry last successful response (i.e. instead of having download entire request again) , better debuggability , introspection existing http tools.

custom streaming response

here's example shows how return observable streamwriter , custom observable client consume streamed response: https://gist.github.com/bamboo/5078236 uses custom json serialization ensure each element written before it's flushed stream client consuming stream can expect each read retrieve entire record. custom serialization more difficult if using binary serializer protocol buffers.

returning binary , stream responses in servicestack

the imageservice shows different ways of returning binary or stream responses in servicestack:

returning stream in httpresult

public object any(imageasstream request) {     using (var image = new bitmap(100, 100))     {         using (var g = graphics.fromimage(image))         {             g.clear(request.format.toimagecolor());         }         var ms = new memorystream();         image.save(ms, request.format.toimageformat());         return new httpresult(ms, request.format.toimagemimetype());      } } 

returning raw byte[]

public object any(imageasbytes request) {     using (var image = new bitmap(100, 100))     {         using (var g = graphics.fromimage(image))         {             g.clear(request.format.toimagecolor());         }         using (var m = new memorystream())         {             image.save(m, request.format.toimageformat());             var imagedata = m.toarray(); //buffers             return new httpresult(imagedata, request.format.toimagemimetype());         }     } } 

the examples above show how can add additional metadata http response wrapping stream , byte[] responses in httpresult, if prefer can return byte[], stream or istreamwriter responses directly.

writing directly response stream

public void any(imagewritetoresponse request) {     using (var image = new bitmap(100, 100))     {         using (var g = graphics.fromimage(image))         {             g.clear(request.format.toimagecolor());         }          base.response.contenttype = request.format.toimagemimetype();         image.save(base.response.outputstream, request.format.toimageformat());         base.response.close();     } } 

returning custom result

public object any(imageascustomresult request) {     var image = new bitmap(100, 100);     using (var g = graphics.fromimage(image))     {         g.clear(request.format.toimagecolor());         return new imageresult(image, request.format.toimageformat());      } } 

where can write response stream directly implementing istreamwriter.writeto():

//your own custom result, writes directly response stream public class imageresult : idisposable, istreamwriter, ihasoptions {     private readonly image image;     private readonly imageformat imgformat;      public imageresult(image image, imageformat imgformat = null)     {         this.image = image;         this.imgformat = imgformat ?? imageformat.png;         this.options = new dictionary<string, string> {             { httpheaders.contenttype, this.imgformat.toimagemimetype() }         };     }      public void writeto(stream responsestream)     {         using (var ms = new memorystream())         {             image.save(ms, imgformat);             ms.writeto(responsestream);         }      }      public void dispose()     {         this.image.dispose();     }      public idictionary<string, string> options { get; set; } }