[dcc2] Multi headers + metadata

[Dave Johnson]

Dan Smith writes:
> It appears that we have come to a consensus regarding XML.  It would 
> require many authors to add additional dependances to their projects, and 
> may hinder adaptation.  MIME/HTTP style headers are simple to parse and can 
> mimic the same data structures that xml can (with clever work).  If they 
> are good enough for http and SIP, I am sure we will not have a problem with 
> them.  What do you think about this type of representation for our header 
> metadata?



> Speaking of metadata, what additional information should we 
> standardize?  Under the current transfer proposal, we need a unique id for 
> each file listed in a header, file name+path, and size.  Any additional 
> information such as md5 hash, description, your ideas, etc.. can be 
> optionally included.  Any ideas as to what other metadata would be useful 
> to client authors?

These come to mind:

content-type
executable bit
md5 hash
sha1 hash
description
note

It'd be nice if a client can add additional metadata that other
clients can ignore if they dont understand.

> What about the transport?  Right now the idea is to create one connection 
> and send the header information, then request the files one at a time over 
> this connection.  They will be sent in the order that they are 
> requested.  This seems clean and simple.  Are we in agreement on this or 
> are there other ideas out there?

Fine with me.

> Regarding compression:  IRC file transfers have traditionally required the 
> size of the file to be known before the transfer, and the entire contents 
> to be written to the connection.  This is why sending multiple files is 
> needed, to eliminate the need to create a compressed archive on disk or in 
> memory (to know the size) before sending a group of files, which is not 
> feasible for large files.  Adding a streaming compression would make the 
> size unknown and we would have to implement some sort of chunking 
> mechanism, at which point it would be easier to abandon IRC transfers and 
> use some other well know protocol (http for instance).  Is there really a 
> need for this?  Perhaps it would work for smaller files (such as our 
> headers) and we could make it an option?

Compression would be a transport level option (just like encryption),
so the size is of the uncompressed data is used.  Encrypting something
will change the size as well, but it doesn't matter as the counting of
bytes should be done after any unencrypt and uncompress.

I'd like to see optional compression too.  If it is specified as a
transport level option then it can be transparent to the app layer
(file or chat*) and can be stacked (compress+encrypt) 

data -> compress -> encrypt -> transmit
receive -> decrypt -> decompress -> data

* compression and chat wont mix well if the compression is a block
  compressor.  I know bzip2 is block, and I think zlib is too, but I'm
  not sure.  In any event, a stream or variable block size compressor
  would work for chat (or just about any other app) too.

-- 
Dave