[dcc2] list/get

[Szymon Stefanek]

On Thursday 12 February 2004 17:37, Dan Smith wrote:

> 1.  Where should we implement LIST/GET, as a CTCP or in DCC2?  In CTCP,
> after it has been established which file is to be transferred, DCC2
> negotiation could take place as to how it will be downloaded?

I think that LIST through a CTCP is not feasible. The problem is that a LIST request will probably
generate multiple responses from each client: this would mean multiple CTCP messages.
This is not feasible on IRC where a limit is imposed on the outgoing messages. The servers
have a builtin antiflood mechanism which allows only few messages per second to be sent.

(Side note: for the same reason we should keep in mind that the DCC2 negotiations should
exchange the minimum possible number of messages)

So the answer is DCC2 for sure (just like the actual XDCC LIST is).

> 2.  Will there be a standard output of LIST that is machine readable?  And
> if so, will it specify some kind of file handle that can be used with the
> corresponding GET request?

Well, the idea can be summarized as follows:

- Each irc client keeps a list of shared files (just like napster or any other file sharing protocol).
	These files are either managed by the user (he can add/remove them manually),
	or by the client itself (just like in the case of the AVATAR protocol in KVIrc).

- Each shared file has a file handle which is a sort of unique identifier.
	Having a file handle instead of the real file name is a good idea: it protects the
	sending user privacy and avoids a lot of common mistakes that can lead to
	huge security holes.
	The file handle will usually be derived from the real file name (and maybe the file path) but it is
	not mandatory. Many existing file server implementations use just numbers as file handles.
	There should be a well defined set of characters that can appear in each file handle:
	for example, the '\' and '/' should be forbidden (this should be valid also for the file names
	offered through a simple DCC2 SEND).
	
- The sharing client obviously knows a set of parameters for each shared file, for example the size
	and maybe an md5 hash.

- The DCC2 request would just report this data to the requesting client in a standardized format through
	a direct connection. The format could be something like:

		Comment: Files shared by client X<crlf>
		<crlf>
		Handle: <filehandle><crlf>
		Name: <real file name><crlf>
		Size:<file size><crlf>
		Md5Sum: <md5 sum><crlf>
		AnyFieldYouWant: <something><crlf>
		<crlf>
		Handle: <filehandle><crlf>
		...
		...

	This is similar to transferring a text file through DCC SEND. The difference is that the format is standardized
	and the clients (and not only users) can decode and show it (maybe in a clickable list view or something similar).
	We could also think of a hierarchy in the files, maybe something like the virtual file system offered by the http servers,
	but this is a secondary argument.

- The requesting client would be able to request the files by issuing a DCC2 GET File=filehandle.

Here comes out the importance of an authentication method too. The clients handling LIST and GET commands
should have a way to recognize the requesting users. Some shared files could be accessible by only by providing
a correct username and/or password and the LIST could be generated accordingly too.
The authentication would be obviously optional: open file servers are also regular citzens.

> 3.  For clients that allow multiple file get/list via MULTI, how should a
> request be initiated?  A type=ircfilelist or something along those lines
> seems to fit the pattern we have been talking about.  This extra type would
> be necessary since the receiving client is requesting that offering client
> send a shared file list, which is opposite of the type=ircfile request.

I must admit that I didn't completly understand the meaning of the MULTI transfer.
Please point out errors and misunderstanding in the following monologue.

If I understand well the MULTI transfer would be used to transfer multiple files
without repeating the connection negotiation. This is generally a good idea.
The draft is not so clear in defining if the set of files is just fixed (choosen by the sender)
or the receiver can choose from the XML list.
If the standard is meant to transfer a fixed set of files then I see the following two problems:

- The connection negotiation doesn't show the file names that are going to be transferred.
	The receiving user has no way to know what he is going to receive before
	accepting the connection. This is not good since the DCC protocol has been massively used for exploits.
	The receiving client would then need to ask the user twice and it it doesn't look that good
	to me (:D)

- The second problem is even bigger from my point of view. The MULTI transfer uses xml and thus it needs an xml parser.
	This is a huge effort for the clients that use libraries without xml parsing facilities: not many of them will implement it.
	Even when using an off-the-shelf xml parsing component the interpretation is still complicated.
	I think that xml is good for really structured data: a file list is better transfered (and parsed) as plain text.

If the standard means that the user can choose the files to be transferred selecting from the list then
there are a couple of additional glitches.
While the user is selecting the files the connection between the clients is still open.
- The first problem is that the users are usually slow in selecting (especially if the list is long)
	and keeping the connection open and idle is not that nice.
- The second problem is that most of the times the user will not choose anything!
	Think of any file sharing protocol: you usually need to see a lot of results
	before finding exactly what you're looking for, even if the results are responses
	to a well defined query. In this case there is no query at all: the user sees
	a list of "random" shared files to select from.

I don't want to tell that MULTI should be thrown away. It is still a good idea to transfer multiple files without repeating
the connection negotiation. The real question is: when it is needed ?
I think that there are two major cases:

- The sender wants to transfer a defined set of files. The set will be either a directory
	or a set of files choosen on "random" basis from the sender permanent storage.
	In the first case the directory structure must be preserved. In the second case
	the files can be thought as a single flat directory with all the files.
	The solution is simple: pack, eventually compress, send, eventually uncompress and unpack.
	This could be achieved by using one of the free file archiving methods: for example tar.gz or tar.bz2
	So finally this kind of transfer is better implemented with something like ZIPSEND.
		
- The second case is when the receiver wants to download more than one file from the same sender
	and the files have been choosen earlier.
	The receiver will usually choose the files from a list received earlier.
	This could be my interpretation of MULTI: a MULTI GET.
	- The file receiver already knows the filest that he wishes to transfer (maybe selected from a list
		obtained from a DCC2 LIST request).
	- He requests a MULTI GET, negotiates the parameters and connects to the file sender.
	- For each file he wants to download he issues a:
		GET <filehandle> [fileoffset]
	- After all the files have been transferred the receiver issues a COMPLETE command and the connection terminates.


-- 

Szymon Stefanek

------------------------------------------------------------------------------
-
- Opportunity favors the prepared mind.
-
------------------------------------------------------------------------------