dcdev mailing list

Hi,

Alot of discussion has been focused on search commands and it's features.
I don't think we need (nor should) do it too complex and with many
features.

perhaps it is because it is the main feature (in both user usage and bandwidth load).

"SEARCH <dest[:port]> <id> <mime> <size|#[type#]hash> :<pattern>\r\n"

The first 3 arguments are given, no discussion there.
(destination must be a ip with port for active search and for passive it
is the client's GUID or name... whatever will be used).
The fourth argument can be for two uses. either a size (range) or a
hash. If it's a hash it must begin with a # char.
Next comes the pattern (wich can contain spaces, therefor the : to tell
the beginning of the message).
Pattern can also have two meanings, either it can be a regex (poisix or
perl, doesnt matter) or it is the exact (or wildcarded) name of the
file. Clients can decide, upon submitting, if they want to replace
spaces with a * to work the way searching currently does.
To tell a regex from name the work is simple - as no filename on a
filesystem can contain directory delimiters and therefor noone can
search for one, we simply add them in the beginning and end of the
regex. And woila, we still have a valid regex: "/pattern goes here/" :)
(some regex libs cant handle the / /, but it's easy to filter out by the
end-client).

I think (but it is only my opinion) there is too many problems with this description. 1) the mime is useless because most of the time, clients do not share a common database with known values and I don't speak about OS (windoz knows mime types ?). Moreover, below, you speak about "simple" users that are just able to do basic queries ... do you think they now mime types ? and what is the mime type of the current "any" type (when a user is search for an album (mp3 and covers)).
2) the size/hash part is embedable as pattern (except if you mean 'file pattern' and not just 'pattern').

No need for fancy if-statements and such in the search query. A user
mostly just searches for a name anyway and goes through the results
manually, it's the quickest and most userfriendly. Ok - it's simple to
write a if-statement yourself.. but imagine a gui to that.. to be able
to make a complex if statement without actually write the if statement
it would require alot of controls. As per my example you only need a
single text field. Pherhaps you like to some checkboxes to tell if you
want it as a regex, wildcard of plain to ensure syntax before sending
the request to save traffic, but this is optional.

I hope I won't seem to be to (too) rude but it is too simple for me and I like complex things :) The protocol should not limit software evolution. What you propose looks a lot like the current search query with very few modifications and if it is true a lot of users only use a small set of features (who says mickey$oft office :) ), this does not mean some other do not need more.

The previous suggestion "( N =~ avi$ || N =~ ogm$ ) && ( T =~ ^video/ ) && ( S

450000000 )" provides simultaneously an enhanced query type and a query

type that can evolve without breaking "syntactic compatibility" with previous client (supporting the same query scheme). You even have encountered a limit of your own suggestion with your "Perhaps ..." because if you start to add new features by breaking the previous protocol definition 2 minutes after its creation, I don't want to imagine the result after 6 months :)

About the GUI having problem to create big patterns, I think it is a fake problem. IMHO, a GUI should not create big patterns, it should provide a simple search support (a bit like the current one). This search form will be used by most of the users (including expert ones when they don't want to do weird things). On the other side, the GUI should provide an input field allowing expert users to create their own query (manually or with the help of the GUI). Your way of describing the search problem looks like mickey$oft way of programming: I will program with my mouse :) Honestly, I have never seen a program working with such a method.

I know we are to discuss the actual protocol - but what's the point of
making a protocol that is not usable?

At least, we agree on something :)

Also, you quickly want to find something.. dont want to sit 2minutes
configuring a criteria when it only takes 30seconds to scroll through
100+ results.

Not really a good example because IMHO, a client should provide a default configuration allowing a basic user to perform search like you describe it (see above).

Eric
DCTC/dchub

Hello!
Alot of discussion has been focused on search commands and it's features.
I don't think we need (nor should) do it too complex and with many features.

The example I'm going to show is simple, and the messages are not that long, and with few arguments. And if you'd like to get an easy overview of it in a telnet client, it's human readable.

Note that I use irc-style on messages, for faster parsing; only one space containing string allowed in each message. Easily found after the first " :" string and continues to end of message (CRLF).

Here follows the syntax

"SEARCH <dest[:port]> <id> <mime> <size|#[type#]hash> :<pattern>\r\n"

The first 3 arguments are given, no discussion there.
(destination must be a ip with port for active search and for passive it is the client's GUID or name... whatever will be used).
The fourth argument can be for two uses. either a size (range) or a hash. If it's a hash it must begin with a # char.
Next comes the pattern (wich can contain spaces, therefor the : to tell the beginning of the message).
Pattern can also have two meanings, either it can be a regex (poisix or perl, doesnt matter) or it is the exact (or wildcarded) name of the file. Clients can decide, upon submitting, if they want to replace spaces with a * to work the way searching currently does.
To tell a regex from name the work is simple - as no filename on a filesystem can contain directory delimiters and therefor noone can search for one, we simply add them in the beginning and end of the regex. And woila, we still have a valid regex: "/pattern goes here/" :) (some regex libs cant handle the / /, but it's easy to filter out by the end-client).

No need for fancy if-statements and such in the search query. A user mostly just searches for a name anyway and goes through the results manually, it's the quickest and most userfriendly. Ok - it's simple to write a if-statement yourself.. but imagine a gui to that.. to be able to make a complex if statement without actually write the if statement it would require alot of controls. As per my example you only need a single text field. Pherhaps you like to some checkboxes to tell if you want it as a regex, wildcard of plain to ensure syntax before sending the request to save traffic, but this is optional.
I know we are to discuss the actual protocol - but what's the point of making a protocol that is not usable?

Also, you quickly want to find something.. dont want to sit 2minutes configuring a criteria when it only takes 30seconds to scroll through 100+ results.

with if-statement i mean something like this
"( N =~ avi$ || N =~ ogm$ ) && ( T =~ ^video/ ) && ( S > 450000000 )"

another thing, using a particular type of hashing. simply set the hash/size argument to "#MD5#615b3de4b7a572679b457de271305229" for using MD5 the algorithm (of course, this requires a shared database of names of hashing types, but they shouldn't be too many to choose from).

some examples...

A file named "foo.mpg" of type video/* and greater than 4MB in size.
"SEARCH 217.210.105.76:412 01 video/* >4096 :foo.mpg\r\n"

A text file named "*bar*" with maximum or equal 3000 bytes in size
"SEARCH 217.210.105.76:412 02 text/* <=300 :*bar*\r\n"

a part of a WinRAR packed file name "foo" with exact size 15000000 bytes
"SEARCH 217.210.105.76:412 03 text/* =15000000 :/^foo\.part[0-9]{2,3}\.rar$/\r\n"

a textfile named "why_dc.txt" with hash "185410fd"
"SEARCH 217.210.105.76:412 04 text/plain #185410fd :why_dc.txt\r\n"

/Carl-Adam Brengesjö

Ps. New to this list. hello everyone :)
Working with Fabrice on ODCH - if you wondered.

Carl-Adam Brengesjö writes:
> Hello!
> Alot of discussion has been focused on search commands and it's
> features.  I don't think we need (nor should) do it too complex and
> with many features.
> [...]
> Here follows the syntax
> > "SEARCH <dest[:port]> <id> <mime> <size|#[type#]hash> :<pattern>\r\n"
> [...]

In my opinion, there are two great disadvantages with doing this:

1. As eric stated in his mail, this approach is not easily
  extensible. Say that you want to add an attribute later that could
  be searched for (not that I can think of one right now - if I
  could, I would have liked to add it, of course). That could not
  easily be done with an approach such as this, but very easily with
  the approach that eric and I have almost agreed upon.

2. There is a very important reason to be able to specify very strict
  search results. Considering the fact that search results are
  delivered by UDP, which doesn't have arrival acknowledgments,
  relevant search results will be dropped if you search for something
  popular without being able to make it more strict.
  I'm currently working on a protocol draft which will incorporate a
  search pattern syntax that differs slightly from the one I
  presented earlier and an example algoritm of how to parse and use
  it in a very efficient way. I'm having a test tomorrow, though, but
  expect me to publish it this weekend, and you may have a look
  yourself.

> No need for fancy if-statements and such in the search query. A
> user mostly just searches for a name anyway and goes through the
> results manually, it's the quickest and most userfriendly. Ok -
> it's simple to write a if-statement yourself.. but imagine a gui to
> that.. to be able to make a complex if statement without actually
> write the if statement it would require alot of controls. As per my
> example you only need a single text field. Pherhaps you like to
> some checkboxes to tell if you want it as a regex, wildcard of
> plain to ensure syntax before sending the request to save traffic,
> but this is optional.

That's not necessarily a problem. Since, as you said yourself, the
average user won't often (or at all) use this functionality very
often, the default UI can very well display just a standard text box
and insert the pattern that is entered into a more complex search
expression. Thus, those who actually want to use the more complex
options can simply click a button to have whatever they type be sent
verbatim.

Just because _most_ users don't use it is no reason in my mind to ruin
the fun for those who would use it. It's an extremely useful facility,
since it allows (especially for automated clients) the bandwidth usage
to drop significantly. I'm sure hub owners would appreciate very much
if passive clients use it.

> I know we are to discuss the actual protocol - but what's the point of > making a protocol that is not usable?

How exactly do you mean that it makes the protocol itself unusable?

> Also, you quickly want to find something.. dont want to sit 2minutes > configuring a criteria when it only takes 30seconds to scroll through > 100+ results.

Again (just to reiterate), the default UI would display just a single
text box that really just inserts whatever you type in it directly
into (printf format) "( N ~= %s )".

> another thing, using a particular type of hashing. simply set the > hash/size argument to "#MD5#615b3de4b7a572679b457de271305229" for using > MD5 the algorithm (of course, this requires a shared database of names > of hashing types, but they shouldn't be too many to choose from).

That is clearly not a good idea. Hashing takes long enough as it is
(hashing my share of 160 GB takes several hours), and thus you clearly
don't want to hash it several times using different algorithms. It
will probably keep chewing for a day while bringing your load to 10
and ruining your buffer cache forever. Especially, think of the poor
guys who share like 600 GB or more - they'll probably have to be
munching numbers for a week, so I don't think that they'll appreciate
this, which is bad since everyone on the network appreciate them very
much.

Instead, we absolutely need to standardize a specific hashing
algorithm. I suggest SHA1, but it doesn't really matter. It would be
good if everyone could agree soon, though.

Also, the question remains how to deal with MIME types. Finding them
probably isn't that big a problem. On UNIX, just call "file -i" on the
file, and on Windows (although I'm not a Windows programmer, so I
don't know this for sure), I believe the MIME type is stored in the
registry for each file extension, right.
However, the problem remains that the detected MIME types differ from
system to system and computer to computer, essentially. Of course,
there are the standardized ones, but the */x-* types aren't really
very reliable, to say the least. For example, if you want to search
for AVI files alone (if you don't want any MPEG ones because of the
bad quality), how would you do then? I have seen at least three
different reported mime types for those.

Also, MIME types don't always properly convey the file type. For
example, when it comes to AVI files, you aren't actually interested in
knowing that it's AVI - most often you want to know the format of the
video stream. I guess this isn't really an issue that a DC
implementation even should be required to handle. However, it could be
a good thing to be able to recover meta-data from files (such as AVI
or MP3), to, for example, filter out those pesky dubbed movies or
search for an artist using the ID3 tag. On the other hand, it would
make DC clients much more complex, which isn't necessarily a good
thing.

I don't want you to think badly of me for disapproving your
suggestion, but as eric said, the search function is the single most
important feature of Direct Connect, and thus needs to be done very
carefully.

This is my opinion. However, discussion is always good, so please try
to convince me.

Fredrik Tolf

As for hashing, yes, load is a problem if you use the current way of doing it. The hashing part in my command is clearly optional, and you often doesnt search for hashes if you havn't already downloaded a "MD5" or "*.sfv" file (or simlar) of the files you are going to download. But say that a user does this. And another client receives this query, when that client understands that we are looking for a specific hash, that client could look into a indexe'd database for a quick search that would result in fileinfo; filename & size. Ofcourse, this requires a client implementation. So if the client doesnt like to compare hashes.. just dont do it! instead dump the query and dont process it. Yes this results  in fewer results to the client who made the search, but it would /encourage/ people doing more advanced clients, no?

And for mime types - yes, that is problems with that aswell - but as for the other search command, how do you decide what type the file is anyways? File extension? If so, why not make a basic sets of mimetypes for different mimetypes that is _recommended_ to use in clients? Or a command that would return (from hub to client) a basic set of mimetypes for some filextensions so that all clients in the same hub uses same (this is just a sloppy idea, dont take it seriously). As for now I say we must decide a way wether to use mimetypes or not. We still have to use some sort of type definitions of files, so what will we use? mimetypes are a global (partially) way to do it, and will be used more in the future, so why not use it? What we DONT have to decide at this stage is HOW to use it. (take a look on my other mail, "Developing")

How would the simpliest form of query (just size and filename) look like with the other? (i havn't quite understood it yet) maybey have a simple and advanced search command, if that would result in shorter messages and easier/quicker to parse. (just speculating)

About filename matching in the other search command, how do you know where the string starts (I understand that $ ends)
( N =~ avi$ ) would match (not) "avi" or (not) " avi".
isn't it just better to have it surrounded by "" (use $ if you like, " is just a more standard way, and any char is as good as the other)

/Carl-Adam

ps. i originally wrote "developing" and this reply as one mail, but they kinda got off-subject from eachother, so I splitted them.

Fredrik Tolf wrote:

Instead, we absolutely need to standardize a specific hashing
algorithm. I suggest SHA1, but it doesn't really matter. It would be
good if everyone could agree soon, though.

[2004-01-15 14:44] <sandos> someone care to mention to the mlist that TTHs are already semi-standard (well, in my small DC world atleast)

Yes, Tiger Tree Hashes have been part of BCDC for several months now.

Part of the description is here:

http://members.verizon.net/todd.pederzani/dc++/draft-jchapweske-thex-02.html

Richard has already implemented them in DC:Pro, and Tim has talked about implementing them in Intradream DC  (IDC) to make  his multi-source implementation corruption free.

- Todd

Carl-Adam Brengesjö wrote:

Pattern can also have two meanings, either it can be a regex (poisix or perl, doesnt matter) or it is the exact (or wildcarded) name of the file. Clients can decide, upon submitting, if they want to replace spaces with a * to work the way searching currently does.

[2004-01-15 14:51] <nysin> (Might someone suggest on dcdev-list that these people try some sample queries in their regex syntax on a largeish share?

If any of you need it, I have several large filelists (in bz2 format) that I passed along to cologic (nysin).  One in particular is 493gb and 172,000 files.  I think modeling a typical search load on this share might quickly tell if regexp searching is feasible.

A previous discussion on regexps also yielded concerns about... ahem... resource intensive expressions.  Nothing quite like being able to DOS all the clients in a hub simply by sending a well crafted regexp search.

a part of a WinRAR packed file name "foo" with exact size 15000000 bytes
"SEARCH 217.210.105.76:412 03 text/* =15000000 :/^foo\.part[0-9]{2,3}\.rar$/\r\n"

Ah yes.  A wonderful example of why searching by MIME types is not so useful.  What exactly is the proper type for RAR files?
application/rar ?
application/x-rar-compressed ?
How about .MKV?  OGM?  OGG?

- Todd

Todd Pederzani wrote:

Fredrik Tolf wrote:

Instead, we absolutely need to standardize a specific hashing
algorithm. I suggest SHA1, but it doesn't really matter. It would be
good if everyone could agree soon, though.

[2004-01-15 14:44] <sandos> someone care to mention to the mlist that TTHs are already semi-standard (well, in my small DC world atleast)

Yes, Tiger Tree Hashes have been part of BCDC for several months now.

why lock it to a single hash algorithm? you can't know the hash of the file you're looking to unless you have downloaded a *.sfv (what algorithm? is it called sfv, or?) or MD5 (or simlar) file telling it. Or if someone told you. and it's not for certain that all the hashing are the same type. I think it's better to have some keycodes for wellknown hashtypes so that it can be given by the client searching and extracted by the client matching.
If one can't get the hash of a file using a specific algorithm, he can't use the feature. Instead try to search on the hash he has.

But of course we can encourage use of a specific hashing algorithm, but I dont think we should explicitly use one.

--On fredag den 16 januari 2004 02:22 +0100 Carl-Adam Brengesjö <[email protected]> wrote:

why lock it to a single hash algorithm? you can't know the hash of the
file you're looking to unless you have downloaded a *.sfv (what
algorithm? is it called sfv, or?) or MD5 (or simlar) file telling it.

The way it is implemented right now is, the client calculates hashes for the files in your share, and supplies them when sending out search hits.

But of course we can encourage use of a specific hashing algorithm, but I
dont think we should explicitly use one.

I'm pretty sure a lot of the developers (and especially arne) want to standardize on one specific hashing algorithm, for simplicity's sake.

TTH is what we're using right now, and unless you can supply a good reason why it's unfeasable, I doubt people will change.

I agree that having a choice is better than not having it in most cases, but this is not one of those cases. :)

/fnord, who doesn't even code ;)

Carl-Adam Brengesjö wrote:

And all other suggestions for deciding filetype have been so far to have it being based on filextensions, wich is not the same!

The best suggestion so far is to leave the current system of file types (where a type is some set of file extensions) in place.

Again: what does using MIME types gain us over file extensions?

Sure,  we could basically recreate/port the unix way of guessing mime type, but... why?  It gains us nothing, and it adds complexity and uncertainty.  Not all will handle MIME guessing the same, so you've just introduced interoperability issues.

- Todd

Mattias Bergsten wrote:

--On fredag den 16 januari 2004 02:22 +0100 Carl-Adam Brengesjö <[email protected]> wrote:

why lock it to a single hash algorithm? you can't know the hash of the
file you're looking to unless you have downloaded a *.sfv (what
algorithm? is it called sfv, or?) or MD5 (or simlar) file telling it.

The way it is implemented right now is, the client calculates hashes for the files in your share, and supplies them when sending out search hits.

Yes in search results it's good with only one hash algorithm. I mean when _searching_ for a file with a hash.

Say when having a file in your queue (together with a MD5 file in same directory) and want to search for alternative sources than the one you have. You open the MD5 file and looks for the file you want... and copies the md5 sum, say `3efdb9c9e399679cccd23179c4f1a1f2', then you searches for files with that hash.

> I agree that having a choice is better than not having it in most
> cases, but this is not one of those cases. :)

In that case (my example) you want choices :)

Sorry I mixed up search results with queries.

/Carl-Adam

Todd Pederzani wrote:

Carl-Adam Brengesjö wrote:

Pattern can also have two meanings, either it can be a regex (poisix or perl, doesnt matter) or it is the exact (or wildcarded) name of the file. Clients can decide, upon submitting, if they want to replace spaces with a * to work the way searching currently does.

[2004-01-15 14:51] <nysin> (Might someone suggest on dcdev-list that these people try some sample queries in their regex syntax on a largeish share?

If any of you need it, I have several large filelists (in bz2 format) that I passed along to cologic (nysin).  One in particular is 493gb and 172,000 files.  I think modeling a typical search load on this share might quickly tell if regexp searching is feasible.

A previous discussion on regexps also yielded concerns about... ahem... resource intensive expressions.  Nothing quite like being able to DOS all the clients in a hub simply by sending a well crafted regexp search.

Hm, 439GB ... what is that guy sharing exactly? Piracy is a crime...
no, silly answer :)

Any form of a more advanced query requires some time for execution (more or less). So how much time requires the query type suggested (fredrik was it, no?) ?
I'm not trying to promote regex now, just wanna know.

I can make a test with some various filelists. (C#, don't know C/C++ that good) Results coming up in a few days (or hours, depending how little I plan to sleep this night ;)

a part of a WinRAR packed file name "foo" with exact size 15000000 bytes
"SEARCH 217.210.105.76:412 03 text/* =15000000 :/^foo\.part[0-9]{2,3}\.rar$/\r\n"

Ah yes.  A wonderful example of why searching by MIME types is not so useful.  What exactly is the proper type for RAR files?

> application/rar ?
> application/x-rar-compressed ?
> How about .MKV?  OGM?  OGG?

sorry about that one, it was a really bad example, I know. I was in a hurry and didn't much bother to check what I had written :(

check this out:
http://www.iana.org/assignments/

a more 'readable' list or mimetypes have been put together here:
http://www.isi.edu/in-notes/iana/assignments/media-types/media-types

This is specified in RFC2045 and RFC2056 (as stated in the latter link)

These lists ARE standard. Unfortunatly, rar is not listed. And as so we can decide it (if we want to), as long the decition corresponds to the RFC's. And thus it can be, as Todd wrote, application/x-rar-compressed (RFC2046, page 9).
I say (if we use mimetypes) we use strictly standard, meaning if a client searches for "application/x-rar" - well bad for him if he dont get any results.
RFC2046 specifies that application/octet-stream should be used with "uninterpreted binary data". And doesn't rar files come under that category? :) If the user really wants to search for .rar files, well use the filename pattern instead!

/Carl-Adam

--On fredag den 16 januari 2004 03:37 +0100 Carl-Adam Brengesjö <[email protected]> wrote:

Yes in search results it's good with only one hash algorithm. I mean when
_searching_ for a file with a hash.

Well.. since everyone with hashing support in the client already calculates Tiger Tree Hashes for all their files, why support more hashing algorithms when searching for files?

Say when having a file in your queue (together with a MD5 file in same
directory) and want to search for alternative sources than the one you
have. You open the MD5 file and looks for the file you want... and copies
the md5 sum, say `3efdb9c9e399679cccd23179c4f1a1f2', then you searches
for files with that hash.

Sure. But if you originally downloaded the file from someone with hashing support in the client, you already have hashes for the file in your queue.xml, and if you don't, you really have no way of knowing if it's actually the same file or not, except if a .md5 file (.sfv is CRC32, which IMHO isn't good enough since it's a checksum, not a hash) is supplied with the file. Most people are never going to keep .md5 files in their share.

TTH, on the other hand, is done mostly transparent to the user. Which is good, since users are all idiots. :P

/fnord, can you tell i work as a sysadmin?

... it's difficult to get the ass out of the chair in the middle of
interesting conversations ;) *must sleep*

optimize your time, sleep on your keyboard ;)

Eric

Carl-Adam Brengesjö wrote:

why lock it to a single hash algorithm?

Because searches are expensive.  Because hashing is expensive.  Because some algorithms (uuhash, crc32, compound md4) have weaknesses that fundamentally undermine their purpose (uniquely identifying files).  Because some algorithms don't lend themselves to incremental

you can't know the hash of the file you're looking to unless you have downloaded a *.sfv (what algorithm? is it called sfv, or?) or MD5 (or simlar) file telling it.

MD5 and CRC32 are crude.  If we support a single hash, as three clients already do, sites such as ShareReactor, ShareLive, or FileNexus will pop up with direct download links for the DC supported hash, allowing our users to start a download from a hash.

Here's a rough overview of how BCDC does hashing (some of these steps are only visible when it's advertising itself as BCDC):  It crawls your entire share in a low-priority thread (it completes as the client is running, it doesn't block startup) hashing files and adding the full hash tree to a database.  When it returns a search result, it will replace the hub name field with TTH:<hash>.  When connecting to another client, it includes TTH in its $Supports list; when a fellow TTH supporting source is found (when downloading files in the user's queue), it gets the full hash tree (once) using a new client to client command: $GetMeta.  This tree can be used to verify both segments and the whole file.

If I seem brief... it's because I am.  We've had plenty of excellent discussion in this on the DC Dev public and private hubs, as well as on the DC++ forum (http://dcplusplus.sourceforge.net/forum/viewtopic.php?t=277) over the course of the last year.  Tiger Tree Hashes were chosen for some unique properties of the root file hash and of the tree of segment hashes.  This is really a Good Way(tm) to do hashing... otherwise I wouldn't bother bringing it up.

- Todd

Mattias Bergsten wrote:

Sure. But if you originally downloaded the file from someone with hashing support in the client, you already have hashes for the file in your queue.xml, and if you don't, you really have no way of knowing if it's actually the same file or not, except if a .md5 file (.sfv is CRC32, which IMHO isn't good enough since it's a checksum, not a hash) is supplied with the file. Most people are never going to keep .md5 files in their share.

TTH, on the other hand, is done mostly transparent to the user. Which is good, since users are all idiots. :P

Yea, never said it would not be could done like that. But say if a user is not an idiot (does that statement relates to the other one; you being sysadmin? ;P) this may be a good feature, no? Mean, why not tth in results, like you said, and different choices in searching?

No, I have no other/better to suggest. So use TTH then, fine with me.

Hm, was working on a (another) scenario where you'd require the choice of hashes, but I realized (in the middle of the writing) that the teqnique already used is enough for the purpose. Dunno what I thinked... shouldn't write emails 04:20 AM ;) *forget my idea*

Said .sfv because that's what I've seen being used on DC, figured that you are used to it - no idea what it really is, or why it's used. Personally, I use MD5 because.. i dunno.. i've always used it and is the most widley used :) (btw, if case of md5, the files are simply called "MD5", not *.md5)

Just got Todd's mail before sending this one;
Yes, I know - if you read my other mails (after that one you replied to) you realize that I did not understood you where talking about search replies. Thought you were talking about the search query. :/

/Carl-Adam, please ignore my letters after 2 AM...

ps. I'm going to bed now, before I cause any more mess in this list ;)

Carl-Adam Brengesjö wrote:

Hm, 439GB ... what is that guy sharing exactly? Piracy is a crime...
no, silly answer :)

The point is that unless you want to drive those with large shares and large filecounts away, any new query system (regexps) needs to be usable.

Any form of a more advanced query requires some time for execution (more or less). So how much time requires the query type suggested (fredrik was it, no?) ?
I'm not trying to promote regex now, just wanna know.

I don't care too much myself, my share is of a size that the number of hubs and number of searches/users does not load my machine at all.  However, that isn't the case for all users.  fusbar is one user (and list subscriber) for whom searches take up non-trivial amounts of CPU usage, on a respectable computer.

I can make a test with some various filelists. (C#, don't know C/C++ that good) Results coming up in a few days (or hours, depending how little I plan to sleep this night ;)

Like I said, I have real user's lists.  A bunch of them.  If you don't, go into a random hub and grab everyone's list.... Testing with made up data when real data is easily accessible is going to cast needless doubt on your conclusions.

These lists ARE standard. Unfortunatly, rar is not listed. And as so we can decide it (if we want to), as long the decition corresponds to the RFC's. And thus it can be, as Todd wrote, application/x-rar-compressed (RFC2046, page 9).
I say (if we use mimetypes) we use strictly standard, meaning if a client searches for "application/x-rar" - well bad for him if he dont get any results.
RFC2046 specifies that application/octet-stream should be used with "uninterpreted binary data". And doesn't rar files come under that category? :) If the user really wants to search for .rar files, well use the filename pattern instead!

Guessing/deciding non-standard MIME types on our own?  So if MIME searching doesn't totally replace the existing system, why make any new proposal more complex by including it?

- Todd

Carl-Adam Brengesjö wrote:

I can make a test with some various filelists. (C#, don't know C/C++ that good) Results coming up in a few days (or hours, depending how little I plan to sleep this night ;)

I think I misinterpreted this line. :)

- Todd

Carl-Adam Brengesjö writes:
> About filename matching in the other search command, how do you know > where the string starts (I understand that $ ends)
> ( N =~ avi$ ) would match (not) "avi" or (not) " avi".
> isn't it just better to have it surrounded by "" (use $ if you like, " > is just a more standard way, and any char is as good as the other)

Actually, I had intended every logical part of the search string to be
a seperate token in the line sent over the protocol that I
quick-drafted earlier. If you look in the archives for the `Text
protocol draft' draft that I sent Dec 2nd, you'll see what I
mean. It's also included in my upcoming protocol draft, which is more
complete than that mail.

Fredrik Tolf

Todd Pederzani wrote:

Guessing/deciding non-standard MIME types on our own?  So if MIME searching doesn't totally replace the existing system, why make any new proposal more complex by including it?

Ok, then use the *nix program `file' - wich can easily be ported to any type of OS, or atleast with some if it's functions. And therefor using a shared magic.mime (a file that provides data to guess mime types) can easily be done - if that's the problem.
The current way to determine file types isn't pretty slick, so that's why (at least) I am trying to figure out a new one.
And all other suggestions for deciding filetype have been so far to have it being based on filextensions, wich is not the same!

Using a shared magic.mime may not be the best way, but atleast the best suggestion so far.

/Carl-Adam

Actually, I had intended every logical part of the search string to be
a seperate token in the line sent over the protocol that I
quick-drafted earlier. If you look in the archives for the `Text
protocol draft' draft that I sent Dec 2nd, you'll see what I
mean. It's also included in my upcoming protocol draft, which is more
complete than that mail.

It can be good. With this, we will perhaps be able to "stabilize" some parts of the protocol to start programmation of clients/hubs.

Eric

Todd Pederzani wrote:

Carl-Adam Brengesjö wrote:

And all other suggestions for deciding filetype have been so far to have it being based on filextensions, wich is not the same!

The best suggestion so far is to leave the current system of file types (where a type is some set of file extensions) in place.

>

Again: what does using MIME types gain us over file extensions?

Guessing MIME types are done by look on the actual data, not just the name of the file. And therefor searching for a particular file extension should be done in the filename pattern. So if you want to search for the _type_ of file, file extension are no good.
Using MIME types allows the user to search for the _type_ not the fileextension, and I say again. If you want to search for the file extension: use the filename pattern!

I searched on DC now for rar files, using filename pattern "." and `filetype' compressed. What I got was *.zip and *.rar, but what with *.r01, *.r02 (etc) ? Using mime types here would return _all_ files that are rar files.

Sure,  we could basically recreate/port the unix way of guessing mime type, but... why?  It gains us nothing, and it adds complexity and uncertainty.  Not all will handle MIME guessing the same, so you've just introduced interoperability issues.

Not all clients will decide filetypes on extension either. When I did my search described above - I got 7 (SEVEN!!) hits on directories that ended with ".ra", and therefor treated as compressed files (the clients who responded used DC++ 0.303, is this fixed in newer releases?).

I say again, filetypes with mime decides the type depending on the actual _data_, what it contains. File extension is basicly just a cheap trick to `know' it's type even before the file has been opened, and can therefor be easily faked - or mistaken.

But I seem to be the only one that likes the idea with mime types, and everyone is fighting against me on the subject... `resistance is futile' - at least it seems to be :(

I have to say same thing over and over in this matter (with different words) and either is it you that don't understand what I'm saying, or it's me who doesn't understand what I want to be said (or needed to be said) :/ either way we are not making any progress. I give up! :p

/Carl-Adam

eric writes:
> > > Actually, I had intended every logical part of the search string
> > to be a seperate token in the line sent over the protocol that I
> > quick-drafted earlier. If you look in the archives for the `Text
> > protocol draft' draft that I sent Dec 2nd, you'll see what I
> > mean. It's also included in my upcoming protocol draft, which is
> > more complete than that mail.
> > It can be good. With this, we will perhaps be able to "stabilize"
> some parts of the protocol to start programmation of clients/hubs.

I already edit it in my public_html directory, so if you want a sneak
peak, you can look in at:

http://www.dolda2000.com/~fredrik/doldaconnect/protocol.html

Note that since it's far from completed, it's rather incoherent,
though, and it contains references to sections that I have yet to
write. However, it would be good if people would start looking through
the sections that I've written about the semantics of the protocol and
give me some early feedback.

Fredrik Tolf

Carl-Adam Brengesjö wrote:

ps. I'm going to bed now, before I cause any more mess in this list ;)

Even posts with fallout are better than no discussions.

I trust you've downloaded the full archive to see some of the past topics? :)

- Todd

Todd Pederzani wrote:

Carl-Adam Brengesjö wrote:

ps. I'm going to bed now, before I cause any more mess in this list ;)

Even posts with fallout are better than no discussions.

I trust you've downloaded the full archive to see some of the past topics? :)

- Todd

Of course, but I can't memorize them all :(

... it's difficult to get the ass out of the chair in the middle of interesting conversations ;) *must sleep*

/Carl-Adam

I say again, filetypes with mime decides the type depending on the
actual _data_, what it contains. File extension is basicly just a cheap
trick to `know' it's type even before the file has been opened, and can
therefor be easily faked - or mistaken.

You are fully right.

But I seem to be the only one that likes the idea with mime types, and
everyone is fighting against me on the subject... `resistance is futile'
- at least it seems to be :(

IMHO, mime types is not the solution. Something like mime-magic is probably better but has its limit. How to handle files with unknown content but known extension and how to handle new file format or "rare" ones (who knows PES format ?).

Eric

eric wrote:

But I seem to be the only one that likes the idea with mime types, and
everyone is fighting against me on the subject... `resistance is futile'
- at least it seems to be :(

IMHO, mime types is not the solution. Something like mime-magic is probably better but has its limit. How to handle files with unknown content but known extension and how to handle new file format or "rare" ones (who knows PES format ?).

Magic-mime will give a file with unkown content the type `text/plain' and so it always have a mime type, though it may not be accurate. But event the current system isn't accurate with stuff like that.

What does the current system do if it encounters an unkown file extension when searching for a particular type? It ignores the file! Why cannot this do the same? If you search on `any' with mimetypes the mimetype would be `*/*', and the file will show up with `text/plain' and listed then. Just like it's done now.

And for new file formats - the magic.mime file can easily be updated with a new version of the client, and most people use a newer version of a client when it is released. The file can even be hosted on sites like the hublists, but this requires some bandwidth on the server and new fileformats isn't really released every 5-minutes (relevant fileformats), so it's enough with the mime.magic is updated with a new version of the client (if required).

Filetypes like rar and so is not listed in the current magic.mime (i have version 3.41 of file) but the file has a simple syntax and new fileformats and other unkown can easily be added.

And as a last solution the mime types can be overriden by the client depending on file extension. But this is not recommended as each client could override it with it's own mime types (though a quick way of dealing with it).

As for janvidar's reply to Opera: Yea, you manage to say what I've been trying to say the whole time (I suck on explaining stuff).

/Carl-Adam

http://www.dolda2000.com/~fredrik/doldaconnect/protocol.html

This is just my opinion but I give my reasons:

Format:
======
CRLF vs CR: I think using a single character (not only CR) as command separator is better because it uses less bandwidth but also because it is faster to search (only 1 byte to compare instead of 2) and commands can be splitted using a simple strchr().

Quoting: That seems good to me but we must define how backslashes are handled when encountered inside double quote (ignore the \ and discard it or keep it working as outside double quote). I think backslash should work everywhere in the same way and more over, this will allow this "somewords\"otherwords".

Semantics:
========

Roles: DC without this architecture is not DC :)

Word sets: I think commands should be case-sensitive (for faster comparison) but we should forbid commands having same case-independent values (like toto and Toto).

User identification: ok for me.

Search criteria: we should also define the maximum number of reply a client can send.

Context: I don't understand the need of context in a client, I think the user privilege should be only maintain in the hub. With my hub for example, it is possible to have more than one "level" of operator with more or less privilege (some op can "only" kick users but some others can also modifiy the hub configuration). On connect, perhaps the hub should send a list of all commands a client can send (IMHO, a bit heavy).

References:
=========

DESC: someone still uses a 32bit integer to store the share ? ;)

About connection stage, I assume GUID is the first command you sent (correct me if I am wrong). Why not having the following connection establishment:

1: H=>C: the hub sends its ID (something like the GUID of the client but for the hub)

2: C=>H: the client sends its ID and its password simultaneously (I modified your "GUID ID" command into "GUID ID password"). Then we can do the full authentification in a single step and without error, thanks to the GUID of the hub

3: H=>C: the hub accepts an continue the process into "normal" mode or close the connection after a message.

Carl-Adam Brengesjö writes:
> Todd Pederzani wrote:
> Guessing MIME types are done by look on the actual data, not just the > name of the file. And therefor searching for a particular file extension > should be done in the filename pattern. So if you want to search for the > _type_ of file, file extension are no good.
> Using MIME types allows the user to search for the _type_ not the > fileextension, and I say again. If you want to search for the file > extension: use the filename pattern!
> > I searched on DC now for rar files, using filename pattern "." and > `filetype' compressed. What I got was *.zip and *.rar, but what with > *.r01, *.r02 (etc) ? Using mime types here would return _all_ files that > are rar files.
> > > Sure,  we could basically recreate/port the unix way of guessing mime > > type, but... why?  It gains us nothing, and it adds complexity and > > uncertainty.  Not all will handle MIME guessing the same, so you've just > > introduced interoperability issues.
> > Not all clients will decide filetypes on extension either. When I did my > search described above - I got 7 (SEVEN!!) hits on directories that > ended with ".ra", and therefor treated as compressed files (the clients > who responded used DC++ 0.303, is this fixed in newer releases?).
> > I say again, filetypes with mime decides the type depending on the > actual _data_, what it contains. File extension is basicly just a cheap > trick to `know' it's type even before the file has been opened, and can > therefor be easily faked - or mistaken.
> > But I seem to be the only one that likes the idea with mime types, and > everyone is fighting against me on the subject... `resistance is futile' > - at least it seems to be :(
> > I have to say same thing over and over in this matter (with different > words) and either is it you that don't understand what I'm saying, or > it's me who doesn't understand what I want to be said (or needed to be > said) :/ either way we are not making any progress. I give up! :p

I think that I might have made myself misunderstood. I do agree with
you, really, but only in the perfect world. MIME types are most
certainly superior in theory, and I certainly don't like determining
the file type from the extension (note, however, that Windoze
determines MIME types from the extension...).

The problem is that there are too many MIME types in this world that
are not standardized, and in my experience, it is especially the file
types that people on DC would be interested in. DC users aren't
typically interested in message/rfc-822 or text/plain - they are
interesting in what _would_ be video/ms-avi, application/rar or
application/iso-9660-img, for example, only those formats are not
standardized. Therefore, MIME types are not of very much interest to
DC users.

However, the last thing I want to do is to ban MIME types from
usage. On the contrary; consider my search format, for example. It can
be implemented at once, or once these file formats have standard MIME
types, but at any point, it's easy to add something like
( M =~ video/.* ). That way, it will be optional in the way that it
should be. Those clients that don't have the MIME type attribute
implemented will simply return false on those queries.

That's my thought.

Fredrik Tolf

> IMHO, mime types is not the solution. Something like mime-magic is
> probably better but has its limit. How to handle files with unknown
> content but known extension and how to handle new file format or "rare"
> ones (who knows PES format ?).

Magic-mime will give a file with unkown content the type `text/plain'
and so it always have a mime type, though it may not be accurate. But
event the current system isn't accurate with stuff like that.

that's why I said something LIKE mime-magic, not mime-magic itself, it is too limited.

Eric

However, the last thing I want to do is to ban MIME types from
usage. On the contrary; consider my search format, for example. It can
be implemented at once, or once these file formats have standard MIME
types, but at any point, it's easy to add something like
( M =~ video/.* ). That way, it will be optional in the way that it
should be. Those clients that don't have the MIME type attribute
implemented will simply return false on those queries.

Yes.... Vote for Fredrik.... Fredrik President ;)

Eric

eric writes:
> > > http://www.dolda2000.com/~fredrik/doldaconnect/protocol.html
> > This is just my opinion but I give my reasons:
> > Format:
> ======
> CRLF vs CR: I think using a single character (not only CR) as
> command separator is better because it uses less bandwidth but also
> because it is faster to search (only 1 byte to compare instead of
> 2) and commands can be splitted using a simple strchr().

Yeah... I just recently implemented a protocol parser for the UI
controller channel for my new client, and I just couldn't find a
really elegant way of handling CRLFs... I'm pretty much for that as
well. I'll change it in the spec.

> Quoting: That seems good to me but we must define how backslashes are handled > when encountered inside double quote (ignore the \ and discard it or keep it > working as outside double quote). I think backslash should work everywhere in > the same way and more over, this will allow this "somewords\"otherwords".

That was precisely what I was thinking. I thought I had made that
clear enough? I'll look over my wording.

> Semantics:
> ========
> > Roles: DC without this architecture is not DC :)
> > Word sets: I think commands should be case-sensitive (for faster
> comparison) but we should forbid commands having same
> case-independent values (like toto and Toto).

OK, I can agree to that; it has another reason as well. The ISO C spec
doesn't actually have a wcscasecmp function declared. When I checked
it out, wcscasecmp was a GNU extension to glibc. Therefore, that might
actually be a good idea in more than one way. So... uppercase or
lowercase? =)

In that case it'll have to be lowercase - easier to type. I'm sure
everyone agrees. I'll change that too.

> User identification: ok for me.
> > Search criteria: we should also define the maximum number of reply a client > can send.

That might actually be a good idea. However, I see that being part of
the search protocol, not the criteria. In my mind, that's not really a
matching operator.

> Context: I don't understand the need of context in a client, I think the user > privilege should be only maintain in the hub. With my hub for example, it is > possible to have more than one "level" of operator with more or less > privilege (some op can "only" kick users but some others can also modifiy the > hub configuration). On connect, perhaps the hub should send a list of all > commands a client can send (IMHO, a bit heavy).

Yeah, I know the context concept is a bit heavy, but I think that it's
a good idea, nonetheless. That way, it will always be very clearly
defined which commands are valid.

The whole thing is that I want to stay as far away as can ever be
possible from the state machine stuff used in current DC
implementations. Like the current client-to-client protocol - if you
don't send the commands in the exact right order, the client will
simply stop responding. I think that's ugly and repulsive, and this
will more or less force people into building better implementations.

It also makes it easier to extend the protocol, as I stated in the
document. The client could request an extra context for some specific
commands, and if that context is implemented in the hub/peer, then all
is good, otherwise it isn't. It's a very clear way of doing it, in my
mind.

Please continue to have your say on this.

> References:
> =========
> > DESC: someone still uses a 32bit integer to store the share ? ;)

Nah, I don't really think so. I just thought that some lesser
programmer might decide to make a DC implementation, so I wanted to
make it clear from the very start.

> About connection stage, I assume GUID is the first command you sent (correct > me if I am wrong). Why not having the following connection establishment:
> > 1: H=>C: the hub sends its ID (something like the GUID of the client but for > the hub)
> > 2: C=>H: the client sends its ID and its password simultaneously (I modified > your "GUID ID" command into "GUID ID password"). Then we can do the full > authentification in a single step and without error, thanks to the GUID of > the hub
> > 3: H=>C: the hub accepts an continue the process into "normal" mode or close > the connection after a message.

Again, my utter goal is to avoid this state machine stuff. Such a
thing as a "first command" should not be defined. It is equally OK to
send any command that is part of the `identify' context, be it DESC or
GUID. The important thing is what should be done, not in what order.

Fredrik Tolf

 > About connection stage, I assume GUID is the first command you sent
 > (correct me if I am wrong). Why not having the following connection
 > establishment:
 >
 > 1: H=>C: the hub sends its ID (something like the GUID of the client but
 > for the hub)
 >
 > 2: C=>H: the client sends its ID and its password simultaneously (I
 > modified your "GUID ID" command into "GUID ID password"). Then we can do
 > the full authentification in a single step and without error, thanks to
 > the GUID of the hub
 >
 > 3: H=>C: the hub accepts an continue the process into "normal" mode or
 > close the connection after a message.

Again, my utter goal is to avoid this state machine stuff. Such a
thing as a "first command" should not be defined. It is equally OK to
send any command that is part of the `identify' context, be it DESC or
GUID. The important thing is what should be done, not in what order.

Can you give an example (or 2) of your way a connection is established between a client and a hub. I think state machines are good but it must be a one with no dead end.

Eric

eric writes:
> Can you give an example (or 2) of your way a connection is
> established between a client and a hub. I think state machines are
> good but it must be a one with no dead end.

I, too, think state machines are good, but not here.

The point of it is so that the identification conversation can look in
all these ways and more:

1:
C->H: GUID xxx
H->C: ACCP
C->H: DESC xxx
H->C: WELC

2:
C->H: DESC xxx
H->C: ACCP
C->H: GUID xxx
H->C: WELC

3:
C->H: GUID xxx1
H->C: ACCP
C->H: GUID xxx2
H->C: ACCP
C->H: GUID xxx3 (note that this will be the authoritative one)
H->C: ACCP
C->H: DESC xxx
H->C: WELC

Essentially, a state machine where you must send GUID first and then
DESC is like this context engine, only that it has even more contexts
and even more implicit changes. Therefore, I think this is clearer,
more flexible, and also allows for human input error when eg. typing
in a Telnet session. Also, since all contexts are named, that gives a
very strict definition of the protocol - there can be no doubts
anywhere about when you should/could send what.

At least that is the idea I had behind it.

Fredrik Tolf

1:
C->H: GUID xxx
H->C: ACCP
C->H: DESC xxx
H->C: WELC

2:
C->H: DESC xxx
H->C: ACCP
C->H: GUID xxx
H->C: WELC

3:
C->H: GUID xxx1
H->C: ACCP
C->H: GUID xxx2
H->C: ACCP
C->H: GUID xxx3 (note that this will be the authoritative one)
H->C: ACCP
C->H: DESC xxx
H->C: WELC

but this remains a state machine (even a very simple one):

Node:
S0 : initial state
S1: state when GUID was received first
S2: state when DESC was received first
S3: end state (welc)

"Function meaning":
Sx (Y) = Sz means when state is Sx and Y is received, go to Sz.

S0 (GUID) = S1  [accp sent]
S1 (GUID) = S1 [accp sent]
S1 (DESC) = S3 [welc sent]

S0 (DESC) = S2  [accp sent]
S2 (DESC) = S2 [accp sent]
S2 (GUID) = S3 [welc sent]

Can you give an example with a password. I'd like to know at which time you reject invalid login and/or invalid password.

Eric

Fredrik Tolf wrote:

Essentially, a state machine where you must send GUID first and then
DESC is like this context engine, only that it has even more contexts
and even more implicit changes. Therefore, I think this is clearer,
more flexible, and also allows for human input error when eg. typing
in a Telnet session. Also, since all contexts are named, that gives a
very strict definition of the protocol - there can be no doubts
anywhere about when you should/could send what.

At least that is the idea I had behind it.

Instead of sending contexts forth and backward, isn't is just simplier to have a strict order to send the commands? And/or more specific reply messages instead of `ACPT' for every kind of command :/

> Yeah, I know the context concept is a bit heavy, but I think that it's
> a good idea, nonetheless. That way, it will always be very clearly
> defined which commands are valid.

Why bother? If the command isn't valid just reply with a message that he/she are not authorized to use it (you gotta have such a message anyways) And if you mean to disable the function to send the command at all from the client, the client could aswell `detect' if it has been given operator status.

If you use context, use binary flags instead of string names - you can save alot of bandwidth - instead of 18 bytes ("base,main,operator") you have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using table described below).

base     0
identify 1
auth     2
main     4
operator 8

base|main|operator = 12

And last, about adding more operator `levels' - dont! I can only see the usage of adding a `voice' (vip), and a global operator level if channel support are going to be supported.
Adding too many of userlevels in the protocol don't allow much of the customization that is available to DC today regarding user modes/access.

/Carl-Adam

eric writes:
> > > 1:
> > C->H: GUID xxx
> > H->C: ACCP
> > C->H: DESC xxx
> > H->C: WELC
> >
> > 2:
> > C->H: DESC xxx
> > H->C: ACCP
> > C->H: GUID xxx
> > H->C: WELC
> >
> > 3:
> > C->H: GUID xxx1
> > H->C: ACCP
> > C->H: GUID xxx2
> > H->C: ACCP
> > C->H: GUID xxx3 (note that this will be the authoritative one)
> > H->C: ACCP
> > C->H: DESC xxx
> > H->C: WELC
> > but this remains a state machine (even a very simple one):

Oh yes, most things can be described as a state machine. There are two
main things with these contexts, though.
First of all, the semantics is not that of a state machine. While a
context can be described as a state machine, I did not mean for it to
be thought of as one. The thing is that all commands within a single
context are always valid, no matter what commands were sent
earlier. As you said yourself, it can be described as a state machine,
but the semantics that underlie the notion of a context is not that of
a state machine.
Second, there can be several contexts valid at a single time. Sure,
this can still be described as a state machine, but the abstraction
that lies in the notion of a context simplifies the overall picture.

The thing is that I want to avoid these state machines that, in your
words, have "dead ends". That is why I want the thought of having a
certain set of commands that are valid at a specific time, and that
the effect of these commands are primarily on data, not on the set of
commands that will be valid after the command has been executed.

As you can see, both ways of implementing it can easily be transferred
into the other, but what I want to achieve is semantics, not
implementation.

I hope I made myself clear, although I doubt it a bit. =)

> Can you give an example with a password. I'd like to know at which time you > reject invalid login and/or invalid password.

Sure. I'll include the context changes in this example:

(Connection accepted, the active contexts are `base' and `identify')
C->H: guid xxx
H->C: accp
C->H: desc xxx
H->C: npwd noecho "Password: " (I'm thinking a PAM-like scheme right now)
(The `npwd' reply has an implied context change, that removes the
`identify' context and adds the `auth' context. Sure, a kind of state
machine, but with different semantics. The active contexts are now
`base' and `auth')
C->H: pass " Secret "
H->C: fail (If the hub really didn't like the client, it can close the
       connection now, although `auth' remains in place, so if the
       hub _doesn't_ close the connection, the client can try again.)
C->H: pass "  Secret "
H->C: welc
(The `welc' reply has as implied context change that removes `auth'
and adds `main'. Thus, the active contexts are now `base' and `main'.)

At least this is what I have had in mind.

Fredrik Tolf

Carl-Adam Brengesjö writes:
> Fredrik Tolf wrote:
> > Essentially, a state machine where you must send GUID first and then
> > DESC is like this context engine, only that it has even more contexts
> > and even more implicit changes. Therefore, I think this is clearer,
> > more flexible, and also allows for human input error when eg. typing
> > in a Telnet session. Also, since all contexts are named, that gives a
> > very strict definition of the protocol - there can be no doubts
> > anywhere about when you should/could send what.
> > > > At least that is the idea I had behind it.
> > > > Instead of sending contexts forth and backward, isn't is just simplier > to have a strict order to send the commands? And/or more specific reply > messages instead of `ACPT' for every kind of command :/

I never intended for contexts to be sent back and forth, unless
explicitly desired. I have intended for many commands to have implied
context changes. The "strict order" is exactly what I have wanted to
avoid, since it's so extremely inflexible. For example, if you send
commands out-of-order in the existing client-to-client DC protocol,
the peer usually stops responding or disconnects. In that case, why
even have command names to begin with? What good would ever come from
requiring that one command be sent before another when there is no
apparent meaning of the order (ie. why would you want to _require_
that GUID be sent before DESC when it can just as well be made the
other way around)?

What kind of more specific replies is it that you want? ACPT is just
the basic idea of the command having been executed as it was expected
to. Exceptional events are indeed handled through other replies.

>  > Yeah, I know the context concept is a bit heavy, but I think that it's
>  > a good idea, nonetheless. That way, it will always be very clearly
>  > defined which commands are valid.
> > Why bother? If the command isn't valid just reply with a message that > he/she are not authorized to use it (you gotta have such a message > anyways) And if you mean to disable the function to send the command at > all from the client, the client could aswell `detect' if it has been > given operator status.

If you get a reply indicating that the command wasn't accepted, then
why is that? Isn't it actually because it was sent in the wrong
context? Since the hub apparently has to have some kind of idea of
concepts anyway (or it wouldn't know whether to accept or deny a
command), why not actually standardize it, so that all implementors
know what is valid at what point?

Also, in my mind, contexts also allow for easy extensibility to the
protocol. Instead of sending commands to probe whether or not they are
valid, the client can request a certain context and see if the hub is
willing to give it. In that way, it also shows the hub's
capabilities.

> If you use context, use binary flags instead of string names - you can > save alot of bandwidth - instead of 18 bytes ("base,main,operator") you > have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using > table described below).

Since I want to define contexts as a means of extending the protocol,
having a set number of maximum contexts would be a bad idea.

> And last, about adding more operator `levels' - dont! I can only see the > usage of adding a `voice' (vip), and a global operator level if channel > support are going to be supported.
> Adding too many of userlevels in the protocol don't allow much of the > customization that is available to DC today regarding user modes/access.

I wasn't just planning on using it for adding more operator
"levels". I invented it to do anything. For example, if you make a hub
and a client that support a very specific search request, a specific
context could be used to identify that as well.

I think that extensibility is always of the essence, since we want a
flexible protocol this time.

Fredrik Tolf

Carl-Adam Brengesjö wrote:

I searched on DC now for rar files, using filename pattern "." and `filetype' compressed. What I got was *.zip and *.rar, but what with *.r01, *.r02 (etc) ? Using mime types here would return _all_ files that are rar files.

If they had used a recent WinRAR version, those would be .part##.rar files.  The .R## and .Q## files should be going the way of the dodo (though it looks like you can enable them with the -vn switch, if you must).  What other types of files do you see as having bad extensions and being in need of mime-type guessing?

Not all clients will decide filetypes on extension either. When I did my search described above - I got 7 (SEVEN!!) hits on directories that ended with ".ra", and therefor treated as compressed files (the clients who responded used DC++ 0.303, is this fixed in newer releases?).

What you're seeing is actually a RAR file hit, mangled into a directory result by a bug in 0.303.  And yes, 0.304+ fixes that bug (but still doesn't return ".ra" files in response to a compressed file search).

I say again, filetypes with mime decides the type depending on the actual _data_, what it contains. File extension is basicly just a cheap trick to `know' it's type even before the file has been opened, and can therefor be easily faked - or mistaken.

Nonetheless, file extension is a widely accepted method of doing things.  If we were programming for Mac OS 9.x and were using the creator and type fields of the resource fork (a much nicer concept than either mime type or extension), you'd get no argument from me.

But I seem to be the only one that likes the idea with mime types, and everyone is fighting against me on the subject... `resistance is futile' - at least it seems to be :(

I don't see (in my experience or your arguments) how file extensions are broken.  If you're concerned about simplicity, define a standard set of extensions for a given "type" - or let unix clients use mime-type to sort their shares into types.

I have to say same thing over and over in this matter (with different words) and either is it you that don't understand what I'm saying, or it's me who doesn't understand what I want to be said (or needed to be said) :/ either way we are not making any progress. I give up! :p

Ditto.

- Todd

If you use context, use binary flags instead of string names - you can
save alot of bandwidth - instead of 18 bytes ("base,main,operator") you
have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using
table described below).

base     0
identify 1
auth     2
main     4
operator 8

base|main|operator = 12

earlier, we have said we won't use binary values inside a text protocol to have something clean.

And last, about adding more operator `levels' - dont! I can only see the
usage of adding a `voice' (vip), and a global operator level if channel
support are going to be supported.
Adding too many of userlevels in the protocol don't allow much of the
customization that is available to DC today regarding user modes/access.

I agree, such things must be kept inside the hub and be invisible to client.

Eric

Jonathan Hess wrote:
> Sorry, catching up on old mail. Why not encode the bitflag to an ascii > int and pass that down the wire. Still text, still compact. ;/

I curse myself for replying directly to eric instead of to the list !!!
anyways.. here's my mail together with eric's reply.

----- My mail ----------------------------------------
eric wrote:

>> If you use context, use binary flags instead of string names - you can
>> save alot of bandwidth - instead of 18 bytes ("base,main,operator") you
>> have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using
>> table described below).
>>
>> base     0
>> identify 1
>> auth     2
>> main     4
>> operator 8
>>
>> base|main|operator = 12
>
>
>
> earlier, we have said we won't use binary values inside a text protocol to have something clean.

Well dont treat is at a binary value then, use text. The integer "12" is 2 bytes (characters) in text, and 1 byte in a binary one, that what I meant with `cheating' (using binary values inside a text protocol).
Do you get me?
Instead of having a string with names (seperated by commas) sent in messages you send the integer values instead. And per my example instead of 18 bytes (characters) you send 2 bytes.
I dont say that we use a binary protocol, but we still send bytes because we are working with computers!!! I assumed you'd understand that!

/Carl-Adam

----- Eric's reply ------------------------------------
>Well dont treat is at a binary value then, use text. The integer "12" is
>> 2 bytes (characters) in text, and 1 byte in a binary one, that what I
>> meant with `cheating' (using binary values inside a text protocol).
>> Do you get me?
>> Instead of having a string with names (seperated by commas) sent in
>> messages you send the integer values instead. And per my example instead
>> of 18 bytes (characters) you send 2 bytes.
>> I dont say that we use a binary protocol, but we still send bytes
>> because we are working with computers!!! I assumed you'd understand that!

Ok, you want to use numeric value instead of plain text name. I agree, it is
shorter.

Eric

Carl-Adam Brengesjö writes:
[...]
>> If you use context, use binary flags instead of string names - you can
>> save alot of bandwidth - instead of 18 bytes ("base,main,operator") you
>> have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using
>> table described below).
>>
>> base     0
>> identify 1
>> auth     2
>> main     4
>> operator 8
>>
>> base|main|operator = 12
[...]
> ----- Eric's reply ------------------------------------
> Ok, you want to use numeric value instead of plain text name. I agree, > it is
> shorter.

With all due respect, I don't, since if we use a bitfield number for
flags, we only get 32 (or possibly 64) possible contexts, which also
require a centralized place for registration. I was hoping on using
the text-based contexts for extending the protocol as well. For
example, if a specific client implementation has some extra command
that it wants to use, it could request an extra context named
"myextension" or similar from the hub/peer, and if the request is
successful, then those commands will be available.

So you see, I was also planning on having the context scheme double as
capability arbitration. Don't you think that's even slightly good?

Fredrik

With all due respect, I don't, since if we use a bitfield number for
flags, we only get 32 (or possibly 64) possible contexts, which also
require a centralized place for registration. I was hoping on using
the text-based contexts for extending the protocol as well. For
example, if a specific client implementation has some extra command
that it wants to use, it could request an extra context named
"myextension" or similar from the hub/peer, and if the request is
successful, then those commands will be available.

So you see, I was also planning on having the context scheme double as
capability arbitration. Don't you think that's even slightly good?

we can also use a mix of both solutions. Having some standard values defined in the protocol like:

base     0
identify 1
auth     2
main     4
operator 8

and allow both "auth|identify" or "2|1" or "3". This allows non standard extension to be coded like "myextension|3" (more compact than "myextension|identify|auth" but still understandable by clients not supporting the extension).

Eric

Sorry, catching up on old mail. Why not encode the bitflag to an ascii int
and pass that down the wire. Still text, still compact. ;/

Jon Hess

-----Original Message-----
From: [email protected] [mailto:[email protected]]On
Behalf Of eric
Sent: Friday, January 16, 2004 11:21 PM
To: Direct Connect developers; Carl-Adam Brengesjö
Subject: Re: [dcdev] Searching

If you use context, use binary flags instead of string names - you can
save alot of bandwidth - instead of 18 bytes ("base,main,operator") you
have just 2 bytes (in a text protocol, 1 byte if we can cheat) (using
table described below).

base     0
identify 1
auth     2
main     4
operator 8

base|main|operator = 12

earlier, we have said we won't use binary values inside a text protocol to
have something clean.

And last, about adding more operator `levels' - dont! I can only see the
usage of adding a `voice' (vip), and a global operator level if channel
support are going to be supported.
Adding too many of userlevels in the protocol don't allow much of the
customization that is available to DC today regarding user modes/access.

I agree, such things must be kept inside the hub and be invisible to client.

Eric

eric writes:
> we can also use a mix of both solutions. Having some standard values defined > in the protocol like:
> >> base     0
> >> identify 1
> >> auth     2
> >> main     4
> >> operator 8
> and allow both "auth|identify" or "2|1" or "3". This allows non standard > extension to be coded like "myextension|3" (more compact than > "myextension|identify|auth" but still understandable by clients not > supporting the extension).

While that admittedly is an idea, I really don't think that it's
necessary. Most ordinary context changes like that will usually happen
through implied context changes on certain replies, and therefore
don't requre any bandwidth at all, anyway.

Fredrik Tolf