Blom causes big latencies

[klondike]

I have discovered that BLOM has been the responsible of a big set of related issues involving big latencies, from the classical "It takes to long to authenticate" to "I get big latencies on my messages when hashing". This seems to be caused by the fact the filter is sent using the same channel as the rest of the data to the hub which causes a big bottleneck when said filter is big if a big number of files is available.

I'd like that we discuss and find a solution to this issue if you have time for it. Some coming out of the back of my head are:
* Set some escape sequence during file transfer that can be used to pause the transfer so other commands can be sent over the same channel.
* Send over packets the data transfers so other information can be interleaved (there are multiple ways to do it)
* Related to the previous two: Have switchable modes (suggested by iceman)
* Try to use a second channel for file transfers (either a second connection established with a token or, when using SSL a second channel).

[poy]

perhaps clients could simply split the data into smaller chunks, sent with multiple SND messages?

looking at the ADCH++ source, i'm guessing it would refuse partial tables with the "Invalid number of bytes" message; but it shouldn't be very difficult to amend that.

[poy]

perhaps clients could simply split the data into smaller chunks, sent with multiple SND messages?

looking at the ADCH++ source, i'm guessing it would refuse partial tables with the "Invalid number of bytes" message; but it shouldn't be very difficult to amend that.

an alternative that may be more in line with common ADC conventions would be for the hub to do that splitting, by requesting partial data in the GET messages that it sends.

[poy]

or hubs could just put an upper bound on the m parameter.

that's actually suggested in the algo description: "Also, with m grows the required bandwidth to transfer the bloom filter, so the hub may wish to cap m. In that case, it should still choose k according to m / n * ln 2, but send an h as big as possible to alleviate biasing problems."
all that will happen with smaller ms is that the probability of false positives will increase (less searches filtered).

[kathi]

I've currently switched to the other extreme: now using the (cid, hub) pair for everything. That at least guarantees that nothing will go wrong in that aspect. I can always make better use of CIDs and possibly provide fallbacks later on.