How many files can be adressed by the muwire network?

[robabok978]

How many files can be found with an average time for searching?

If a file exist it will be found. True? How much true in percent?

If you search by hash , how much time does the search need in average to find a file?

I think you have a db for the hashes and do a lot of compare. How many searches per seconds are possible.

Thanks

[zlatinb]

The way it works is the node that starts the search broadcasts it to all nodes that it is directly connected to (the number of connections is on the bottom right). Those nodes in turn forward it to a smaller number their neighbors. So searching is pretty quick, if the result doesn't arrive within 10-15 seconds it will probably never arrive.

It's hard to answer how many files are searchable because it depends on how many files each node has shared. There is no central database of files or hashes anywhere.

Whether a file will be found if it's shared is hard to answer, but if you have high number of neighbor connections chances are the search will reach the node that is sharing the file.

[robabok978]

Thanks for yor answers

How many searches can be started. Are they searched in parallel or queued? 100 at once possible?

Does searching by hash or name does make any dfference speed wise?

If i would split he tie from start search to downloaded file
like this

start the seach to begin search
Getting results Best 100ms to Worst 15000ms
Reaction time uploader to start upload
Downloading Time/speed

I know it difficult to tell because of the variability oof the numbers above
but if i would put together all the times and call this latency.
How much would the speed/latency be comparable to a file uploaded like an image by a webserver while visiting.
number x 3, because of the 3 hops of i2p to have a fair comparison?
Would it be responive?

[zlatinb]

How many searches can be started. Are they searched in parallel or queued? 100 at once possible?

A node can start as many searches as it wants in parallel, but the network is limited to 10 queries/second. If a node receives more than 10 queries in a single second it will drop any further queries until the next second. Also, if the searches contain regular expression (only available in 0.8.12 beta) they are limited to 5/second and also 2 per user per second. This is because regular expression queries are more expensive to evaluate. This is done to prevent denial-of-service attacks and overload.

Does searching by hash or name does make any dfference speed wise?

Hash searches are fastest to evaluate. Keyword searches are second, and regular expression searches are slowest. But it should all be within milliseconds. Also, the GUI refreshes only twice a second so even the user moves the mouse very quickly they can start a download only after 500 ms. The slowest part however is not the query evaluation but delivery of the results because it requires establishing a connection from the responding node to the searching node and that involves a 3-way handshake.

So if we split the time like you propse, it's roughly:

start search - user has to type and press enter, so it depends on user speed
getting results - variable
reaction time to start download - at least 500ms
downloading time - depends on network speed, file size, etc.

[robabok978]

A node can start as many searches as it wants in parallel, but the network is limited to 10 queries/second. If a node receives more than 10 queries in a single second it will drop any further queries until the next second

Does the search initiating node diistribute the search queriy as one to every direct connected node or is the query split and very n node gets 1/n of the search query?

If a node drops more han 10 how does i know htat my search has been dropped or does it repeat itself or does the node push the dropped searches to other nodes by itself.? Is it lost somehow?

3-way handshake.

That are the hops on the way back, aren't they? This does mean that this time can't get any better if muwire would get faster in some routines. It's a i2pnetwork thing?

Would reducing hops for the uploader/result answer back to the searcher make it faster(less secure)?

Would it be possible to nott only send the answer back but also send the date at the time/with the first packet. Like a autodownload/autoupload? THis should not be possible for searches with regex or strings, because this would often results in multiply results. But for hashes this should be possible, because the result can only be destinct.

Also, the GUI refreshes only twice a second so even the user moves the mouse very quickly they can start a download only after 500 ms

But a scripted search could be easy faster than that? Let's say from list. How long would it be then, <1ms?
If such amount of searches occur, does muwire weight or use some algo to conclude how to distribute the search between direct connected nodes? Like Latency to the nodes or some metadate (if some is available by the hash or elsewhere)

Thanks

[zlatinb]

Does the search initiating node diistribute the search queriy as one to every direct connected node or is the query split and very n node gets 1/n of the search query?

The searching node sends the query message to all its neighbors. They however forward it only one hop away and only to a random fraction of their neighbors. Right now that fraction is 2*SQRT(numNeighbors).

If a node drops more han 10 how does i know htat my search has been dropped or does it repeat itself or does the node push the dropped searches to other nodes by itself.? Is it lost somehow?

It's lost. There is no feedback because the feedback mechanism could be abused for DDOS as well. Or at least I haven't thought of a way to build such mechanism without that risk.

That are the hops on the way back, aren't they? This does mean that this time can't get any better if muwire would get faster in some routines. It's a i2pnetwork thing?

Yes, it's an i2p network thing. I would say 95-97% of the time it takes to get a result back is dependent on how the I2P network performs.

Would reducing hops for the uploader/result answer back to the searcher make it faster(less secure)?

Yes. You can do that in the Options->I2P->Speed-vs-Anonymity slider. That will let you choose between 1 and 3 hop tunnels. If you want to build 0-hop tunnels you need to edit configuration files or the source.

Would it be possible to nott only send the answer back but also send the date at the time/with the first packet.

Not with the first packet because that would require changes to the I2P library, but after the 3-way handshake it is possible and easy to do for all types of queries. Why do you want this information?

But a scripted search could be easy faster than that? Let's say from list. How long would it be then, <1ms?

Yes, microseconds. As soon as the query is received you can kick off the download programmatically.

If such amount of searches occur, does muwire weight or use some algo to conclude how to distribute the search between direct connected nodes? Like Latency to the nodes or some metadate (if some is available by the hash or elsewhere)

No, it's not so smart. Also I'm targeting human users who are unlikely to make more than 10 searches in a second. So if you want to write a script that searches very quickly you will need to change the logic that broadcasts the query to neighbors.

[robabok978]

Would it be possible to nott only send the answer back but also send the date at the time/with the first packet.

Not with the first packet because that would require changes to the I2P library, but after the 3-way handshake it is possible and easy to do for all types of queries. Why do you want this information?

I am trying to rudece time. Wnat i want is something like an automated download with search hashes.


Times would be:
search found nodes returns answer 3 Hops
intiate download to uploader 3hops? ----> could this be spared?
uploader sends 3 hops
Retrieving first data packets.

The uploader should begin to send when it has received a search for a local existing file. Could this be done on/by the uploader side. Maybe only for hash searches. Retrieve a search - send to source.

Would reducing hops for the uploader/result answer back to the searcher make it faster(less secure)?

Yes. You can do that in the Options->I2P->Speed-vs-Anonymity slider. That will let you choose between 1 and 3 hop tunnels. If you want to build 0-hop tunnels you need to edit configuration files or the source.

Specific for the searches?

Something slightly off topic:

Are the files splited into blocks like emule did, so that peer could send blocks of smaller size and complete them with a kind of smart building like last block first block for video avi etc. Lesser distributed blocks loads first. Or a the files one identity and i can only download from one src/peer for one file?


Thanks

[zlatinb]

The uploader should begin to send when it has received a search for a local existing file.

Sure it can be done, but searching for a hash does not imply intent to download. For example I sometimes search for hashes and just check the size and comments about a file. If you are talking about some purpose-built solution for specific needs I can tell you which parts of the code need to change.

Specific for the searches?

The tunnel length setting applies to all I2P communication. Theoretically it is possible to have different length tunnels for different purposes, but that needs to be done at the I2P library level.

Are the files splited into blocks

Yes. All blocks are the same size except for the last and the block size is a function of the file size. It is possible to request them from different peers in whatever order the downloader prefers.

In fact, the MuWire hash of a file is not a hash of the file but a hash of the list of hashes of the blocks. So the first thing that happens when MuWire downloads a file is to request the list of hashes so that it can verify each block independently.

[robabok978]

Would it be possible with an extern application like a webbrowseraddon to parse a hash to muwire or a list of hashes for a search?

Muwire would return found, status of download, download complete and return a string of the local link where the file has been downloaded?

That's why i am asking of the possibility of download automation and the time it takes to search and downloaded a hash searched file and how many files the network can address.

Here is the reason why i ask all of this:

The general idea is a webbrowseraddon which search on a webpage for a hash in a tag like <muwiretag> in the src.
After that it will parse the hash to muwire. Muwire search and downloads the file and returns the link to the downloaded file as a string back to the addon. (it would use the local file if the file has already been downloaded, as a faster cache, should make the loading faster)
The addon now replace the hash and tag with the link to the downloaded file.

Muwire would act here like a distributed filesystem.

If you dowloaded the file you will distribute it.
You can filter content by the hash with filterlists
It'slike freenet but you will decide what you share.
It would act like distrubted datastorage in an easy way just publish your hashes anywhere, other people download it it gets mirrored automatically.
Think about websites where images,videos with a min amount of size, are only represented by hashes of files which can get filtert,downloaded, mirrored etc.
Maybe download packages repos for distros
and so on...

For this task muwire has to be able to parse, return, automatically/fasten download, maybe smarter search and i think thats it for muwire. The other work is more on the addon.

It would be nice to hear what you think about it, if this is possible with muwire + addon and the amount of work/difficulties you think this have/will have.

I like the idea, if it works. But there need to be a hint
For sure at first time slow, but if an amount of files had been downloaded it should get better. But everything on i2p is a little bit slower.

[zlatinb]

Wow that's an interesting idea. TL;DR: Yes it is possible, but it's a serious amount of work.

Longer answer is that it will work acceptable for larger files, but if you want a webpage with many small images then it will be slow to load. I don't want to change the 10/second limit on searches. Maybe something more clever can be done by the webbrowser extension where a .zip with all the images gets pulled as a single file from MuWire and then unpacked in the browser? That would work better.

Things like videos where the first=and-last blocks need to be requested will be tricky. I suggest as a first iteration to not support streaming.

Other than that MuWire supports something like plugins, so most if not all of the work can be done outside of the main code. If you clone the source from git and look inside the "mwmodules" directory you will see an example plugin.