So lately I’ve been trying to find a torrent client that can handle thousands of torrents at a time and that is capable of downloading data in place of the .torrent file. For this I chose transmission for os x.
I have my music setup so that it’s
So now I’m able to use the collector at various torrent sites to download entire artists .torrent files at once. Once they are in the proper place where they will permenantly reside, then, using the a search, I locate all .torrent files in my Music directory and drag/drop them over into Transmission.
Transmission then loads all the .torrent files (removing the .torrent at the same time), and begins downloading the data to the directory in which it found the .torrent.
Currently the setup is a Mac Mini (late 2009) w/8GB RAM and gigabit uplink to my network.
It’s been laggy as hell, I can’t even get it to respond long enough to tweak the settings.
Just for a little background the network is setup as such:
Internet Uplink -> pfSense router -> gigabit switch -> Servers/Computers
The data is all being redundantly stored on a Synology 1812+ NAS disk station in RAID-5 format.
Update 1: So eventually I found out that the weakest link currently was my internal HDD in the Mac Mini that I’m using for torrenting. Whenever there is heavy throughput to the internal HDD, the mini will freeze up. So instead of having the files cache in the ~/Downloads directory, I’m not going to have them cache, but instead just write directly to the fileserver.
Update 2: Writing directly to the fileserver doesn’t work because it’s pre-allocating all the files on the fileserver, and thus having to transfer the data for each downloaded torrent twice to the fileserver, slowing everyting way down. Instead I’ll try having the incoming data cache on an external USB 3.0 disk.
Update 3: Everything working much better now that files are being cached on the external USB disk. A .torrent is loaded, the files are pre-allocated locally on the USB drive, the transfer is completed, then at high speed (40-50MB/sec) the data is uploaded to the fileserver and removed from the USB HDD. When a torrent finishes and the data is being uploaded to the fileserver, the incoming transfer rate drops to nearly 0 (but this isn’t a big deal as it takes less than a minute for the data to be copied over and the downloads to resume).
Update 4: Disabled the Web GUI to see if it makes things any quicker, seems to have amde things slightly faster.
Update 5: Set it up so that when a torrent finishes downloading, it completes after 1 minute instead of seeding. First lets get everything downloaded, then we’ll worry about seeding. Currently at 1500 torrents loaded simultaneously into transmission. Whenever a torrent finishes, Transmission lags up really bad while the data is being copied over to the fileserver. You cannot click anything or use the interface, and download stop downloading. While downloading you can navigate the interface as usual, though it is quite slow and choppy at times. For example, when switching between All, Active, Downloading, etc download lists, sometimes the full list won’t load (ill switch from Active to All, but the list still only displays my currently active transfers). Also wish there was a way to filter out only torrents that are still in progress (i.e. I don’t want to see all the torrents that are complete).
Update 6: Increased max simultaneous torrents to 40 and decreased the time it takes before a download is considered stalled to 1 minute. Peer is set to 80. Getting 12-13MB/sec download bursts, over 1TB transfer yesterday. Currently at ~3000 torrents loaded simultaneously into transmission.
Update 7: Increased max peers to 120 as there are many torrents running with only one peer and my downstream is no longer being saturated. It appears that Transmission will allow more than your max number of running downloads if it pauses a download because its stalled, then resumes it because it became available again. So even though I have my max concurrent downloads set to 40, there’s 80 torrents running right now. Currently at ~5000 torrents loaded simultaneously to transmission. Still saturating my downlink (though sporadically). Not going to load any more torrents until some of the currently loaded ones get fnished up. Switching views on the interface has become extremely slow, laggy, and seldom works corretly. I now have to restart Transmission every now and then if I want to add files or really check the status.
Update 8: Another day, nearly another TB transferred. Now running around 8k torrents, transmission still slow as hell.
Here’s traffic for the last week. Note that these are averaged, thus you don’t see all the 12-13MB/sec bursts. When watching a live monitor, I see that it normally goes from 0 to 12MB then back to 0.