Switching back to dropbox

So after switching to BittorrencSync for a few months (which was working well) they bloated out the software and made it no longer completely free. Next I tried Synching, but it had issues with files that were open while it was syncing (for example KeePass).

So now its back to dropbox with EncFs encryption. DropBox’s synchronization just works really well. They have fast servers, and over all its a great solution. Only issue is trusting them, but so long as we are encrypting everything that doesn’t matter.

Additionally what i’ve done is created a small script to run on any freshly formatted computer i’m trying to setup for use.

This makes my dock sync across all computers, the screen shot default location to dropbox, and documents being stored on dropbox.

mv ~/Library/Preferences/com.apple.dock.plist ~/Library/Preferences/com.apple.d$

ln -s ~/Dropbox/File/osx/com.apple.dock.plist ~/Library/Preferences/com.apple.d$

defaults write com.apple.screencapture location ~/Dropbox/Screenshots

mv ~/Documents ~/Documents.old

ln -s ~/Dropbox/Documents ~/Documents

Switching from iCloud to gmail

I get way too much email now and using the default apple service to manage email is hopeless. Both iPhone and OS X email applications suck at handling multiple email accounts or thousands of messages.

Since I’m using Google Apps on my own domain, the option to import email from your old email provider is not available so I’m using a tool called imapsync. According to google there is approximately a 500 megabyte per day IMAP transfer limit, however I found this not to exist, sending over 5GB of email over to google last night.

Now its jus ta matter of sorting out the hundreds of folders that were sent over and converted into google labels.

<update>

After using google apps gmail for a month or so now, i’m loving it. It works great. Only real thing lacking now is the ability to automatically PGP sign all outgoing emails without having to use thunderbird.

Gmail’s filters blow apple’s out of the water.

The search feature is fast and easy to use.

 

Transmission OSX + many many torrents (lets test it to the limit)

So lately I’ve been trying to find a torrent client that can handle thousands of torrents at a time and that is capable of downloading data in place of the .torrent file. For this I chose transmission for os x.

I have my music setup so that it’s

/media/Music/Aritst/<type>/Albums

So now I’m able to use the collector at various torrent sites to download entire artists .torrent files at once. Once they are in the proper place where they will permenantly reside, then, using the a search, I locate all .torrent files in my Music directory and drag/drop them over into Transmission.

Transmission then loads all the .torrent files (removing the .torrent at the same time), and begins downloading the data to the directory in which it found the .torrent.

Currently the setup is a Mac Mini (late 2009) w/8GB RAM and gigabit uplink to my network.

It’s been laggy as hell, I can’t even get it to respond long enough to tweak the settings.

Just for a little background the network is setup as such:

Internet Uplink -> pfSense router -> gigabit switch -> Servers/Computers

The data is all being redundantly stored on a Synology 1812+ NAS disk station in RAID-5 format.

Update 1: So eventually I found out that the weakest link currently was my internal HDD in the Mac Mini that I’m using for torrenting. Whenever there is heavy throughput to the internal HDD, the mini will freeze up. So instead of having the files cache in the ~/Downloads directory, I’m not going to have them cache, but instead just write directly to the fileserver.

Update 2: Writing directly to the fileserver doesn’t work because it’s pre-allocating all the files on the fileserver, and thus having to transfer the data for each downloaded torrent twice to the fileserver, slowing everyting way down. Instead I’ll try having the incoming data cache on an external USB 3.0 disk.

Update 3: Everything working much better now that files are being cached on the external USB disk. A .torrent is loaded, the files are pre-allocated locally on the USB drive, the transfer is completed, then at high speed (40-50MB/sec) the data is uploaded to the fileserver and removed from the USB HDD. When a torrent finishes and the data is being uploaded to the fileserver, the incoming transfer rate drops to nearly 0 (but this isn’t a big deal as it takes less than a minute for the data to be copied over and the downloads to resume). 

Update 4: Disabled the Web GUI to see if it makes things any quicker, seems to have amde things slightly faster.

Update 5: Set it up so that when a torrent finishes downloading, it completes after 1 minute instead of seeding. First lets get everything downloaded, then we’ll worry about seeding. Currently at 1500 torrents loaded simultaneously into transmission. Whenever a torrent finishes, Transmission lags up really bad while the data is being copied over to the fileserver. You cannot click anything or use the interface, and download stop downloading. While downloading you can navigate the interface as usual, though it is quite slow and choppy at times. For example, when switching between All, Active, Downloading, etc download lists, sometimes the full list won’t load (ill switch from Active to All, but the list still only displays my currently active transfers). Also wish there was a way to filter out only torrents that are still in progress (i.e. I don’t want to see all the torrents that are complete).

Update 6: Increased max simultaneous torrents to 40 and decreased the time it takes before a download is considered stalled to 1 minute. Peer is set to 80. Getting 12-13MB/sec download bursts, over 1TB transfer yesterday. Currently at ~3000 torrents loaded simultaneously into transmission.

Update 7: Increased max peers to 120 as there are many torrents running with only one peer and my downstream is no longer being saturated. It appears that Transmission will allow more than your max number of running downloads if it pauses a download because its stalled, then resumes it because it became available again. So even though I have my max concurrent downloads set to 40, there’s 80 torrents running right now. Currently at ~5000 torrents loaded simultaneously to transmission. Still saturating my downlink (though sporadically). Not going to load any more torrents until some of the currently loaded ones get fnished up. Switching views on the interface has become extremely slow, laggy, and seldom works corretly. I now have to restart Transmission every now and then if I want to add files or really check the status.

Update 8: Another day, nearly another TB transferred. Now running around 8k torrents, transmission still slow as hell.

status_rrd_graph_img

 

Here’s traffic for the last week. Note that these are averaged, thus you don’t see all the 12-13MB/sec bursts. When watching a live monitor, I see that it normally goes from 0 to 12MB then back to 0.

Synology 1812

So after a few days of deliberation, I finally decided to buy a Synology DS1812+

I received the unit a few days ago from amazon. I immediately loaded it up with 4x2TB and 4×1.5TB drives and configured it in a RAID-5 array (this way when I need more space I can swap the 4×1.5TB for 4x2TB drives).

It took the unit days to complete the parity check, but other than that setup has been a total breeze. I'm currently sending stuff over from a MacMini server I have at 50-80MB/sec (over single GigE).

ipkg

After a little research I found the easiiest way to add functionality to the underlying BusyBox linux operating system that the Synology 1812 runs is to add ipkg.

Once added I proceeded to install lftp, nano, as well as a couple other tools.

<pre> This is a test </pre>