Using Tsunami to Upload Files to Ec2
Tsunami UDP Protocol
Tsunami UDP Protocol: A fast user-space file transfer protocol that uses TCP control and UDP data for transfer over very high speed long distance networks (≥ 1 Gbps and even ten GE), designed to provide more throughput than possible with TCP over the aforementioned networks. The projection is based on original Indiana University 2002 Seismic sea wave source code, merely has been significantly improved and extended. As such, large portions of the program today are courtesy past Aalto Academy Metsähovi Radio Observatory.
Includes FTP-like client and server command line applications for normal file transfers. Information technology has additionally been extended for high charge per unit existent-time data streaming in eVLBI radio astronomy and geodesy (VSIB, PCEVN DAQ). Licensed under the original IU open up source license.
Updates
Version i.1 cvsbuild xl has automated achieved rate memorization and several changes for smoother transfers. Beginning with version 1.i cvsbuild 29 Seismic sea wave at present (09/2007) supports transfers betwixt unlike-endian machines. List shared files with dir is supported since cvsbuild 27. Mac Bone 10 / Darwin works starting time with cvsuild 36. For more changes, see the changelog.
The previous CVS update was on 22 December 2009.
The code was migrated from CVS to Git in July 2019.
Background
The original early Tsunami UDP transfer protocol developed and released to the public in 2002 past Marker Meiss et al. at the Pervasive Technology Labs at Indiana University (IU) can still exist found at http://anml.iu.edu/research.shtml?prim=lab_research.
This project is a derived piece of work - if not a complete rewrite - of the original IU source code. The new Seismic sea wave includes a large number of problems fixes, improvements and new features. File releases in the form of compressed source code archives tin can be downloaded from the projection page at http://sourceforge.net/projects/seismic sea wave-udp, but I recommend you'd use the CVS version.
Source code access
Currently no binaries are distributed, just the source code is available and should be compiled on the target system. It should compile fine on all POSIX systems and on GNU/Linux, no special libraries or toolkits are required. Microsoft Windows and Vista are non supported (XP offers an optional POSIX layer, merely some parts of Tsunami code would accept to exist slightly adapted neverheless, and unfortunately in Vista Microsoft has abandoned POSIX altogether).
Whatsoever current version of source code pulled from Git master will compile and work, and has been tested somewhat extensively. The Git version is likely to contain new features. There are no guarantees any new features are going to work correctly with older versions of the Tsunami client/server. Nonetheless, in the basic features, Seismic sea wave generally is backwards uniform.
If you want to get a non-changing stable release version, you may check out the following Git tags (former CVS tags): v10_RTM_protorev20061025 (v1.0 released 31. Nov 2006), v11_build11_protorev20061025 (v1.1 build 11 released 5. Jan 2007), v11_build16_protorev20061025 (v1.1 build 16 released 24. May 2007), v11_build29_protorev20061025 (v1.1 build 29 released 29. Aug 2007), v11_build36_protorev20061025 (v1.1 build 36 released 26. May 2008). At that place are other tags, delight run into ViewCV. Alternatively, some probably significantly less up to engagement source code .tar.gz files can be downloaded from file releases.
| Source code and compile cheatsheet | |
|---|---|
| | git clone https://git.code.sf.net/p/tsunami-udp/code_git seismic sea wave-udp-code_git (and for a detail tag then e.g. 'git checkout v11_build29_protorev20061025') |
| | https://sourceforge.internet/p/tsunami-udp/code_git/ci/master/tree/seismic sea wave-udp/ |
| | https://sourceforge.internet/p/tsunami-udp/code_git/ci/master/tree/docs/ |
| | COMPILING.txt |
| | ChangeLog |
| Users Guide, Troubleshooting and Realtime | |
| | USAGE.txt including Troubleshooting |
| | howTsunamiWorks.txt |
09Nov07: Those with a sluggish Petabit/sec network connection ;) can try the protocol version ane.ii in CVS branch protocol_v12_branch. Yet unstable and experimental. The set charge per unit value in v1.1 has simply 32 bits and thus limits the user-settable rate to 4 Gbps. Non then in v1.2. Checkout like in a higher place but with tag
. Note that v1.2 can't connect to v1.ane protocol version software and vice versa. After goals are to add blocking send_data() and receive_data() API functions to make Tsunami less a file and more than a information transfer protocol.
Hardware requirements
Here is a curt summary of vague hardware requirements. It's nothing fancy apart from the RAID-0 deejay requirement.- Consumer PC motherboards are highly suitable
- Deejay: to achieve loftier transfer rates, disk I/O must be adequately fast. Information technology is essential to use fast RAID, due east.g. RAID0 with a large chunk size setting such as 512 kB. Linux software RAID using mdadm performs well enough - typical performance with iv SATA disks is around 1.8 Gbit/s (05/2007). You lot tin check your disk throughput with
hdparm -t /dev/md0 , it should exist 100 MB/s or better. - CPU: the CPU should be Pentium4 or better. Consumer multi-cadre CPUs are quite prissy.
- RAM: the client program requires upwards to 128MB of free retentiveness for internal buffers, and swapping has to be avoided at all price, thus at least 512MB of RAM is strongly recommended.
- NIC: whatever motherboard 1 Gbit/s Ethernet is usually sufficiently fast. Server grade NICs are completely unnecessary. Usual x Gbit/southward PCIe NICs can exist used, too.
- Network: the network connection should exist faster than the typical home broadband connection, for example a 630 Mbit/s or 1/10/40/100 Gbit/s fibre connection. Colossal frames are not strictly required, but they can heave functioning if y'all want to utilize rates higher than v Gbit/s. Lightpaths, guaranteed bandwidth QoS services etc are a matter of debate, my take is to simply consider them after (if ever) network people start lament about your traffic (and then first try the 'you lot sold united states of america a ten Gbit/southward connection so we darn well are going to transfer at x Gbit/south' route ;-).
Typical transfer rate examples
Below are a few examples of transfer rates that Seismic sea wave tin can achieve over the Cyberspace. On a local LAN, for disk-to-deejay transfers rates about the limit of the PC NIC or on-board LAN are easily achieved, which is commonly around 940 Mbps for 1G LAN. This is provided that the disk organisation in both PCs is adequately fast, east.g. simple software RAID-0 with three or four SATA disks (no RAID controller necessary).
Figure one - international transfer over the Cyberspace, averaging 800 Mbit/second
Effigy 1 shows a throughput plot of a deejay-to-disk transfer where several multi-gigabyte files are transferred one later on another from Helsinki, Finland to Bonn, Federal republic of germany. Both PCs have a fast (>1 Gbps capable) RAID-0. Linux software RAID-0 with 4 SATA disks on the sending side, with a Dell Optiplex GX620 cheap PC. SATA RAID card at the receiving side. The server side had 10 Gbps fibre to the Finnish 'Funet' GEANT2 NREN, the client side had a shared fibre to the Viola GEANT2 NREN with ~1 Gbps allocated capacity.
Figure two - a curt 20GB file transferred with Tsunami v1.two over local 10 Gbps LAN at Metsähovi.
Effigy 2 shows a Tsunami v1.2 examination transfer on the local 10G LAN at Metsähovi. RAID-0 were built with Samsung SpinPoint F1 disks on dual and quad-core consumer figurer systems with Chelsio 10G cards connected via a HP6400cl switch. The transfer rate is limited to 4.3 Gbps by disk I/O. With diskless transfer a rate of 7 Gbps could be achieved.
Source: http://tsunami-udp.sourceforge.net/
Post a Comment for "Using Tsunami to Upload Files to Ec2"