ASCP AND NEXT GENERATION ASCP4
Aspera's ascp is the core file transfer binary that ships with every version of Aspera transfer client and server software to deliver the fastest possible transfer speeds regardless of file sizes, transfer distance, or network conditions. introduce a new architecture for ultra-high-speed transfer of small files in addition to large data sets.
Ascp4 is the next generation binary that ships alongside ascp to introduce a new architecture for ultra high-speed transfer of small files in addition to large data sets. Based on a new architecture with highly parallel I/O and per file metadata sent over the FASP channel, the architecture transfers one million (1MM) files per minute for even the smallest file sizes (e.g.
What's New in Ascp (ascp)
The FASP protocol and Aspera's core cross-platform transfer binary ascp at the center of the transfer stack has been extended to be faster, more secure, and more manageable than ever before.
- Substantially improved performance across all core transfer applications due to new FASP receiver architecture providing explicit parallelization of key operations in reading from the network, decryption and per message tracking. Unencrypted CPU-bound transfers are 100% faster and encrypted CPU-bound transfers are 20% faster.
- A new transfer scheme, faux://, enables testing a transfer without reading from disk and/or writing to disk.
- New clustered transfer option configurable through aspera.conf, command line, and Node API (with restart and resume from point of interruption) allows ascp to automatically split single large files or directories across a configurable number of concurrent sessions for transfers to cloud (object) or block storage. The splitting of the total data size is automatically load balanced across the sessions; performance scales nearly linearly. 20 Gigabit per second single transfer jobs are achievable against multi-node clusters, e.g. via the Aspera Transfer Service.
- Supports all generations and types of Aspera user authentication.
- External Authorization token using a Node API user and password, or an access token.
- HTTPS Basic Authorization using the Node API user and password.
- OAuth/JWT Bearer token Authorization using an access key and secret (master and sub keys supported).
- Reporting of the authenticated user through session tags for Console.
- Multiple SSH private keys (both DSA & RSA) are supported in the same ascp command.
- SSH private key strings can be passed to ascp in a new environment variable, ASPERA_SCP_KEY.
- OpenSSL upgraded for security enhancements (from 1.0.2i to 1.0.2j).
- Setuid and setgid are now supported in ascp and the Node API when the permissions are configured in aspera.conf.
- Resume mode checks in ascp now default to use a SHA-2-based checksum algorithm with configurable alternatives in aspera.conf.
- New complete include and exclude filter options support glob and regular expression filter of source paths, equivalent to include/exclude options of async.
- When using a local URI docroot or destination (such as file:////data), e.g. server-side encryption-at-rest, temporary partial files can now be created during ascp transfers by configuring <partial_file_suffix> in aspera.conf.
- New FASP Manager API sends an event when individual items in persistent sessions complete transfer.
- New ascp options to indicate source prefix or destination path are base64 encoded: --dest64 and --source-prefix64.
- New included 'aclean' tool supports automatic scan and delete of files and directories based on aging policies.
- Experimental in 3.7 - Ascp now includes a new asynchronous "in transfer" post-processing engine with Lua scripts. A Lua interpreter engine is packaged with Enterprise Server, allowing for post-processing, validation, and authorization functions through embedded Lua scripts. Scripts execute asynchronously, and their progress is reported through a new validating state in the transfer session, without delaying completion of the file transfer or slowing the transfer pipeline.
GUI & General
- New Connection endpoints for Aspera Transfer Service subscriptions (AWS, IBM SoftLayer, IBM Bluemix, MS Azure beta, Google upcoming).
- New option for Hot Folders in pull mode, Transfer source directory contents only, downloads only the contents of a directory and not the directory itself, stripping the prefix from the filepath of each source file.
- New storage options include BlueMix S3 storage and Azure Page Blobs.
- New support for Hot Folder transfers to Azure Blob storage.
- Connection credentials now use the native operating system key store for secret management.
- New support for alee usage reporting on Windows Enterprise Server (for usage entitlements).
- Full support for Windows Server 2016 and AIX.
- Better handling of cases where the connection from the proxy to the Aspera server host times out.
- For proxies using DNS names, with port-reuse disabled, the FASP TCP and UDP sessions are correctly routed to the same target host.
- New bind source address option for aspera.conf.
What's New in ascp4
Many new features have been added to enable ascp4 with comprehensive file transfer, management and streaming functionality.
- Options to parallelize all file operations with a configurable number of threads for directory traversal and file attribute checking, as well as read/write
- Now supported on OS X, as well as Windows and Linux
- New FASP Streaming Support for TCP (in addition to UDP) stream providers, allowing ascp4 'tunnel' any local streaming source over the FASP protocol over the WAN
- Supports a huge range of use cases to carry live data feeds and video over WANs using FASP:
- Stream to stream - Sender side ascp4 listens to a port or connects to an ipaddr:port and transfers TCP packets through a FASP tunnel. The receiver side ascp4 connects to an ipaddr:port to output the data
- Stream to file - Sender side ascp4 listens to a port or connects to an ipaddr:port in and transfers TCP packets through a FASP tunnel. Receiver side ascp4 writes TCP data to a file
- File to stream - Sender side ascp4 reads data from a file and transfers data through a FASP tunnel. Receiver side ascp4 makes a TCP connection to an ipaddr:port and writes receiving data to that socket
- For live video supports input/output over source specific multicast as well as standard multicast and unicast UDP, and variable sized UDP datagrams
- For live data, new Kafka provider to send directly via the TCP streaming interface (upcoming)
Transfers to Object (S3-Compatible) Storage
- Faster performance due to optimizations in the per file attribute checking and directory creation used in resume and skip of existing files
- Easier deployment with configurable chunk size options and no storage root options supported in aspera.conf
- Supports transfers of empty directories
File Handling & Security
- New --delete-before and --delete-after transfer options deletes extraneous files from the destination storage
- Now supports transfers to Windows UNC paths in the storage roots
- New include and exclude options for specifying patterns of pathnames to include or exclude
- Expanded resume and overwrite support
- All file resume, skip, and overwrite options of standard ascp, including the following:-k0, -k1, -k2, etc.
- Always overwrite; overwrite if newer and different; overwrite if newer
- Resume based on attributes, sparse checksum, and full checksum
- Preservation of file owner uid and gid (--preserve-file-owner-uid, --preserve-file-owner-gid)
- Transfer of symlinks via skip, follow and copy policies
- Configurable log size options for easy management
- Support for token authentication (upcoming)
- Ascp4 support for persistent session
Management and Reporting
- Complete support for FASP management APIs including cancel actions and correct reporting of transfer direction, skip offset, and session errors
- Many enhancements and corrections for complete support for Aspera Console and GUI management
- Streaming sessions fully reported through FASP management