New Aspera WatchFolder Service
The new breakthrough Aspera WatchFolder service is designed to power next generation, large scale, automated file and directory transfers including ultra large directories with 10 million items+ and "growing file" sources. Collections of files and folders can now be grouped into a single “drop” and transferred to remote nodes as a single logical unit. Operators can specify the cool-off period before initiating the transfer and any local or remote post-transfer processing steps to be executed once the entire set of files in the drop has been transferred, and enable the transfer of "growing" (in progress) files. A new RESTful API enables complete programmatic control for customized and automated transfer processing. The new WatchFolder service is available for Windows and Linux in the core server products in version 3.6.1 and manageable through Console 3.0.
What's New in watchd and watchfolderd
The new architecture supports watchfolder automation for directories of 10s of millions of files and scales out to support many concurrent watchfolders per host and has a complete REST API for integration, reporting and management. Aspera introduced full support for watchfolder management in Console with Enterprise Server 3.7 and will bring it into the Desktop GUI in v3.8 with several exciting new capabilities:
- Displays all configured watchfolders including local source, remote host and destination and current status including transfer Activity, Drop events, and Files status
- Activity View includes summary of the current activity of each watchfolder including bytes transferred, files transferred, skipped, failed and any outstanding drop events
- Drops View includes the Drop events for the currently selected Watchfolder and their status (pending, transferring, completed, etc.)
- Files View includes a searchable view of the status of all files processed by the Watchfolder, searchable by status (transferring, completed, failed) and detailed error information to allow users to easily track the progress of individual files
- Admins can configure the criteria for Watchfolder Drops, the "trigger events" that define a "unit" of files and directories to be processed together as a single change
- Criteria include cooloff period (time to wait for new changes) and top level directories under which changes are considered
FLEXIBLE SERVICE MANAGEMENT AND ACCESS CONTROL
- System admins can create new watchfolder service instances and configure the system accounts they run under, and define access control policies allowing named watchfolder admins to manage watchfolders under each instance
- Watchfolder admins create new watchfolders within each instance through a new Watchfolders tab in the GUI
- End users see and access watchfolders defined for their user account
- Services Status Reporting tab displays the running status and current service account for all watchfolder and watchd instances
The watchfolderd and watchd services also have many new capabilities in version 3.7:
- New, smoother transfer of growing files
- New maxcool option to prevent spin loops when files are continuously changing
- New support for token authentication, encryption at rest, session cookies, tags, preservation of timestamps
- New <raw_options> capability in aspera.conf allows passing arbitrary options to ascp
- Various robustness improvements
- Gracefully handles cases where sessions fail to start within configurable timeouts
- Handles drop events with very large numbers of files (100,000+)
- Configuration preserved and editable after service restart
- Robust deletion of watchfolders and new options to delete remaining keys after removing watchd instances
- Improved handling of multiple include and exclude filters
- Now supported on AIXFaster directory scanning by the Aspera Watch Service, particularly of directories that contain many (10,000s) subdirectories (3.7.4)
- Support for IPv6 addresses (3.7.4)
- Enables deletion of files from the source as soon as the file is successfully transferred, rather than waiting for the session to complete (3.7.4)
What's New in Watchfolder 3.6
- New cross-platform directory watchfolder service for Linux, Windows.
- Built on asperawatchd, a new file system notification service designed for speed, scale and distributed change watching.
- Supports advanced workflow capabilities including:
- Transfers of growing ("in progress") files with full integrity. Number of growing items to send in parallel is configurable.
- Packaging capability allows operators and workflows to automatically recognize "drops" of groups of files and directories, and hold transfer until a group of named files and directories are available and allows for controlling the order in which files are sent, such as a "go last" file.
- Advanced regular expression filters allow for filtering source file and directory pathnames.
- API allows for observing drop events for convenient reporting and notification on user intention, summary of status (files and bytes transferred) and configuration of new watchfolders.
- Configurable and manageable from Aspera Console 3.0.