Welcome to EverybodyWiki ! Nuvola apps kgpg.png Sign in or create an account to improve, watchlist or create an article like a company page or a bio (yours ?)...

Rclone

From EverybodyWiki Bios & Wiki
Jump to navigation Jump to search


Rclone
Original author(s)Nick Craig-Wood
Developer(s)Nick Craig-Wood
Initial releaseJune 19, 2012; 8 years ago (2012-06-19)[1]
Stable release
1.52.2[1] / 24 June 2020; 41 days ago (2020-06-24)[1]
Repositorygithub.com/rclone/rclone
Written inGo[2]
Engine
    Operating systemLinux, Windows, macOS, FreeBSD, NetBSD, OpenBSD, Plan9, Solaris[2]
    PlatformIntel/AMD-64, Intel/AMD-32, ARM-32, ARM-64, MIPS-Big-Endian, MIPS-Little-Endian[2]
    LicenseMIT
    Websiterclone.org

    Amazon.com Logo.png Search Rclone on Amazon.

    Rclone is an open source, command line computer program to manage files on cloud storage. Its capabilities include sync, transfer, encrypt, cache, union and mount. The rclone website lists fifty supported backends including S3 services and consumer file sharing platforms.[2]

    Descriptions of rclone often carry the strapline Rclone syncs your files to cloud storage.[2] Those prior to 2020 include the alternative Rsync for Cloud Storage.[3] Users have called rclone The Swiss Army Knife of cloud storage,[4] and likened it to magic.[5]

    Whilst rclone is well known for its rlone sync and rclone mount commands,[6] it provides further file management functions analogous to those ordinarily used for local disks.

    Rclone is widely used at hpc research sites to transfer large datasets. Examples include the University of Maryland,[7] Iowa State University,[8] University of Utah,[9] National Institutes of Health,[10] Wharton,[11] Yale,[12] Minnesota,[13] Michigan State,[14] and Berkeley.[15]

    Official Ubuntu, Debian, Fedora, Brew, Chocolatey, and other package managers include rclone.[16]

    History[edit]

    Nick Craig-Wood was inspired by rsync.[3] His wife's concerns about the noise and power costs arising from their home computer servers prompted him to begin developing rclone[17] in 2012.[1]

    Rclone was promoted to stable version 1.00 in July 2014.[1]

    In May 2017 Amazon barred new rclone users from its consumer Amazon Drive data storage product. Amazon Drive had been advertised as offering unlimited storage for £55 per year. Amazon blamed security concerns and also banned other upload utilities.[18] Amazon's more expensive AWS S3 service continues to support new rclone users.

    The original rclone logo was retired to be replaced with the present one in September 2018.[19]

    Amazon's AWS April 2020 public sector blog explained how the Fred Hutch Cancer Research Center were using rclone in their Motuz tool to migrate very large biomedical research datasets in and out of AWS S3 object stores.[20]

    May 2020 reports stated rclone had been used by hackers that attacked Diebold Nixdorf ATMs.[21][22]

    Backends[edit]

    The following services are supported as rclone backends. There are others, built on standard protocols such as WebDAV or S3, that work.[2]

    1Fichier | Alibaba (Aliyun) Object Storage System (OSS) | Amazon Drive (See note) | Amazon S3 | Backblaze B2 | Box | Ceph | Citrix ShareFile | C14 | DigitalOcean Spaces | Dreamhost | Dropbox | FTP | Google Cloud Storage | Google Drive | Google Photos | HTTP | Hubic | Jottacloud | IBM COS S3 | Koofr | Mail.ru Cloud | Memset Memstore | Mega | Microsoft Azure Blob Storage | Microsoft OneDrive | Minio | Nextcloud | OVH | OpenDrive | OpenStack Swift | Oracle Cloud Storage | ownCloud | pCloud | premiumize.me | put.io | QingStor | Rackspace Cloud Files | rsync.net | Scaleway | Seafile | SFTP | StackPath | SugarSync | Tardigrade | Wasabi | WebDAV | Yandex Disk

    One, or more, remotes are defined from these backends, local disk, or memory (as S3), with rclone config. Rclone can further layer those remotes through virtual, encrypted, chunked, cached, vfs-cached or union backends.

    Once defined, the remotes are referenced by other rclone commands interchangeably with the local drive. Remote names are followed by a colon to distinguish them from local drives. For example, a remote example_remote containing a folder, or pseudofolder, myfolder is referred to within a command as example_remote:/myfolder.[23] Many rclone commands refer to directories, not files.[24]

    Rclone directly addresses remotes, or mounts them, for other programs, as if a conventional disk. It also serves remotes over SFTP, HTTP, WebDAV, FTP and DLNA.[2]

    Commands[edit]

    Usage:

     rclone [command] [flags]
    

    Available Commands:[25]

     about           Get quota information from the remote.
     authorize       Remote authorisation.
     cachestats      Print cache stats for a remote
     cat             Concatenates any files and sends them to stdout.
     check           Checks the files in the source and destination match.
     cleanup         Clean up the remote if possible
     config          Enter an interactive configuration session.
     copy            Copy files from source to dest, skipping already copied
     copyto          Copy files from source to dest, skipping already copied
     copyurl         Copy url content to dest.
     cryptcheck      Cryptcheck checks the integrity of a crypted remote.
     cryptdecode     Cryptdecode returns unencrypted file names.
     dbhashsum       Produces a Dropbox hash file for all the objects in the path.
     dedupe          Interactively find duplicate files and delete/rename them.
     delete          Remove the contents of path.
     deletefile      Remove a single file from remote.
     genautocomplete Output completion script for a given shell.
     gendocs         Output markdown docs for rclone to the directory supplied.
     hashsum         Produces an hashsum file for all the objects in the path.
     help            Show help for rclone commands, flags and backends.
     link            Generate public link to file/folder.
     listremotes     List all the remotes in the config file.
     ls              List the objects in the path with size and path.
     lsd             List all directories/containers/buckets in the path.
     lsf             List directories and objects in remote:path formatted for parsing
     lsjson          List directories and objects in the path in JSON format.
     lsl             List the objects in path with modification time, size and path.
     md5sum          Produces an md5sum file for all the objects in the path.
     mkdir           Make the path if it doesn't already exist.
     mount           Mount the remote as file system on a mountpoint.
     move            Move files from source to dest.
     moveto          Move file or directory from source to dest.
     ncdu            Explore a remote with a text based user interface.
     obscure         Obscure password for use in the rclone.conf
     purge           Remove the path and all of its contents.
     rc              Run a command against a running rclone.
     rcat            Copies standard input to file on remote.
     rcd             Run rclone listening to remote control commands only.
     rmdir           Remove the path if empty.
     rmdirs          Remove empty directories under the path.
     serve           Serve a remote over a protocol.
     settier         Changes storage class/tier of objects in remote.
     sha1sum         Produces an sha1sum file for all the objects in the path.
     size            Prints the total size and number of objects in remote:path.
     sync            Make source and dest identical, modifying destination only.
     touch           Create new file or change file modification time.
     tree            List the contents of the remote in a tree like fashion.
     version         Show the version number.
    

    Academic evaluation[edit]

    In 2018, University of Kentucky researchers published a conference paper comparing use of rclone and other command line, cloud data transfer agents for big data.[26] The paper was published as a result of funding by the National Science Foundation.[27]

    In 2018, the University of Utah Center for High Performance Computing examined the impact of rclone options on data transfer rates.[28]

    Future[edit]

    Rclone can serve a beta, web based remote control, graphical interface - rclone rc[29].

    An rclone virtual, compression backend is under development to address small file, and compressible file format, transfer capacity.[30]

    There have been persistent calls for a --magic flag to improve rclone.[31][32][33][34][35]

    Rclone and rsync[edit]

    Rsync transfers files between computers it is installed on. Rsync operates at the block level and has a delta algorithm so that it only needs to transfer changes in files. Rclone has a wider range of file management capabilities, and types of backend it can address, but only works at a whole file/object level.[36] Rclone decides which of those whole files to transfer after obtaining checksums, to compare, from the remote server. Where checksums are not available, rclone can use file size and time.[37]

    Eponymous cloud storage service rsync.net provides remote unix filesystems so that customers can run rsync and other standard unix tools.[38] They also offer rclone only accounts.[39]

    See also[edit]


    Other articles of the topic Free and open-source software : Free Silicon Foundation, Lemmy (software), Conectiva, VPN.ht, YXA, Shampoo (software), The Bug Genie
    Some use of "" in your query was not closed by a matching "".Some use of "" in your query was not closed by a matching "".

    • Cyberduck
    • Rsync
    • Transmit (file transfer tool)

    References[edit]

    1. 1.0 1.1 1.2 1.3 1.4 "Changelog". Rclone. Retrieved July 29, 2020.
    2. 2.0 2.1 2.2 2.3 2.4 2.5 2.6 "Rclone". Rclone. Archived from the original on June 17, 2020. Retrieved July 29, 2020. Unknown parameter |url-status= ignored (help)
    3. 3.0 3.1 "rclone/rclone". July 31, 2020. Archived from the original on June 30, 2020. Retrieved July 29, 2020 – via GitHub. Unknown parameter |url-status= ignored (help)
    4. "Issues with special characters in filenames". Rclone Forum. June 6, 2020. Retrieved July 29, 2020.
    5. "Stopping and starting rclone mid transfer". Rclone Forum. October 11, 2016. Retrieved July 29, 2020.
    6. "Sync Microsoft OneDrive in Linux With Rclone [Step by Step]". It's FOSS. June 18, 2020. Archived from the original on June 14, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    7. "HPC Software Rclone". hpcc.umd.edu. University of Maryland. Archived from the original on October 24, 2019. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    8. "Rclone - High Performance Computing". hpc.iastate.edu. Iowa State University.
    9. "rclone - Center for High Performance Computing - The University of Utah". chpc.utah.edu. The University of Utah.
    10. "Rclone on Helix and Biowulf". hpc.nih.gov. National Institute of Health. Archived from the original on January 8, 2017. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    11. "Rclone". research-it.wharton.upenn.edu. University of Pennsylvania. Archived from the original on December 18, 2019. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    12. "Rclone - Yale Center for Research Computing". docs.ycrc.yale.edu. Yale University. Archived from the original on May 16, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    13. "rclone - The Minnesota Supercomputing Institute". msi.umn.edu. University of Minnesota. Retrieved August 1, 2020.
    14. "Rclone on HPCC - Institute for Cyber-Enabled Research - Michigan State University". icer.msu.edu. Michigan State University. Retrieved August 1, 2020.
    15. "Transferring Data Between Savio and Your bDrive (Google Drive) Account". research-it.berkeley.edu. University of California, Berkeley. Archived from the original on April 13, 2019. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    16. "rclone package versions - Repology". Repology.
    17. "Is rclone more efficient than an nfs mount". Rclone Forum. June 10, 2020. Retrieved July 29, 2020.
    18. "Amazon Drive bans rclone storage client". www.theregister.com.
    19. "A New Logo for rclone". rclone forum. September 30, 2018.
    20. "How Fred Hutch unlocks siloed data with AWS and open-source software". Amazon Web Services. April 16, 2020. Archived from the original on July 12, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    21. "Diebold Nixdorf ATM attack by ProLock ransomware used QakBot trojan to access networks". Archived from the original on June 9, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    22. "ProLock Ransomware teams up with QakBot trojan for network access". BleepingComputer. Archived from the original on June 13, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    23. McKay, Dave. "How to Use rclone to Back Up to Google Drive on Linux". How-To Geek.
    24. "rclone copy". rclone.org. Archived from the original on July 28, 2020. Retrieved July 30, 2020. Unknown parameter |url-status= ignored (help)
    25. "Commands". rclone.org.
    26. "Navigating the Unexpected Realities of Big Data Transfers in a Cloud-based World" (PDF). par.nsf.gov. National Science Foundation. Retrieved August 1, 2020. Unknown parameter |url-status= ignored (help)
    27. "NSF Award Search: Award#1541380 - CC*DNI Networking Infrastructure: An Software Defined Networking-Enabled Research Infrastructure". National Science Foundation. May 9, 2018. Archived from the original on September 28, 2018. Retrieved August 1, 2020. Unknown parameter |url-status= ignored (help)
    28. "rclone - Center for High Performance Computing - The University of Utah". chpc.utah.edu. The University of Utah. Retrieved August 1, 2020.
    29. "GUI". rclone.org.
    30. "Beta testers wanted for new experimental compression remote". rclone forum. March 7, 2020.
    31. "Rclone 1.52 release". rclone forum. May 27, 2020.
    32. "Mkdir on b2: consider adding some output?". rclone forum. July 6, 2020.
    33. "Copy Local Files to Google Drive TD With --drive-server-side-across-configs". rclone forum. May 25, 2020.
    34. "Google Drive very slow and way below quota". rclone forum. July 21, 2020.
    35. "[Total newbie] Question about rclone cloud to cloud copy speeds". rclone forum. July 2, 2020.
    36. "Difference between Rclone and Rsync when syncing to the Local Filesystem?". rclone forum. July 9, 2017.
    37. "SFTP". rclone.org.
    38. "rsync.net Cloud Storage for Offsite Backups". www.rsync.net.
    39. "Cloud Storage for Offsite Backups - rclone support". www.rsync.net.

    External links[edit]


    This article "Rclone" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Rclone. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.


    Compte Twitter EverybodyWiki Follow us on https://twitter.com/EverybodyWiki !

    Farm-Fresh comment add.png You have to Sign in or create an account to comment this article !