snakemake.remote package

Submodules

snakemake.remote.EGA module

class snakemake.remote.EGA.EGAFile(dataset, path)

Bases: tuple

dataset

Alias for field number 0

path

Alias for field number 1

class snakemake.remote.EGA.EGAFileInfo(size, status, id, checksum)

Bases: tuple

checksum

Alias for field number 3

id

Alias for field number 2

size

Alias for field number 0

status

Alias for field number 1

class snakemake.remote.EGA.RemoteObject(*args, protocol=None, keep_local=False, stay_on_remote=False, provider=None, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteObject

download()[source]
exists()[source]
mtime()[source]
property parts
size()[source]
class snakemake.remote.EGA.RemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, retry=5, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteProvider

api_request(url_suffix, url_prefix='https://ega.ebi.ac.uk:8051/elixir/', json=True, post=False, **params)[source]

Make an API request.

Parameters
property available_protocols

List of valid protocols for this remote provider.

property default_protocol

The protocol that is prepended to the path when no protocol is specified.

get_files(dataset)[source]
property token

snakemake.remote.FTP module

snakemake.remote.GS module

snakemake.remote.HTTP module

class snakemake.remote.HTTP.RemoteObject(*args, keep_local=False, provider=None, additional_request_string='', allow_redirects=True, **kwargs)[source]

Bases: snakemake.remote.DomainObject

This is a class to interact with an HTTP server.

download(make_dest_dirs=True)[source]
exists()[source]
get_header_item(httpr, header_name, default)[source]

Since HTTP header capitalization may differ, this returns a header value regardless of case

httpr(verb='GET', stream=False)[source]
property list
mtime()[source]
size()[source]
upload()[source]
class snakemake.remote.HTTP.RemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteProvider

property available_protocols

List of valid protocols for this remote provider.

property default_protocol

The protocol that is prepended to the path when no protocol is specified.

remote(value, *args, insecure=None, **kwargs)[source]

snakemake.remote.NCBI module

class snakemake.remote.NCBI.NCBIHelper(*args, email=None, **kwargs)[source]

Bases: object

dbs_for_options(file_ext, rettype=None, retmode=None)[source]
exists(accession, db='nuccore')[source]
fetch_from_ncbi(accession_list, destination_dir, force_overwrite=False, rettype='fasta', retmode='text', file_ext=None, combined_file_prefix=None, remove_separate_files=False, chunk_size=1, db='nuccore', **kwargs)[source]

This function downloads and saves files from NCBI. Adapted in part from the BSD-licensed code here:

guess_db_options_for_extension(file_ext, db=None, rettype=None, retmode=None)[source]
is_valid_db(db)[source]
is_valid_db_request(db, rettype, retmode)[source]
mtime(accession, db='nuccore')[source]
options_for_db_and_extension(db, file_ext, rettype=None, retmode=None)[source]
parse_accession_str(id_str)[source]
This tries to match an NCBI accession as defined here:

https://www.ncbi.nlm.nih.gov/Sequin/acc.html

search(query, *args, db='nuccore', idtype='acc', **kwargs)[source]
size(accession, db='nuccore')[source]
property valid_dbs
property valid_extensions
class snakemake.remote.NCBI.RemoteObject(*args, keep_local=False, stay_on_remote=False, provider=None, email=None, db=None, rettype=None, retmode=None, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteObject

This is a class to interact with NCBI / GenBank.

property accession
download()[source]
exists()[source]
property file_ext
property list
mtime()[source]
size()[source]
upload()[source]
property version
class snakemake.remote.NCBI.RemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, email=None, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteProvider

property available_protocols

List of valid protocols for this remote provider.

property default_protocol

The protocol that is prepended to the path when no protocol is specified.

remote_interface()[source]
search(query, *args, db='nuccore', idtype='acc', retmode='json', **kwargs)[source]

snakemake.remote.S3 module

class snakemake.remote.S3.RemoteObject(*args, keep_local=False, provider=None, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteObject

This is a class to interact with the AWS S3 object store.

download()[source]
exists()[source]
property list
mtime()[source]
property name
property s3_bucket
s3_create_stub()[source]
property s3_key
size()[source]
upload()[source]
class snakemake.remote.S3.RemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteProvider

property available_protocols

List of valid protocols for this remote provider.

property default_protocol

The protocol that is prepended to the path when no protocol is specified.

remote_interface()[source]
supports_default = True
class snakemake.remote.S3.S3Helper(*args, **kwargs)[source]

Bases: object

bucket_exists(bucket_name)[source]
delete_from_bucket(bucket_name, key)[source]

Delete a file from s3

This function deletes an object from a specified AWS S3 bucket.

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • key – the key of the object to delete from the bucket

Returns

The name of the object deleted

download_from_s3(bucket_name, key, destination_path=None, expandKeyIntoDirs=True, make_dest_dirs=True, create_stub_only=False)[source]

Download a file from s3

This function downloads an object from a specified AWS S3 bucket.

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • destination_path – If specified, the file will be saved to this path, otherwise cwd.

  • expandKeyIntoDirs – Since S3 keys can include slashes, if this is True (defult) then S3 keys with slashes are expanded into directories on the receiving end. If it is False, the key is passed to os.path.basename() to get the substring following the last slash.

  • make_dest_dirs – If this is True (default) and the destination path includes directories that do not exist, they will be created.

Returns

The destination path of the downloaded file on the receiving end, or None if the destination_path could not be downloaded

exists_in_bucket(bucket_name, key)[source]

Returns whether the key exists in the bucket

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • key – the key of the object to delete from the bucket

Returns

True | False

key_last_modified(bucket_name, key)[source]

Returns a timestamp of a key based on a HEAD request

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • key – the key of the object to delete from the bucket

Returns

timestamp

key_size(bucket_name, key)[source]

Returns the size of a key based on a HEAD request

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • key – the key of the object to delete from the bucket

Returns

Size in kb

list_keys(bucket_name)[source]
upload_to_s3(bucket_name, file_path, key=None, use_relative_path_for_key=True, relative_start_dir=None, extra_args=None, config=None)[source]

Upload a file to S3

This function uploads a file to an AWS S3 bucket.

Parameters
  • bucket_name – the name of the S3 bucket to use (bucket name only, not ARN)

  • file_path – The path to the file to upload.

  • key – The key to set for the file on S3. If not specified, this will default to the name of the file.

  • use_relative_path_for_key – If set to True (default), and key is None, the S3 key will include slashes representing the path of the file relative to the CWD. If False only the file basename will be used for the key.

  • relative_start_dir – The start dir to use for use_relative_path_for_key. No effect if key is set.

Returns: The key of the file on S3 if written, None otherwise

snakemake.remote.S3Mocked module

snakemake.remote.SFTP module

snakemake.remote.XRootD module

snakemake.remote.dropbox module

class snakemake.remote.dropbox.RemoteObject(*args, keep_local=False, provider=None, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteObject

This is a class to interact with the Dropbox API.

download(make_dest_dirs=True)[source]
dropbox_file()[source]
exists()[source]
property list
mtime()[source]
property name
size()[source]
upload(mode=WriteMode('overwrite', None))[source]
class snakemake.remote.dropbox.RemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteProvider

property available_protocols

List of valid protocols for this remote provider.

property default_protocol

The protocol that is prepended to the path when no protocol is specified.

remote_interface()[source]

snakemake.remote.gfal module

snakemake.remote.gridftp module

snakemake.remote.iRODS module

snakemake.remote.webdav module

Module contents

class snakemake.remote.AbstractRemoteObject(*args, protocol=None, keep_local=False, stay_on_remote=False, provider=None, **kwargs)[source]

Bases: object

This is an abstract class to be used to derive remote object classes for different cloud storage providers. For example, there could be classes for interacting with Amazon AWS S3 and Google Cloud Storage, both derived from this common base class.

abstract close()[source]
abstract download(*args, **kwargs)[source]
abstract exists()[source]
file()[source]
inventory(cache: snakemake.io.IOCache)[source]

From this file, try to find as much existence and modification date information as possible.

abstract list(*args, **kwargs)[source]
local_file()[source]
local_touch_or_create()[source]
abstract mtime()[source]
abstract name(*args, **kwargs)[source]
abstract remote(value, keep_local=False, stay_on_remote=False)[source]
remote_file()[source]
abstract remove()[source]
abstract size()[source]
abstract upload(*args, **kwargs)[source]
class snakemake.remote.AbstractRemoteProvider(*args, keep_local=False, stay_on_remote=False, is_default=False, **kwargs)[source]

Bases: object

This is an abstract class to be used to derive remote provider classes. These might be used to hold common credentials, and are then passed to RemoteObjects.

allows_directories = False
abstract available_protocols()[source]

List of valid protocols for this remote provider.

abstract default_protocol()[source]

The protocol that is prepended to the path when no protocol is specified.

glob_wildcards(pattern, *args, **kwargs)[source]
remote(value, *args, keep_local=None, stay_on_remote=None, static=False, **kwargs)[source]
abstract remote_interface()[source]
supports_default = False
class snakemake.remote.DomainObject(*args, **kwargs)[source]

Bases: snakemake.remote.AbstractRemoteObject

This is a mixin related to parsing components out of a location path specified as (host|IP):port/remote/location

property host
property local_path
property name
property path_prefix
property path_remainder
property port
property remote_path
class snakemake.remote.StaticRemoteObjectProxy[source]

Bases: ObjectProxy

Proxy that implements static-ness for remote objects.

The constructor takes a real RemoteObject and returns a proxy that behaves the same except for the exists() and mtime() methods.

exists()[source]
is_newer(time)[source]
mtime()[source]