Welcome to pyrep V. 3.1.0 documentation!¶

PYthon REPository or pyrep package provides a pythonic way to organize dumping and pulling python objects and other type of files to a folder or a directory that is called repository. A Repository can be created in any directory or folder, it suffices to initialize a Repository instance in a directory to start dumping and pulling objects into it. Any directory or a folder that contains a .pyrepinfo binary file in it, is theoretically a pyrep Repository. By default dump and pull methods use pickle to serialize storing python objects. Practically any other method can be used simply by providing the means and the required libraries in a simple form of string.

Installation guide:¶

pyrep is a pure python 2.7.x module that needs no particular installation. One can either fork pyrep’s github repository and copy the package to python’s site-packages or use pip as the following:

pip install pyrep

Package Functions:¶

pyrep.get_version()¶: Get pyrep’s version number.

pyrep.get_author()¶: Get pyrep’s author’s name.

pyrep.get_email()¶: Get pyrep’s author’s email.

pyrep.get_doc()¶: Get pyrep’s official online documentation link.

pyrep.get_repository()¶: Get pyrep’s official online repository link.

pyrep.get_pypi()¶: Get pyrep pypi’s link.

Usage:¶

from __future__ import print_function
import os
import warnings
from pprint import pprint

# numpy imports
import numpy as np

# import Repository
from pyrep import Repository

# initialize Repository instance
REP=Repository()

# create a path pointing to user home
PATH = os.path.join(os.path.expanduser("~"), 'pyrepTest_canBeDeleted')

# check if directory exist
if REP.is_repository(PATH):
    REP.remove_repository(path=PATH, removeEmptyDirs=True)


# print repository path
print("repository path --> %s"%str(REP.path))

# create repository in path
print("\n\nIs path '%s' a repository --> %s"%(PATH, str(REP.is_repository(PATH))))
success,message = REP.create_repository(PATH)
assert success, message
print('\nRepository path --> %s'%str(REP.path))

# add directories
success,message = REP.add_directory("folder1/folder2/folder3")
if not success:
    print(message)

success,message = REP.add_directory("folder1/archive1/archive2/archive3/archive3")
if not success:
    print(message)

success,message = REP.add_directory("directory1/directory2")
if not success:
    print(message)

# dump files
value = "This is a string data to pickle and store in the repository"
success,message = REP.dump_file(value, relativePath='pickled', dump=None, pull=None, replace=True)
if not success:
    print(message)

value = np.random.random(3)
dump  = "import numpy as np; dump=lambda path,value:np.savetxt(fname=path, X=value, fmt='%.6e')"
pull  = "import numpy as np; pull=lambda path:np.loadtxt(fname=path)"

success,message = REP.dump(value, relativePath='text.dat', dump=dump, pull=pull, replace=True)
if not success:
    print(message)

success,message = REP.dump(value, relativePath="folder1/folder2/folder3/folder3Pickled.pkl", replace=True)
if not success:
    print(message)

success,message = REP.dump(value, relativePath="folder1/archive1/archive1Pickled1", replace=True)
if not success:
    print(message)

success,message = REP.dump(value, relativePath="folder1/archive1/archive1Pickled2", replace=True)
if not success:
    print(message)

success,message = REP.dump(value, relativePath="folder1/archive1/archive2/archive2Pickled1", replace=True)
if not success:
    print(message)

# pull data
data = REP.pull(relativePath='text.dat')
print('\n\nPulled text data --> %s'%str(data))


data = REP.pull(relativePath="folder1/folder2/folder3/folder3Pickled.pkl")
print('\n\nPulled pickled data --> %s'%str(data))


# update
value = "This is an updated string"
REP.update(value, relativePath='pickled')
print('\nUpdate pickled data to --> %s'%value)

# walk repository files
print('\n\nwalk repository files relative path')
print('------------------------------------')
for f in REP.walk_files_path(recursive=True):
    print(f)


# walk repository, directories
print('\n\nwalk repository directories relative path')
print('------------------------------------------')
for d in REP.walk_directories_path(recursive=True):
    print(d)


print('\n\nRepository print -->')
print(REP)


print('\n\nRepository representation -->')
print(repr(REP))


print('\n\nRepository to list -->')
for fdDict in REP.get_repository_state():
    k = list(fdDict)[0]
    print("%s: %s"%(k,str(fdDict[k])))


print('\n\nCreate package from repository ...')
REP.create_package(path=None, name=None)

# Try to load
try:
    REP.load_repository(PATH)
except:
    loadable = False
finally:
    loadable = True

# Print whether repository is loadable
print('\nIs repository loadable -->',loadable)


# remove all repo data
REP.remove_repository(removeEmptyDirs=True)

# check if there is a repository in path
print( "\n\nIs path '%s' a repository --> %s"%(PATH, str(REP.is_repository(PATH))) )

output¶

repository path --> None

Is path '/Users/BA642A/pyrepTest_canBeDeleted' a repository --> False
Repository path --> /Users/BA642A/pyrepTest_canBeDeleted

Pulled text data --> [ 0.5496571   0.8600462   0.05659633]

Pulled pickled data --> [ 0.54965711  0.86004617  0.05659633]

Update pickled data to --> This is an updated string

walk repository files relative path
------------------------------------
pickled
text.dat
folder1/folder2/folder3/folder3Pickled.pkl
folder1/archive1/archive1Pickled1
folder1/archive1/archive1Pickled2
folder1/archive1/archive2/archive2Pickled1

walk repository directories relative path
------------------------------------------
folder1
directory1
folder1/folder2
folder1/archive1
folder1/folder2/folder3
folder1/archive1/archive2
folder1/archive1/archive2/archive3
folder1/archive1/archive2/archive3/archive3
directory1/directory2

Repository print -->
/Users/BA642A/pyrepTest_canBeDeleted
  pickled
  text.dat
  /directory1
    /directory2
  /folder1
    /archive1
    archive1Pickled1
    archive1Pickled2
      /archive2
      archive2Pickled1
        /archive3
          /archive3
    /folder2
      /folder3
      folder3Pickled.pkl

Repository representation -->
pyrep Repository (Version 3.0.0) @/Users/BA642A/pyrepTest_canBeDeleted [6 files] [9 directories]

Repository to list -->
: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
pickled: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}
text.dat: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}
directory1: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
directory1/directory2: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/archive1: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/archive1/archive1Pickled1: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}
folder1/archive1/archive1Pickled2: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}
folder1/archive1/archive2: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/archive1/archive2/archive2Pickled1: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}
folder1/archive1/archive2/archive3: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/archive1/archive2/archive3/archive3: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/folder2: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/folder2/folder3: {'pyrepdirinfo': True, 'type': 'dir', 'exists': True}
folder1/folder2/folder3/folder3Pickled.pkl: {'pyrepfileinfo': True, 'type': 'file', 'exists': True}

Create package from repository ...

Is repository loadable --> True

Is path '/Users/BA642A/pyrepTest_canBeDeleted' a repository --> False

pyrep.Repository.get_pickling_errors(obj, seen=None): Investigate pickling errors.

pyrep.Repository.get_dump_method(dump, protocol=-1): Get dump function code string

pyrep.Repository.get_pull_method(pull): Get pull function code string

pyrep.Repository.path_required(func): Decorate methods when repository path is required.

exception pyrep.Repository.InterpreterError: Bases: Exception

pyrep.Repository.my_exec(cmd, name, description)

class pyrep.Repository.Repository(path=None, pickleProtocol=2, timeout=10)

Bases: object

This is a pythonic way to organize dumping and pulling python objects or any type of files to a folder or directory that we call repository. Any directory can be a repository, it suffices to initialize a Repository instance in a directory to start dumping and pulling object into it. Any directory that has .pyreprepo pickle file in it is theoretically a pyrep Repository. Repository is thread and process safe. Multiple processes and threads can access, create, dump and pull into the repository.

Parameters:

Parameters:	path (None, string): This is used to load a repository instance. If None, Repository is initialized but not assigned to any directory. If Path is given then repository will be loaded from path if existing. pickleProtocol (int): Pickle protocol. Default value is 2 which makes it compatible with python 3 and 2. If user is always going to be using the same version of python, pickle.HIGHEST_PROTOCOL or -1 will ensure the highest protocol. timeout (number): The maximum delay or time allowed to successfully set the lock upon reading or writing to the repository

path (None, string): This is used to load a repository instance.

If None, Repository is initialized but not assigned to any directory.

If Path is given then repository will be loaded from path if existing.
pickleProtocol (int): Pickle protocol. Default value is 2 which makes it compatible with python 3 and 2. If user is always going to be using the same version of python, pickle.HIGHEST_PROTOCOL or -1 will ensure the highest protocol.
timeout (number): The maximum delay or time allowed to successfully set the lock upon reading or writing to the repository

DEBUG_PRINT_FAILED_TRIALS = False

len

locker

info: Get repository information

path: The repository instance path which points to the directory where .pyreprepo is.

uniqueName: Get repository unique name as generated when repository was created

get_stats()

Get repository descriptive stats

Returns:	numberOfDirectories (integer): Number of diretories in repository numberOfFiles (integer): Number of files in repository

reset(): Reset repository instance.

is_repository(path)

Check if there is a Repository in path.

Parameters:	path (string): The real path of the directory where to check if there is a repository.
Returns:	result (boolean): Whether it’s a repository or not.

load_repository(path, verbose=True, ntrials=3, safeMode=True)

Load repository from a directory path and update the current instance. First, new repository still will be loaded. If failed, then old style repository load will be tried.

Parameters:

Parameters:	path (string): The path of the directory from where to load the repository from. If ‘.’ or an empty string is passed, the current working directory will be used. verbose (boolean): Whether to be verbose about abnormalities ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration. safeMode (boolean): loading repository can be done without acquiring from multiple processes. Not acquiring the lock can be unsafe if another process is altering the repository
Returns:	repository (pyrep.Repository): returns self repository with loaded data.

path (string): The path of the directory from where to load the repository from. If ‘.’ or an empty string is passed, the current working directory will be used.
verbose (boolean): Whether to be verbose about abnormalities
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
safeMode (boolean): loading repository can be done without acquiring from multiple processes. Not acquiring the lock can be unsafe if another process is altering the repository

Returns:

repository (pyrep.Repository): returns self repository with loaded data.

create_repository(path, info=None, description=None, replace=True, allowNoneEmpty=True, raiseError=True)

create a repository in a directory. This method insures the creation of the directory in the system if it is missing.

N.B. If replace is True and existing repository is found in path, create_repository erases all existing files and directories in path.

Parameters:

Parameters:	path (string): The real absolute path where to create the Repository. If ‘.’ or an empty string is passed, the current working directory will be used. description (None, str): Repository main directory information. info (None, object): Repository information. It can be None or any pickle writable type of data. replace (boolean): Whether to replace existing repository. allowNoneEmpty (boolean): Allow creating repository in none-empty directory. raiseError (boolean): Whether to raise encountered error instead of returning failure.
Returns:	success (boolean): Whether creating repository was successful message (None, str): Any returned message.

path (string): The real absolute path where to create the Repository. If ‘.’ or an empty string is passed, the current working directory will be used.
description (None, str): Repository main directory information.
info (None, object): Repository information. It can be None or any pickle writable type of data.
replace (boolean): Whether to replace existing repository.
allowNoneEmpty (boolean): Allow creating repository in none-empty directory.
raiseError (boolean): Whether to raise encountered error instead of returning failure.

Returns:

success (boolean): Whether creating repository was successful
message (None, str): Any returned message.

remove_repository(path=None, removeEmptyDirs=True)

Remove all repository from path along with all repository tracked files.

Parameters:	path (None, string): The path the repository to remove. removeEmptyDirs (boolean): Whether to remove remaining empty directories including repository one.

save(description=None, raiseError=True, ntrials=3)

Save repository ‘.pyreprepo’ to disk and create (if missing) or update (if description is not None) ‘.pyrepdirinfo’.

Parameters:

Parameters:	description (None, str): Repository main directory information. If given will be replaced. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (bool): Whether saving was successful. error (None, string): Fail to save repository message in case saving is not successful. If success is True, error will be None.

description (None, str): Repository main directory information. If given will be replaced.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (bool): Whether saving was successful.
error (None, string): Fail to save repository message in case saving is not successful. If success is True, error will be None.

is_name_allowed(path)

Get whether creating a file or a directory from the basenane of the given path is allowed

Parameters:	path (str): The absolute or relative path or simply the file or directory name.
Returns:	allowed (bool): Whether name is allowed. message (None, str): Reason for the name to be forbidden.

to_repo_relative_path(path, split=False)

Given a path, return relative path to diretory

Parameters:	path (str): Path as a string split (boolean): Whether to split path to its components
Returns:	relativePath (str, list): Relative path as a string or as a list of components if split is True

get_repository_state(relaPath=None)

Get a list representation of repository state along with useful information. List state is ordered relativeley to directories level

Parameters:

Parameters:	relaPath (None, str): relative directory path from where to start. If None all repository representation is returned.
Returns:	state (list): List representation of the repository. List items are all dictionaries. Every dictionary has a single key which is the file or the directory name and the value is a dictionary of information including: ‘type’: the type of the tracked whether it’s file, dir, or objectdir ‘exists’: whether file or directory actually exists on disk ‘pyrepfileinfo’: In case of a file or an objectdir whether .%s_pyrepfileinfo exists ‘pyrepdirinfo’: In case of a directory whether .pyrepdirinfo exists

relaPath (None, str): relative directory path from where to start. If None all repository representation is returned.

Returns:

state (list): List representation of the repository. List items are all dictionaries. Every dictionary has a single key which is the file or the directory name and the value is a dictionary of information including:
- ‘type’: the type of the tracked whether it’s file, dir, or objectdir
- ‘exists’: whether file or directory actually exists on disk
- ‘pyrepfileinfo’: In case of a file or an objectdir whether .%s_pyrepfileinfo exists
- ‘pyrepdirinfo’: In case of a directory whether .pyrepdirinfo exists

get_repository_directory(relativePath)

Get repository directory list copy.

Parameters:	relativePath (string): The relative to the repository path .
Returns:	dirList (None, list): List of directories and files in repository directory. If directory is not tracked in repository None is returned

get_file_info(relativePath)

Get file information dict from the repository given its relative path.

Parameters:	relativePath (string): The relative to the repository path of the file.
Returns:	info (None, dictionary): The file information dictionary. If None, it means an error has occurred. errorMessage (string): The error message if any error occurred.

is_repository_directory(relativePath)

Get whether directory is registered in repository.

Parameters:	relativePath (string): The relative to the repository path.
Returns:	result (boolean): Whether directory is tracked and registered.

is_repository_file(relativePath)

Check whether a given relative path is a repository file path

Parameters:	relativePath (string): File relative path
Returns:	isRepoFile (boolean): Whether file is a repository file. isFileOnDisk (boolean): Whether file is found on disk. isFileInfoOnDisk (boolean): Whether file info is found on disk. isFileClassOnDisk (boolean): Whether file class is found on disk.

walk_files_path(relativePath='', fullPath=False, recursive=False)

Walk the repository relative path and yield file relative/full path.

Parameters:	relativePath (string): The relative path from which start the walk. fullPath (boolean): Whether to return full or relative path. recursive (boolean): Whether walk all directories files recursively

walk_files_info(relativePath='', fullPath=False, recursive=False)

Walk the repository relative path and yield tuple of two items where first item is file relative/full path and second item is file info. If file info is not found on disk, second item will be None.

Parameters:	relativePath (string): The relative path from which start the walk. fullPath (boolean): Whether to return full or relative path. recursive (boolean): Whether walk all directories files recursively

walk_directories_path(relativePath='', fullPath=False, recursive=False)

Walk repository relative path and yield directory relative/full path

Parameters:	relativePath (string): The relative path from which start the walk. fullPath (boolean): Whether to return full or relative path. recursive (boolean): Whether walk all directories files recursively.

walk_directories_info(relativePath='', fullPath=False, recursive=False)

Walk the repository relative path and yield tuple of two items where first item is directory relative/full path and second item is directory info. If directory file info is not found on disk, second item will be None.

Parameters:	relativePath (string): The relative path from which start the walk. fullPath (boolean): Whether to return full or relative path. recursive (boolean): Whether walk all directories files recursively.

create_package(path=None, name=None, mode=None)

Create a tar file package of all the repository files and directories. Only files and directories that are tracked in the repository are stored in the package tar file.

N.B. On some systems packaging requires root permissions.

Parameters:

Parameters:	path (None, string): The real absolute path where to create the package. If None, it will be created in the same directory as the repository. If ‘.’ or an empty string is passed, the current working directory will be used. name (None, string): The name to give to the package file If None, the package directory name will be used with the appropriate extension added. mode (None, string): The writing mode of the tarfile. If None, automatically the best compression mode will be chose. Available modes are (‘w’, ‘w:’, ‘w:gz’, ‘w:bz2’)

path (None, string): The real absolute path where to create the package. If None, it will be created in the same directory as the repository. If ‘.’ or an empty string is passed, the current working directory will be used.
name (None, string): The name to give to the package file If None, the package directory name will be used with the appropriate extension added.
mode (None, string): The writing mode of the tarfile. If None, automatically the best compression mode will be chose. Available modes are (‘w’, ‘w:’, ‘w:gz’, ‘w:bz2’)

add_directory(relativePath, description=None, clean=False, raiseError=True, ntrials=3)

Add a directory in the repository and creates its attribute in the Repository with utc timestamp. It insures adding all the missing directories in the path.

Parameters:

Parameters:	relativePath (string): The relative to the repository path to where directory must be added. description (None, string): Any random description about the added directory. clean (boolean): Whether to remove existing non repository tracked files and folders in all created directory chain tree. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether adding the directory was successful. message (None, string): Reason why directory was not added or random information.

relativePath (string): The relative to the repository path to where directory must be added.
description (None, string): Any random description about the added directory.
clean (boolean): Whether to remove existing non repository tracked files and folders in all created directory chain tree.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether adding the directory was successful.
message (None, string): Reason why directory was not added or random information.

get_repository_parent_directory(relativePath)

Get repository parent directory list copy.

Parameters:	relativePath (string): The relative to the repository path .
Returns:	dirList (None, list): List of directories and files in repository parent directory. If directory is not tracked in repository None is returned

remove_directory(relativePath, clean=False, raiseError=True, ntrials=3)

Remove directory from repository tracking.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the directory to remove from the repository. clean (boolean): Whether to os remove directory. If False only tracked files will be removed along with left empty directories. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether removing the directory was successful. reason (None, string): Reason why directory was not removed.

relativePath (string): The relative to the repository path of the directory to remove from the repository.
clean (boolean): Whether to os remove directory. If False only tracked files will be removed along with left empty directories.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether removing the directory was successful.
reason (None, string): Reason why directory was not removed.

rename_directory(relativePath, newName, raiseError=True, ntrials=3)

Rename a directory in the repository. It insures renaming the directory in the system.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the directory to be renamed. newName (string): The new directory name. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether renaming the directory was successful. message (None, string): Some explanatory message or error reason why directory was not renamed.

relativePath (string): The relative to the repository path of the directory to be renamed.
newName (string): The new directory name.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether renaming the directory was successful.
message (None, string): Some explanatory message or error reason why directory was not renamed.

copy_directory(relativePath, newRelativePath, overwrite=False, raiseError=True, ntrials=3)

Copy a directory in the repository. New directory must not exist.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the directory to be copied. newRelativePath (string): The new directory relative path. overwrite (boolean): Whether to overwrite existing but not tracked directory in repository. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether renaming the directory was successful. message (None, string): Some explanatory message or error reason why directory was not renamed.

relativePath (string): The relative to the repository path of the directory to be copied.
newRelativePath (string): The new directory relative path.
overwrite (boolean): Whether to overwrite existing but not tracked directory in repository.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether renaming the directory was successful.
message (None, string): Some explanatory message or error reason why directory was not renamed.

dump_file(value, relativePath, description=None, dump=None, pull=None, replace=False, raiseError=True, ntrials=3)

Dump a file using its value to the system and creates its attribute in the Repository with utc timestamp.

Parameters:

Parameters:	value (object): The value of a file to dump and add to the repository. It is any python object or file. relativePath (str): The relative to the repository path to where to dump the file. description (None, string): Any description about the file. dump (None, string): The dumping method. If None it will be set automatically to pickle and therefore the object must be pickleable. If a string is given, it can be a keyword (‘json’,’pickle’,’dill’) or a string compileable code to dump the data. The string code must include all the necessary imports and a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed. e.g. “import numpy as np; np.savetxt(fname=’$FILE_PATH’, X=value, fmt=’%.6e’)” pull (None, string): The pulling method. If None it will be set automatically to pickle and therefore the object must be pickleable. If a string is given, it can be a keyword (‘json’,’pickle’,’dill’) or a string compileable code to pull the data. The string code must include all the necessary imports, a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed and finally a PULLED_DATA variable. e.g “import numpy as np; PULLED_DATA=np.loadtxt(fname=’$FILE_PATH’)” replace (boolean): Whether to replace any existing file. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether renaming the directory was successful. message (None, string): Some explanatory message or error reason why directory was not dumped.

value (object): The value of a file to dump and add to the repository. It is any python object or file.
relativePath (str): The relative to the repository path to where to dump the file.
description (None, string): Any description about the file.
dump (None, string): The dumping method. If None it will be set automatically to pickle and therefore the object must be pickleable. If a string is given, it can be a keyword (‘json’,’pickle’,’dill’) or a string compileable code to dump the data. The string code must include all the necessary imports and a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed.

e.g. “import numpy as np; np.savetxt(fname=’$FILE_PATH’, X=value, fmt=’%.6e’)”
pull (None, string): The pulling method. If None it will be set automatically to pickle and therefore the object must be pickleable. If a string is given, it can be a keyword (‘json’,’pickle’,’dill’) or a string compileable code to pull the data. The string code must include all the necessary imports, a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed and finally a PULLED_DATA variable.

e.g “import numpy as np; PULLED_DATA=np.loadtxt(fname=’$FILE_PATH’)”
replace (boolean): Whether to replace any existing file.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether renaming the directory was successful.
message (None, string): Some explanatory message or error reason why directory was not dumped.

dump(*args, **kwargs): Alias to dump_file

copy_file(relativePath, newRelativePath, force=False, raiseError=True, ntrials=3)

Copy a file in the repository.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the file that needst to be renamed. newRelativePath (string): The new relative to the repository path of where to move and rename the file. force (boolean): Whether to force renaming even when another repository file exists. In this case old repository file will be removed from the repository and the system as well. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether renaming the file was successful. message (None, string): Some explanatory message or error reason why directory was not updated.

relativePath (string): The relative to the repository path of the file that needst to be renamed.
newRelativePath (string): The new relative to the repository path of where to move and rename the file.
force (boolean): Whether to force renaming even when another repository file exists. In this case old repository file will be removed from the repository and the system as well.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether renaming the file was successful.
message (None, string): Some explanatory message or error reason why directory was not updated.

update_file(value, relativePath, description=False, dump=False, pull=False, raiseError=True, ntrials=3)

Update the value of a file that is already in the Repository.

If file is not registered in repository, and error will be thrown.

If file is missing in the system, it will be regenerated as dump method is called. Unlike dump_file, update_file won’t block the whole repository but only the file being updated.

Parameters:

value (object): The value of a file to update.

relativePath (str): The relative to the repository path of the file to be updated.

description (False, string): Any random description about the file. If False is given, the description info won’t be updated, otherwise it will be update to what description argument value is.

dump (False, string): The new dump method. If False is given, the old one will be used.

pull (False, string): The new pull method. If False is given, the old one will be used.

raiseError (boolean): Whether to raise encountered error instead of returning failure.

ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Parameters:	value (object): The value of a file to update. relativePath (str): The relative to the repository path of the file to be updated. description (False, string): Any random description about the file. If False is given, the description info won’t be updated, otherwise it will be update to what description argument value is. dump (False, string): The new dump method. If False is given, the old one will be used. pull (False, string): The new pull method. If False is given, the old one will be used. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:	success (boolean): Whether renaming the directory was successful. message (None, string): Some explanatory message or error reason why directory was not updated.

update(*args, **kwargs): Alias to update_file

pull_file(relativePath, pull=None, update=True, ntrials=3)

Pull a file’s data from the Repository.

Parameters:

Parameters:	relativePath (string): The relative to the repository path from where to pull the file. pull (None, string): The pulling method. If None, the pull method saved in the file info will be used. If a string is given, the string should include all the necessary imports, a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed and finally a PULLED_DATA variable. e.g “import numpy as np; PULLED_DATA=np.loadtxt(fname=’$FILE_PATH’)” update (boolean): If pull is not None, Whether to update the pull method stored in the file info by the given pull method. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	data (object): The pulled data from the file.

relativePath (string): The relative to the repository path from where to pull the file.
pull (None, string): The pulling method. If None, the pull method saved in the file info will be used. If a string is given, the string should include all the necessary imports, a ‘$FILE_PATH’ that replaces the absolute file path when the dumping will be performed and finally a PULLED_DATA variable. e.g “import numpy as np; PULLED_DATA=np.loadtxt(fname=’$FILE_PATH’)”
update (boolean): If pull is not None, Whether to update the pull method stored in the file info by the given pull method.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

data (object): The pulled data from the file.

pull(*args, **kwargs): Alias to pull_file

rename_file(relativePath, newRelativePath, force=False, raiseError=True, ntrials=3)

Rename a file in the repository. It insures renaming the file in the system.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the file that needst to be renamed. newRelativePath (string): The new relative to the repository path of where to move and rename the file. force (boolean): Whether to force renaming even when another repository file exists. In this case old repository file will be removed from the repository and the system as well. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.
Returns:	success (boolean): Whether renaming the file was successful. message (None, string): Some explanatory message or error reason why directory was not updated.

relativePath (string): The relative to the repository path of the file that needst to be renamed.
newRelativePath (string): The new relative to the repository path of where to move and rename the file.
force (boolean): Whether to force renaming even when another repository file exists. In this case old repository file will be removed from the repository and the system as well.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

Returns:

success (boolean): Whether renaming the file was successful.
message (None, string): Some explanatory message or error reason why directory was not updated.

remove_file(relativePath, removeFromSystem=False, raiseError=True, ntrials=3)

Remove file from repository.

Parameters:

Parameters:	relativePath (string): The relative to the repository path of the file to remove. removeFromSystem (boolean): Whether to remove file from disk as well. raiseError (boolean): Whether to raise encountered error instead of returning failure. ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.

relativePath (string): The relative to the repository path of the file to remove.
removeFromSystem (boolean): Whether to remove file from disk as well.
raiseError (boolean): Whether to raise encountered error instead of returning failure.
ntrials (int): After aquiring all locks, ntrials is the maximum number of trials allowed before failing. In rare cases, when multiple processes are accessing the same repository components, different processes can alter repository components between successive lock releases of some other process. Bigger number of trials lowers the likelyhood of failure due to multiple processes same time alteration.