mlsnippet.utils

class mlsnippet.utils.AutoInitAndCloseable

Bases: object

Classes with init() to initialize its internal states, and also close() to destroy these states. The init() method can be repeatedly called, which will cause initialization only at the first call. Thus other methods may always call init() at beginning, which can bring auto-initialization to the class.

A context manager is implemented: init() is explicitly called when entering the context, while destroy() is called when exiting the context.

__enter__()

Ensure the internal states are initialized.

__exit__(exc_type, exc_val, exc_tb)

Cleanup the internal states.

_close()

Override this method to destroy the internal states.

_init()

Override this method to initialize the internal states.

close()

Ensure the internal states are destroyed.

init()

Ensure the internal states are initialized.

mlsnippet.utils.DocInherit(kclass)

Class decorator to enable kclass and all its sub-classes to automatically inherit docstrings from base classes.

Usage:

import six


@DocInherit
class Parent(object):
    """Docstring of the parent class."""

    def some_method(self):
        """Docstring of the method."""
        ...

class Child(Parent):
    # inherits the docstring of :meth:`Parent`

    def some_method(self):
        # inherits the docstring of :meth:`Parent.some_method`
        ...

Parameters:kclass (Type) – The class to decorate. Returns:The decorated class.

mlsnippet.utils.timed_wait_proc(proc, timeout)

mlsnippet.utils.exec_proc(*args, **kwds)

Execute an external program within a context.

Parameters:

• args – Arguments of the program.
• on_stdout ((bytes) -> None) – Callback for capturing stdout.
• on_stderr ((bytes) -> None) – Callback for capturing stderr.
• stderr_to_stdout (bool) – Whether or not to redirect stderr to stdout? If specified, on_stderr will be ignored. (default False)
• buffer_size (int) – Size of buffers for reading from stdout and stderr.
• ctrl_c_timeout (int) – Seconds to wait for the program to respond to CTRL+C signal. (default 3)
• kill_timeout (int) – Seconds to wait for the program to terminate after being killed. (default 60)
• **kwargs – Other named arguments passed to subprocess.Popen().

Yields:

subprocess.Popen – The process object.

class mlsnippet.utils.ActiveFiles

Bases: object

A set to track active (still in-use) file objects.

This class holds weak references to active file objects, which will be all closed when close_all() of this class is called.

Since the references are weak, once a file object is no longer referenced by any other caller, it will be automatically removed from this set. However, in this situation, the close() method of the de-referenced file object will not be called by this class.

Such a class is majorly designed for keeping track of active file objects opened by a DataFS, which are forced to be closed as soon as the DataFS is closed.

add(file_obj)

Add a file object to this set.

Parameters:file_obj – The file object to be tracked. Returns:The provided file_obj.

close_all()

Close all file objects which are still active.

All close() methods of the active file objects are ensured to be called. Any error raised by the close() method of a file object would be catched and totally ignored, except for KeyboardInterrupt and SystemExit, which will be re-raised after all file objects have been closed.

mlsnippet.utils.iter_files(root_dir, sep='/')

Iterate through all files in root_dir, returning the relative paths of each file. The sub-directories will not be yielded.

Parameters:

• root_dir (str) – The root directory, from which to iterate.
• sep (str) – The separator for the relative paths.

Yields:

str – The relative paths of each file.

mlsnippet.utils.maybe_close(*args, **kwds)

Enter a context, and if obj has .close() method, close it when exiting the context.

Parameters:obj – The object maybe to close. Yields:The specified obj.

class mlsnippet.utils.TemporaryDirectory(suffix=None, prefix=None, dir=None)

Bases: object

Create and return a temporary directory. This has the same behavior as mkdtemp but can be used as a context manager. For .. rubric:: example

with TemporaryDirectory() as tmpdir:

…

Upon exiting the context, the directory and everything contained in it are removed.

cleanup()

mlsnippet.utils.makedirs(name, mode=511, exist_ok=False)

class mlsnippet.utils.MongoBinder(conn_str, db_name, coll_name)

Bases: mlsnippet.utils.concepts.AutoInitAndCloseable

Base class for MongoDB data binder.

A MongoDB data binder may save and load data in a MongoDB. This class provides the basic interface for accessing the MongoDB.

__init__(conn_str, db_name, coll_name)

Initialize the internal states of MongoBinder.

Parameters:

• conn_str (str) – The MongoDB connection string.
• db_name (str) – The MongoDB database name.
• coll_name (str) – The collection name (prefix) of the GridFS.

client

Get the MongoDB client. Reading this property will force the internal states of MongoFS to be initialized.

Returns:The MongoDB client. Return type:MongoClient

coll_name

Get the collection name (prefix) of the GridFS.

collection

Get the MongoDB collection object. Reading this property will force the internal states of MongoFS to be initialized.

Returns:The MongoDB collection object. Return type:Collection

conn_str

Get the MongoDB connection string.

db

Get the MongoDB database object. Reading this property will force the internal states of MongoFS to be initialized.

Returns:The MongoDB database object. Return type:Database

db_name

Get the MongoDB database name.

gridfs

Get the MongoDB GridFS client. Reading this property will force the internal states of MongoFS to be initialized.

Returns:The MongoDB GridFS client. Return type:GridFS

gridfs_bucket

Get the MongoDB GridFS bucket. Reading this property will force the internal states of MongoFS to be initialized.

Returns:The MongoDB GridFS bucket. Return type:GridFSBucket