outputting module¶
outputting Module¶
Author: | Dominic Hunt |
---|
Functions¶
date () |
Calculate today’s date as a string in the form <year>-<month>-<day> and returns it |
dictKeyGen (store[, maxListLen, returnList, …]) |
Identifies the columns necessary to convert a dictionary into a table |
fancy_logger ([log_file, log_level, …]) |
Sets up the style of logging for all the simulations |
file_name_generator ([output_folder]) |
Keeps track of filenames that have been used and generates the next unused one |
flatDictKeySet (store[, selectKeys]) |
Generates a dictionary of keys and identifiers for the new dictionary, including only the keys in the keys list. |
folder_path_cleaning (folder) |
Modifies string file names from Windows format to Unix format if necessary and makes sure there is a / at the end. |
folder_setup (label, date_string[, …]) |
Identifies and creates the folder the data will be stored in |
listKeyGen (data[, maxListLen, returnList, …]) |
Identifies the columns necessary to convert a list into a table |
listSelection (data, loc) |
Allows numpy array-like referencing of lists |
newFlatDict (store[, selectKeys, labelPrefix]) |
Takes a list of dictionaries and returns a dictionary of 1D lists. |
newListDict (store[, labelPrefix, maxListLen]) |
Takes a dictionary of numbers, strings, lists and arrays and returns a dictionary of 1D arrays. |
pad (values, maxListLen) |
Pads a list with None |
pickleLog (results, file_name_gen[, label]) |
Stores the data in the appropriate pickle file in a Pickle subfolder of the outputting folder |
pickle_write (data, handle, file_name_gen) |
Writes the data to a pickle file |
Classes¶
LoggerWriter (writer) |
Fake file-like stream object that redirects writes to a logger instance. |
Saving ([label, output_path, config, …]) |
Creates the folder structure for the saved data and created the log file as log.txt |
Class Inheritance Diagram¶
digraph inheritance52b09d0a13 { rankdir=LR; size="8.0, 12.0"; "LoggerWriter" [URL="#outputting.LoggerWriter",fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5)",target="_top",tooltip="Fake file-like stream object that redirects writes to a logger instance. Taken from"]; "Saving" [URL="#outputting.Saving",fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5)",target="_top",tooltip="Creates the folder structure for the saved data and created the log file as ``log.txt``"]; }Author: | Dominic Hunt |
---|
-
class
outputting.
LoggerWriter
(writer)[source]¶ Bases:
object
Fake file-like stream object that redirects writes to a logger instance. Taken from https://stackoverflow.com/a/51612402
Parameters: writer (logging function) –
-
class
outputting.
Saving
(label=None, output_path=None, config=None, config_file=None, pickle_store=False, min_log_level=u'INFO', numpy_error_level=u'log')[source]¶ Bases:
object
Creates the folder structure for the saved data and created the log file as
log.txt
Parameters: - label (string, optional) – The label for the simulation. Default
None
will mean no data is saved to files. - output_path (string, optional) – The path that will be used for the run output. Default
None
- config (dict, optional) – The parameters of the running simulation/fitting. This is used to create a YAML configuration file.
Default
None
- config_file (string, optional) – The file name and path of a
.yaml
configuration file. DefaultNone
- pickle_store (bool, optional) – If true the data for each model, task and participant is recorded.
Default is
False
- min_log_level (basestring, optional) – Defines the level of the log from (
DEBUG
,INFO
,WARNING
,ERROR
,CRITICAL
). DefaultINFO
See https://docs.python.org/3/library/logging.html#levels - numpy_error_level ({'log', 'raise'}) – Defines the response to numpy errors. Default
log
. See numpy.seterr
Returns: file_name_gen – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (
handle
,extension
) and returns onefileName
stringReturn type: function
See also
folderSetup
- creates the folders
- label (string, optional) – The label for the simulation. Default
-
outputting.
date
()[source]¶ Calculate today’s date as a string in the form <year>-<month>-<day> and returns it
Returns: todayDate – The current date in the format <year>-<month>-<day> Return type: basestring
-
outputting.
dictKeyGen
(store, maxListLen=None, returnList=False, abridge=False)[source]¶ Identifies the columns necessary to convert a dictionary into a table
Parameters: - store (dict) – The dictionary to be broken down into keys
- maxListLen (int or float with no decimal places or None, optional) – The length of the longest expected list. Only useful if returnList is
True
. DefaultNone
- returnList (bool, optional) – Defines if the lists will be broken into 1D lists or values. Default
False
, lists will be broken into values - abridge (bool, optional) – Defines if the final dataset will be a summary or the whole lot. If it is a summary, lists of more than 10 elements are removed.
Default
False
, not abridged
Returns: - keySet (OrderedDict with values of OrderedDict, list or None) – The dictionary of keys to be extracted
- maxListLen (int or float with no decimal places or None, optional) – If returnList is
True
this should be the length of the longest list. If returnList isFalse
this should return its original value
Examples
>>> store = {'string': 'string'} >>> dictKeyGen(store) (OrderedDict([('string', None)]), 1) >>> store = {'num': 23.6} >>> dictKeyGen(store) (OrderedDict([('num', None)]), 1) >>> store = {'array': np.array([[1, 2, 3, 4, 5, 6], [7, 8, 9, 10, 11, 12]])} >>> dictKeyGen(store, returnList=True, abridge=True) (OrderedDict([(u'array', array([[0], [1]]))]), 6L) >>> store = {'dict': {1: "a", 2: "b"}} >>> dictKeyGen(store, maxListLen=7, returnList=True, abridge=True) (OrderedDict([('dict', OrderedDict([(1, None), (2, None)]))]), 7)
-
outputting.
fancy_logger
(log_file=None, log_level=20, numpy_error_level=u'log')[source]¶ Sets up the style of logging for all the simulations
Parameters: - date_string (basestring) – The date the log will start at
- log_file (string, optional) – Provides the path the log will be written to. Default “./log.txt”
- log_level ({logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL}) – Defines the level of the log. Default logging.INFO
- numpy_error_level ({'log', 'raise'}) – Defines the response to numpy errors. Default
log
. See numpy.seterr
Returns: close_loggers – Closes the logging systems that have been set up
Return type: function
See also
logging()
- The Python standard logging library
numpy.seterr()
- The function npErrResp is passed to for defining the response to numpy errors
-
outputting.
file_name_generator
(output_folder=None)[source]¶ Keeps track of filenames that have been used and generates the next unused one
Parameters: output_folder (string, optional) – The folder into which the new file will be placed. Default is the current working directory Returns: new_file_name – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: ( handle
,extension
) and returns onefileName
stringReturn type: function Examples
>>> file_name_gen = file_name_generator("./") >>> file_name_gen("a", "b") './a.b' >>> file_name_gen("a", "b") './a_1.b' >>> file_name_gen("", "") './' >>> file_name_gen = file_name_generator() >>> fileName = file_name_gen("", "") >>> fileName == os.getcwd() False
-
outputting.
flatDictKeySet
(store, selectKeys=None)[source]¶ Generates a dictionary of keys and identifiers for the new dictionary, including only the keys in the keys list. Any keys with lists will be split into a set of keys, one for each element in the original key.
These are named <key><location>
Parameters: - store (list of dicts) – The dictionaries would be expected to have many of the same keys. Any dictionary keys containing lists in the input have been split into multiple numbered keys
- selectKeys (list of strings, optional) – The keys whose data will be included in the return dictionary. Default
None
, which results in all keys being returned
Returns: keySet – The dictionary of keys to be extracted
Return type: See also
reframeListDicts()
,newFlatDict()
-
outputting.
folder_path_cleaning
(folder)[source]¶ Modifies string file names from Windows format to Unix format if necessary and makes sure there is a
/
at the end.Parameters: folder (string) – The folder path Returns: folder_path – The folder path Return type: basestring
-
outputting.
folder_setup
(label, date_string, pickle_data=False, base_path=None)[source]¶ Identifies and creates the folder the data will be stored in
Folder will be created as “./Outputs/<sim_label>_<date>/”. If that had previously been created then it is created as “./Outputs/<sim_label>_<date>_no_<#>/”, where “<#>” is the first available integer.
A subfolder is also created with the name
Pickle
if pickle is true.Parameters: - label (basestring) – The label for the simulation
- date_string (basestring) – The date identifier
- pickle_data (bool, optional) – If true the data for each model, task and participant is recorded.
Default is
False
- base_path (basestring, optional) – The path into which the new folder will be placed. Default is current working directory
Returns: folder_name – The folder path that has just been created
Return type: string
See also
newFile()
- Creates a new file
saving()
- Creates the log system
-
outputting.
listKeyGen
(data, maxListLen=None, returnList=False, abridge=False)[source]¶ Identifies the columns necessary to convert a list into a table
Parameters: - data (numpy.ndarray or list) – The list to be broken down
- maxListLen (int or float with no decimal places or None, optional) – The length of the longest expected list. Only useful if returnList is
True
. DefaultNone
- returnList (bool, optional) – Defines if the lists will be broken into 1D lists or values. Default
False
, lists will be broken into values - abridge (bool, optional) – Defines if the final dataset will be a summary or the whole lot. If it is a summary, lists of more than 10 elements are removed.
Default
False
, not abridged
Returns: - returnList (None or list of tuples of ints or ints) – The list of co-ordinates for the elements to be extracted from the data. If None the list is used as-is.
- maxListLen (int or float with no decimal places or None, optional) – If returnList is
True
this should be the length of the longest list. If returnList isFalse
this should return its original value
Examples
>>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=False, abridge=False) (array([[0, 0], [1, 0], [0, 1], [1, 1], [0, 2], [1, 2], [0, 3], [1, 3], [0, 4], [1, 4], [0, 5], [1, 5]]), 1) >>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=False, abridge=True) (None, None) >>> listKeyGen([[1, 2, 3, 4, 5, 6], [4, 5, 6, 7, 8, 9]], maxListLen=None, returnList=True, abridge=True) (array([[0], [1]]), 6L)
-
outputting.
listSelection
(data, loc)[source]¶ Allows numpy array-like referencing of lists
Parameters: - data (list) – The data to be referenced
- loc (tuple of integers) – The location to be referenced
Returns: selection – The referenced subset
Return type: Examples
>>> listSelection([1, 2, 3], (0,)) 1 >>> listSelection([[1, 2, 3], [4, 5, 6]], (0,)) [1, 2, 3] >>> listSelection([[1, 2, 3], [4, 5, 6]], (0, 2)) 3
-
outputting.
newFlatDict
(store, selectKeys=None, labelPrefix=u'')[source]¶ Takes a list of dictionaries and returns a dictionary of 1D lists.
If a dictionary did not have that key or list element, then ‘None’ is put in its place
Parameters: - store (list of dicts) – The dictionaries would be expected to have many of the same keys. Any dictionary keys containing lists in the input have been split into multiple numbered keys
- selectKeys (list of strings, optional) – The keys whose data will be included in the return dictionary. Default
None
, which results in all keys being returned - labelPrefix (string) – An identifier to be added to the beginning of each key string.
Returns: newStore – The new dictionary with the keys from the keySet and the values as 1D lists with ‘None’ if the keys, value pair was not found in the store.
Return type: Examples
>>> store = [{'list': [1, 2, 3, 4, 5, 6]}] >>> newFlatDict(store) OrderedDict([('list_[0]', [1]), ('list_[1]', [2]), ('list_[2]', [3]), ('list_[3]', [4]), ('list_[4]', [5]), ('list_[5]', [6])]) >>> store = [{'string': 'string'}] >>> newFlatDict(store) OrderedDict([(u'string', ["u'string'"])]) >>> store = [{'dict': {1: {3: "a"}, 2: "b"}}] >>> newFlatDict(store) OrderedDict([(u'dict_1_3', ["u'a'"]), (u'dict_2', ["u'b'"])])
-
outputting.
newListDict
(store, labelPrefix=u'', maxListLen=0)[source]¶ Takes a dictionary of numbers, strings, lists and arrays and returns a dictionary of 1D arrays.
If there is a single value, then a list is created with that value repeated
Parameters: - store (dict) – A dictionary of numbers, strings, lists, dictionaries and arrays
- labelPrefix (string) – An identifier to be added to the beginning of each key string. Default empty string
Returns: newStore – The new dictionary with the keys from the keySet and the values as 1D lists.
Return type: Examples
>>> store = {'list': [1, 2, 3, 4, 5, 6]} >>> newListDict(store) OrderedDict([('list', [1, 2, 3, 4, 5, 6])]) >>> store = {'string': 'string'} >>> newListDict(store) OrderedDict([('string', ['string'])]) >>> store = {'dict': {1: {3: "a"}, 2: "b"}} >>> newListDict(store) OrderedDict([(u'dict_1_3', ['a']), (u'dict_2', ['b'])])
-
outputting.
pickleLog
(results, file_name_gen, label=u'')[source]¶ Stores the data in the appropriate pickle file in a Pickle subfolder of the outputting folder
Parameters: - results (dict) – The data to be stored
- file_name_gen (function) – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (
handle
,extension
) and returns onefileName
string - label (string, optional) – A label for the results file
-
outputting.
pickle_write
(data, handle, file_name_gen)[source]¶ Writes the data to a pickle file
Parameters: - data (object) – Data to be written to the file
- handle (string) – The name of the file
- file_name_gen (function) – Creates a new file with the name <handle> and the extension <extension>. It takes two string parameters: (
handle
,extension
) and returns onefileName
string