Experiment Structure#
KnowIt outputs results in a specific structure, meant to facilitate efficient experimentation.
This structure is illustrated in the following diagram.
Directories are indicated by <…> and files are indicated by […].
Titles in quotations are user defined or determined dynamically during experimentation.
<"experiment output directory">
├── <custom_archs>
│ ├── ["custom_arch_one.py"]
│ ├── ...
│ └── ["custom_arch_N.py"]
├── <custom_datasets>
│ ├── ["custom_dataset_one_meta.pickle"]
│ ├── <"custom_dataset_one">
│ ├── ...
│ ├── ["custom_dataset_N_meta.pickle"]
│ └── <"custom_dataset_N">
└── <models>
├── <"model name one">
├── <"model name two">
├── <"model name three">
│ ├── ["bestmodel-epoch=N-valid_loss=M.ckpt"]
│ ├── [model_args.yaml]
│ ├── <lightning_logs>
│ ├── <predictions>
│ ├── <interpretations>
│ │ ├── ["DeepLiftShap-eval-success-100-True-123.pickle"]
│ │ ├── ...
│ │ └── ["IntegratedGradients-eval-success-50-True-123.pickle"]
│ ├── <visualizations>
│ │ ├── ["visualization_one.png"]
│ │ ├── ...
│ │ └── ["visualization_N.gif"]
│ └── <sweeps>
│ └── <"sweep name one">
│ ├── <"batch_size_64-dropout_0.2-learning_rate_0.1">
│ ├── <"batch_size_64-dropout_0.2-learning_rate_0.01">
│ ├── <"batch_size_64-dropout_0.2-learning_rate_0.001">
│ │ ├── ["bestmodel-epoch=4-valid_loss=0.04.ckpt"]
│ │ ├── [model_args.yaml]
│ │ └── <lightning_logs>
│ └── ...
└── ...
1. Experiment-wide structure#
<”experiment output directory”>#
When a KnowIt object is created it can be linked to an existing output directory.
If no output directory is defined, a temporary one is made in ``KnowIt.temp_experiments``.
What this directory represents is very flexible. It could include a whole series of investigations,
potentially including, but not limited to:
- different splits of the data
- different datasets
- different architectures
- different capacities
- different preprocessing techniques
- different interpretations
Any time a new KnowIt object is created, and linked with this <"experiment output directory">, it will behave like a
continuation of any previous one that was linked with it.
<custom_archs>#
If a new custom architecture is imported using the created KnowIt object it will be stored in this directory as an architecture script.
The custom architecture will be a direct copy of the imported .py script.
See the Architecture How-to doc for the details on how an architecture script can be constructed and imported.
<custom_datasets>#
If a new dataset is imported using the created KnowIt object it will be stored in this directory as a pickle containing meta data and a partitioned parquet directory containing actual data.
Each dataset meta pickle contains a dictionary with the following entries:
- name : str
- components : list
- time_delta : timedelta
- instances_names : dict
- data_structure : dict
- base_nan_filler : str
- nan_filled_components : list
- the_data : dict
This dictionary and its accompanying partitioned parquet directory represents a base dataset in KnowIt.
See the Dataset How-to doc for details about what it contains.
<models>#
All models that are named and trained using the created KnowIt object will be stored in this directory, under a subdirectory
with the same name as the model.
2. Model-specific structure#
[bestmodel-epoch=N-valid_loss=M.ckpt]#
This file is the checkpoint of the saved model with the name corresponding to it's parent directory.
It contains all the parameter values.
> It is meant to represent the current best representation of the dataset on which it is trained.
*N* refers to the epoch at which this model was obtained.
*M* defines the value of the performance metric that was obtained and used as stopping criterion.
In the example validation loss is used.
[model_args.yaml]#
This is a yaml file that contains a dictionary with all the kwargs that were used to train
the [bestmodel-epoch=N-valid_loss=M.ckpt] model. It contains information on what architecture was used,
how the data was split and preprocessed, as well as specifics on how the trainer was set up.
With this file and access to the same datasets, any model trained by KnowIt should be reproducible.
<lightning_logs>#
This directory is generated by the [lightning.pytorch.loggers.CSVLogger](https://lightning.ai/docs/pytorch/stable/extensions/generated/lightning.pytorch.loggers.CSVLogger.html)
during the training process for the [bestmodel-epoch=N-valid_loss=M.ckpt] model.
It contains performance metrics throughout training (in the ``metrics.csv``) and trainer related hyperparameters (in the ``hparams.yaml``).
<predictions>#
All predictions are generated based on the [bestmodel-epoch=N-valid_loss=M.ckpt] model.
When model predictions are generated, they are stored in this directory.
There are two types of files in this directory.
1. Batch predictions#
There is a batch prediction file for each batch in a given split (i.e. train, validation, evaluation) of data.
This is a pickle file called ``set-batch-id.pickle``, where "set" is either train, valid or eval, and "id" is a batch number.
It contains a 3-value tuple:
(for regression tasks)
- s_id : tensor[num_sample_in_batch, ]
- prediction : tensor[num_sample_in_batch, num_output_timestep, num_output_components]
- target : tensor[num_sample_in_batch, num_output_timestep, num_output_components]
(for classification tasks)
- s_id : tensor[num_sample_in_batch, ]
- prediction : tensor[num_sample_in_batch, num_classes]
- target : tensor[num_sample_in_batch, num_classes]
The *s_id* tensor contains sample specific unique (within the current split set) ID's for each prediction point in the batch.
The *target* and *prediction* tensors represent the ground truths and model outputs for the relevant prediction points.
2. “IST” indices#
This is a pickle file called ``_set-ist_inx_dict.pickle``, where "set" is either train, valid or eval.
It contains a 2-value tuple:
- ist_values : list[num_sample_in_set, ]
- inx_dict : dict[num_sample_in_set, ]
Each item in *ist_values* corresponds to a single prediction point in the relevant set. Their order correponds to the unique IDs
stored in the batch predictions. Each item is a 3-valued tuple:
- i : int
- s : int
- t : pandas.Timestamp
*i* indicates the instance that the current prediction point belongs to.
*s* indicates the slice that the current prediction point belongs to.
*t* indicates the time step at which the current prediction point is found.
The *inx_dict* is a dictionary where the key is the unique ID stored in the batch predictions and the value is the batch number which contains it.
<interpretations>#
When model predictions are used for interpretations the results of the interpretation is stored in this directory.
Each interpretation is saved in a seperate .pickle file.
They are all generated for the [bestmodel-epoch=N-valid_loss=M.ckpt] model.
The exact name of the file and it's content will depend on the interpretation method.
Here we describe the currently available feature attribution methods.
The interpretation file name is: "method-set-selection-size-mbi-seed-inx_range.pickle" where,
- **method** is the feature attribution method used (DeepLiftShap, DeepLift, IntegratedGradients).
- **set** is the data split interpreted on (train, valid, eval)
- **selection** is the method used to select prediction points to interpret (all, random, success, failure)
- **size** is the number of prediction points interpreted on
- **mbi** is whether the feature attributions are multiplied by input
- **seed** is the seed used for random selection
The pickle file contains a dictionary containing all the same information as in the file name, in addition to:
- results : dict
- input_features : Tensor[size, num_input_delays, num_input_components]
- targets : list[size, ]
- predictions : list[size, ]
- timestamps : list[size, ]
The item corresponding to the "results" key is a dictionary where each key corresponds to an output logit of the model.
That is, a two-value tuple where the first value refers to an output delay index and the second values refers to an output component index.
The value that corresponds to each logit is also a dictionary containing two keys: attributions, and deltas.
The former contains the feature attribution values at each prediction point for the current logit and the latter contains
the deltas from the Captum feature attribution method used. This is illustrated in the diagram below.
results
├── (0, 0)
│ ├── attributions
│ └── Tensor[size, num_input_delays, num_input_components]
│ └── delta
│ └── Tensor[size * num_baselines, ]
├── ...
The item corresponding to the "input_features" key is a Tensor containing the exact input feature values used for interpretation.
The item corresponding to the "target" key is a list of arrays with shape (num_output_delays, num_output_components) providing the
ground truth target values for each interpreted prediction point.
The item corresponding to the "predictions" key is a list of arrays with shape (num_output_delays, num_output_components) providing the
model output values for each interpreted prediction point.
The item corresponding to the "timestamps" key is a list of 3-valued tuples providing the "IST indices" of each interpreted prediction point.
<visualizations>#
The default example visualizations will be stored in this directory.
<sweeps>#
When performing HP sweeps for our model the resulting sweeps and runs are stored in this directory.
Sweeps are stored as directories containing runs (also directories).
Within each run is the resulting model checkpoint, model_args.yaml, and lightning_logs.
See the HP sweet turorial for more details.