Automation¶
In the previous tutorial, we explained how workflows and tasks can be used to standardize experiments and build semi-automated tune-up routines. In this tutorial, we go one step further and introduce the LabOne Q Automation framework, which can be used to fully automate a collection of routines, workflows, and tasks.
The LabOne Q Automation framework is an abstract graph-based execution framework that may be used either on its own or together with LabOne Q workflows. In this tutorial, we focus on the basic functionality of the core automation framework. For a detailed look at how the automation framework may be used together with LabOne Q workflows, please see the Experiment Workflow Automation tutorial in the LabOne Q Applications Library.
Anatomy of an automation graph¶
The automation graph organizes experiments into layers, where each layer defines a function applied to its associated nodes, executed either sequentially or in parallel. Each node represents a (group of) resource(s) for the function, where we define a resource as the minimal object, or set of objects, that the function can transform.
For example, consider a LabOne Q experiment workflow acting on a set of quantum elements. In this case, the layer executable/function is the workflow builder, and the resources are the quantum elements. It makes sense to have one quantum element per node, since we would like the option to execute the workflow builder with a single quantum element sequentially. If we had experiment workflows acting on pairs of quantum elements, then the resources are now pairs of quantum elements and it would make sense to have a pair of quantum elements per node. Note that for the specific case of LabOne Q experiment workflows, these design choices have been fixed in the corresponding automation subclasses.
In the base Automation classes, the choice of resources-per-node is up to the user and depends on the properties of the function, as well as the level of control that should be passed to the automation.
Every automation graph has a single root layer containing a single root node, which marks the start of the experiment suite. After this, the graph is constructed layer-by-layer, where each layer is displayed as a row of nodes. The edges on the graph indicate the layer/node dependencies.
The graph is then executed layer-by-layer in the order in which they were added, ignoring deactivated nodes. For example, if two layers are at the same distance from the root, then the layer added first will be executed first. We will discuss this in more detail in the Running the automation graph section.
Example problem¶
In order to explain how the automation framework works in more concrete terms, let us look at a simple example.
Consider a suite of experiments that consists of repeatedly pressing a set of four random number generators, which each generate a random integer between 1 and 10. If the total of the four numbers is greater than 20, then we repeat the experiments. Once we have successfully repeated the experiments three times, the experiment suite is complete.
Let us now walk through how we would automate this experiment suite step-by-step.
Imports¶
import random
import attrs
from laboneq.automation import Automation, AutomationLayer, AutomationLayerResult
from laboneq.automation.logic import FixedParameterUpdate
from laboneq.automation.serialization import load_automation_parameters_from_file
from laboneq.automation.web_viewer import start_web_viewer
from laboneq.core.utilities.dsl_dataclass_decorator import classformatter
Defining a layer executable¶
The first step when tackling a custom automation problem is to subclass AutomationLayer and define the abstract method run_executable_core. This method tells us how to pass parameters into our function and how to evaluate the results. For example, we could create a generate_random_ints function...
def generate_random_ints(node_keys: list[str], parameters: dict) -> dict[str, int]:
"""Generate random integers between 1 and 10.
Given a list of node keys and a corresponding parameters dictionary of random seeds, return a dictionary of results.
Arguments:
node_keys: List of node keys.
parameters: Dictionary of parameters, where the keys are the node keys and the values are the sub-dictionary of parameters, which must include the "seed" parameter.
Returns:
Dictionary of results, where the keys are the node keys and the values are the random integers.
"""
results = {}
for node_key in node_keys:
seed = parameters[node_key]["seed"]
random.seed(seed)
results[node_key] = random.randint(1, 10)
return results
...and a corresponding ExampleLayer class with the following run_executable_core method:
@classformatter
@attrs.define
class ExampleLayer(AutomationLayer):
def run_executable_core(self, automation: "Automation") -> AutomationLayerResult:
# Run function
results = self.function(self.target_node_keys, self.target_parameters)
# Evaluate results
eval_successes = {}
sum_of_node_results = sum(results.values())
for node_key in results.keys():
eval_successes[node_key] = True if (sum_of_node_results > 20) else False
return AutomationLayerResult(results=results, successes=eval_successes)
Since the function generate_random_ints is called inside the user-defined run_executable_core method, its inputs/outputs are unconstrained. The run_executable_core method itself, on the other hand, has a fixed set of inputs/outputs. As input, it has access to the layer instance, which stores the parameters, and the Automation object. The output is a AutomationLayerResult object, which stores the results object, i.e., the output of the function execution, and the dictionary of evaluation successes.
Constructing the automation graph¶
Having defined a custom layer and layer executable, we can now construct our automation graph.
Technically, the only argument that we need to initialize an Automation object is a name, which is used for identifying our automation instance. However, in practice, we typically also start with a set of automation parameters. These are the initial parameters that we use to execute the experiment suite.
If we do not pass a set of parameters when initializing the Automation object, or our set of parameters is incomplete, then we will need to pass parameters directly to the layers. Note that once a layer is added to the automation, the initial automation parameters are updated with the passed layer parameters, and this is used to define the initial layer parameters. Automation.parameters is a property that displays the layer parameters for each added layer.
In our case, we would like a set of random seeds that will give us random numbers with a total of greater than 20. Since we do not know which random seeds to pick, we can simply use the seeds: 1, 2, 3, 4. We deliberately set incorrect parameters for layer l2 and omit the parameters for layer l3 below, so that we can demonstrate how to update these directly later, when initializing a layer.
auto_params = {
"l1": {"n1": {"seed": 1}, "n2": {"seed": 2}, "n3": {"seed": 3}, "n4": {"seed": 4}},
"l2": {
"n1": {"seed": 1},
"n2": {"seed": 1},
"n3": {"seed": 1},
"n4": {"seed": 1},
}, # incorrect
# "l3": {"n1": {"seed": 1}, "n2": {"seed": 2}, "n3": {"seed": 3}, "n4": {"seed": 4}}, # omitted
}
Alternatively, we can get our automation parameters dictionary from the yaml file initial_parameters.yml using the deserialization function load_automation_parameters_from_file in automation.serialization.automation_parameters. The deserialization function supports standard converters to improve the readability of the yaml files. The current list of supported converters is:
- Any
__linspace__entry in the parameters dictionary, which hasstart,stop, andnumkeys withfloat,float, andintvalues, is automatically converted into anp.linspace(start, stop, num)object (in-place).
auto_params = load_automation_parameters_from_file("initial_parameters.yml")
Note that the automation parameters dictionary is keyed by layer key (compulsory), with subdictionaries keyed by node key. In our experiment suite, we have three layers, corresponding to the generate_random_ints function, with four nodes each. Each node corresponds to a single resource, which in this case is a random number from a generator.
With the set of automation parameters defined, we can now initialize our automation object as follows:
auto = Automation(name="example", parameters=auto_params)
As soon as we initialize our automation object, it is assigned a timestamp.
auto.timestamp
This timestamp is used for organizing the FolderStore (if activated) and is set every time an automation instance is run completely or manually updated using self.update_timestamp().
Notice that the parameters property is currently empty because no layers have been added.
auto.parameters
Initially, our automation graph has only the root layer containing a single root node. Let us now define and add our layers to the graph using the add_layer method. Similarly, it is possible to remove layers using the remove_layer method.
l1 = ExampleLayer(
generate_random_ints, ["n1", "n2", "n3", "n4"], key="l1", depends_on={"root"}
)
auto.add_layer(l1)
l2 = ExampleLayer(
generate_random_ints,
["n1", "n2", "n3", "n4"],
key="l2",
depends_on={"l1"},
parameters={ # update layer parameters
"n2": {"seed": 2},
"n3": {"seed": 3},
"n4": {"seed": 4},
},
)
auto.add_layer(l2)
l3 = ExampleLayer(
generate_random_ints,
["n1", "n2", "n3", "n4"],
key="l3",
depends_on={"l2"},
parameters={ # add layer parameters
"n1": {"seed": 1},
"n2": {"seed": 2},
"n3": {"seed": 3},
"n4": {"seed": 4},
},
)
auto.add_layer(l3)
Note that there are four compulsory arguments to initialize a layer: the layer function, the node keys in the layer, the layer key, and the set of layers on which the layer depends. Our example is relatively simple in that we execute our layers in a linear fashion, such that root -> l1 -> l2 -> l3.
There are two optional arguments: sequential, which determines whether to execute the layer sequentially, and parameters, which specifies/overwrites parameters for the layer. Since our layer executable checks if the sum of all the random integers generated is greater than 20, it does not make sense for us to run a layer sequentially, since a single integer between 1 and 10 will never be greater than 20. For the layer l3, we set the missing parameters, which updates the automation parameters dictionary before defining the layer parameters.
Let us confirm that the parameters that we set for layer l3 when initializing the layer are reflected in the automation parameters.
auto.parameters
Viewing the automation graph¶
There are two ways to view our automation graph.
To view the automation graph live, we can use the function start_web_viewer and click on the hyperlink.
start_web_viewer(auto)
After clicking on the local address, you should see the following graph in the layer view ...
... and in the node view:
To plot the automation graph, we can use the plot method.
auto.plot();
The recommended way of viewing the automation graph is via the web viewer, which is what we will use for the rest of this tutorial.
Note that the web viewer offers some interactivity in viewing and analysing the graph. For example, we can switch between node/layer view, click on the nodes/layers to see their statuses, and pan/zoom. We can also perform some basic operations, such as running and resetting the graph.
Accessing the layers and nodes¶
Although some operations may be performed via the web viewer, the main way to interact with the automation framework is via the Python API.
The layers and nodes may be accessed using the layers and nodes methods. The nodes method also has an optional layer_key argument to access only the nodes in that layer.
for layer in auto.layers():
print(layer)
for node in auto.nodes(layer_key="l1"):
print(node)
Similarly, the layer keys and node IDs may be accessed via the layer_keys and node_ids methods.
for layer_key in auto.layer_keys():
print(layer_key)
for node_id in auto.node_ids(layer_key="l1"):
print(node_id)
Note that a node key is used to identify a node within a layer, whereas a node ID is used to identify a node on the automation graph. The same node key is used to represent the same resource across different layers. The node ID is the layer key and node key joined with an underscore.
For easy access, we can also get the layers and nodes using the get_layer / get_node methods or directly from the automation object...
auto.get_layer("l1")
auto["l1"] # get the layer with layer key "l1"
auto.get_node("l1_n1")
auto["l1_n1"] # get the node with node ID "l1_n1"
...and we can get the nodes directly from the layers:
auto.get_layer("l1").get_node("n1")
auto["l1"]["n1"] # get the node with node key "n1" from layer "l1"
Running the automation graph¶
Now that we have constructed an automation graph, can view it live in a web viewer, and are comfortable accessing the layers and nodes, we are ready to run the graph.
The LabOne Q Automation framework uses a layer-based execution model, which means that the graph is run layer-by-layer, in the order in which they were added, and the user interacts with layers, rather than directly with nodes.
In the web viewer, layers are numbered in execution order. They are also listed in execution order in the layer legend. When the graph is run completely, we execute the layers in this order, skipping any deactivated nodes. If a node fails, then all of its descendents are deactivated. Nodes/layers can also be manually deactivated/reactivated using the deactivate_node/reactivate_node and deactivate_layer/reactivate_layer methods.
The most basic approach to get started is to run the graph completely, either using the run method or pressing the run button in the web viewer. Internally, this method calls run_layer, which returns the next layer key together with updated parameters (if any).
auto.run()
It is also possible to run a layer independently using the run_layer method. In this method, we can specify a subset of nodes to run (node_keys) and temporary parameters to use (parameters). The parameters passed to run_layer temporary replace the layer parameters. This is particularly useful for debugging. Apart from this, we can use the run_from_node method to execute a node and its descendents.
From the web viewer, we can see that the first layer of our graph failed, and therefore the subsequent layers of the graph cannot be executed.
We can confirm the status of a layer using the status attribute.
After a layer executable is run, statuses are assigned to the nodes in that layer. The possible node statuses are:
root- the root nodeready- a node that is ready to be runrunning- a node that is runningpassed- a node that has passedfailed- a node that has faileddeactivated- a node that has been deactivated (either manually, or automatically as a descendant of a failed node)
The node status is active if it is one of [ready, running, passed, failed] and is inactive if it is deactivated.
The layer statuses are computed from the node statuses as follows (in precedence order):
root- the root layerrunning- any active nodes in the layer arerunningfailed- any active nodes in the layer havefailedready- any active nodes in the layer arereadypassed- any active nodes in the layer havepasseddeactivated- all nodes in the layer aredeactivated
We can check how many times nodes in the layer have failed/passed using the fail_count/pass_count attribute, we can view the maximum fail counts using the max_fail_count attribute, and we can check when the layer was executed using the timestamp attribute.
auto["l1"].status # status of layer "l1"
auto["l1"].fail_count # fail counts for the nodes in layer "l1"
auto["l1"].pass_count # pass counts for the nodes in layer "l1"
auto["l1"].max_fail_count # maximum fail counts for the nodes in layer "l1"
auto["l1"].timestamp # timestamp for layer "l1"
The status information can also be found by clicking on a layer or node in the web viewer.
In the event of failure, it is important to diagnose why the layer failed. We can do this by first examining the results of the layer.
auto["l1"].results
In this case, we can see that the sum of the random numbers that were generated is less than twenty, which is why the layer fails.
Let us reset the automation graph using the reset method and think about how to proceed.
auto.reset()
Adding automation logic¶
In order to make the layer pass, we need to provide new random seeds. We can do this explicitly by changing the parameters of the automation layers and trying again. However, this is tedious since we do not know which random seeds will generate which numbers. An alternative approach is to add logic to the layer, such that the parameters are automatically incremented until success.
The abstract base logic class is called AutomationLogic. As with AutomationLayer, this class needs to be subclassed such that the run_executable_core method is defined. However, together with AutomationLogic, we also provide standard logic subclasses in automation.logic so that the user does not necessarily need to define their own. The current list of standard AutomationLogic subclasses is as follows:
FixedParameterUpdate-- fixed automation parameter update
In this example, we can use the standard logic subclass FixedParameterUpdate. Specifically, we want to increment the seeds by one, until success.
for layer in auto.layers():
layer.logic = FixedParameterUpdate(
new_layer_key=layer.key,
parameter_changes={
"n1": {"seed": 1},
"n2": {"seed": 1},
"n3": {"seed": 1},
"n4": {"seed": 1},
},
)
Note that every AutomationLogic class has an iterations argument. If this is specified, then the logic executes for a fixed number of iterations, regardless of whether the layer passes/fails. Otherwise, as in this case, we iterate until success (or until the max fail count is reached).
The logic subclass run_executable_core method has to return the new layer key and the updated parameters. Internally, when we run the automation, this executes a chain of run_layer methods, each returning the next layer key and updated parameters. Without logic, the next layer key is always the result of the next_layer_key method and the updated parameters are None. With logic, we intercept this chain and use the logic to decide on the next layer key and updated parameters. In this example, the next layer key is set to the current layer key, since we want to repeat the current layer.
We can verify that the logic has been set by printing the layer logic or by printing the automation parameters.
auto["l1"].logic
auto.parameters["l1"]["logic"]
We can also quickly see that a layer has logic in the web viewer via the gear symbol.
Similarly, logic can be deleted using the logic deleter, e.g. del auto["l1"].logic.
Now we can rerun the automation graph and see the effect of the layer logic.
auto.run()
The graph has now passed completely.
Let's examine the fail/pass counts to see what exactly happened.
for layer in auto.layers():
print(f"Fail count for layer {layer.key} = {layer.fail_count}")
print(f"Pass count for layer {layer.key} = {layer.pass_count}\n")
Here we can see that each layer failed twice and then passed once. This means that the random seeds 1, 2, 3, 4 had to be incremented twice before the sum of the random numbers was greater than 20. We can confirm that the sum of the results is now greater than 20 by looking at the results of any layer.
auto["l1"].results
Note that a fail count of 2 is below the max fail count of the layer, which is 4. The max fail count, as well as other node properties, can be adjusted directly per node. If the max fail count is reached, then the layer fails and, in this example, the automation would stop.
auto["l1"].max_fail_count
Accessing/setting parameters¶
As mentioned previously, automation parameters may be passed to initialize an automation object and layer parameters may be passed to initialize a layer object. When a layer is added to the automation, its parameters are amended with the initial automation parameters that serve as defaults. Parameters that have been set directly on the layer have priority. The read-only parameters property of the automation instance then displays the layer parameters of the added layers.
The layer parameters may be viewed / set using the parameters attribute, as shown below.
auto["l3"].parameters # get the parameters for layer "l3"
auto["l3"].parameters = {
"n1": {"seed": 0},
"n2": {"seed": 0},
"n3": {"seed": 0},
"n4": {"seed": 0},
} # set the parameters for layer "l3"
auto.parameters["l3"] # verify that the automation parameters are updated
Unlike the layer, the nodes do not contain input parameters for the layer executable, however, as shown in the previous section, they do contain execution parameters that can be set, such as max_fail_count, as well as the read-only status parameters, such as status.
Saving/loading parameters¶
When we started the automation, we initialized our automation object with a set of automation parameters. At the time, this was our best guess for the parameters that were needed for our experiment suite. Since then, our state of knowledge has been updated, and so have the automation parameters. We can view our current automation parameters...
auto.parameters
...and save them to file using the save_parameters method:
# Uncomment the line below to save the automation parameters
# auto.save_parameters(file="final_parameters.yml")
By default, this saves the parameters as a yaml file in a FolderStore-compatible directory tree. However, the file name and path can be modified using the optional file argument.
This automation parameters file can also be loaded into an existing automation instance, using the load_parameters method.
This tutorial covered the basic functionality of the core automation framework. Don't forget to read the Experiment Workflow Automation tutorial to find out how this framework can be used together with LabOne Q Workflows.