spaic.Backend package

Submodules

spaic.Backend.Builder module

Created on 2020/9/15 @project: SPAIC @filename: Builder @author: Hong Chaofei @contact: hongchf@gmail.com

@description:

spaic.Backend.Backend module

Created on 2020/8/6 @project: SPAIC @filename: Backend @author: Hong Chaofei @contact: hongchf@gmail.com @description: 定义网络仿真使用的backend，如 Pytorch, Tensorflow, CUDA, 达尔文芯片等，以及相应的微分方程求解方法比如 Euler, 2阶 Runge-Kutta等

class spaic.Backend.Backend.Backend(dt=0.1)

Bases: ABC

Basic backend class. All specified backend backend should subclass it. The backend is a parameter for the build function and becomes an attribute of all objects defined in the frontend backend network in building process. These objects build their initial data and specified operations into the attributes of backend, according to _variables and _operations respectively. The data will update in each step according the computation graph. :param dt: the length of a backend timestep, in millisecond. :type dt: float, optional

Variables:

• device (str) – the desired device of returned tensor. Its value can be ‘cpu’ or ‘cuda’. If None, uses the current device for the default tensor type.

• builded (bool) – whether the object defined in the frontend backend network has been builded.

• time (float) – current backend time, in millisecond.

• n_time_step (int) – the num of current time step.

• _variables (OrderedDict) – records all variables from the build function of frontend objects.

• _parameters_dict (OrderedDict) – records the variables to be trained.

• _InitVariables_dict (OrderedDict) – reserves a copy of the initialization variables for initialization.

• _graph_var_dicts (dict) – has following format: {‘variables_dict’: self._variables, ‘temp_dict’: dict(), ‘update_dict’: dict(), ‘reduce_dict’: dict()}, recording the intermediate value of variables in computation progress.

• basic_operate (dict) – dictionary of basic operators, mapping from operator names using in frontend to the funtion objects implemented in backend.

• _operations (list) – records all basic operations from the build function of frontend objects, each of which has following format: [ret_var_name: str, operation_name, input_var_name1: str, input_var_name2 :str, …].

• _graph_operations (list) – redefine each basic operation, that is, add the corresponding keyword in the _graph_var_dicts to each variable, which has following format: [(dict_type, ret_var_name), operation_name, [(dict_type1, input_var_name1),(dict_type2, input_var_name2),…]].

• _standalone_operations (list) – records all standalone operations from the build function of frontend objects, each of which has following format: (ret_var_name: str, function, input_var_names: list).

• _initial_operations (list) – records all initial operations from the build function of frontend objects, each of which has following format: (ret_var_name: str, function, input_var_names: list).

• _monitors (list) – records all monitors defined in fronted network through build function of Monitor object.

build_graph()

build a computation graph before performing the calculation.

graph_update_step()

update value of _graph_var_dicts.

initial_step()

initialize network variables.

update_step()

update the return variables of standalone operations and basic operations and current backend time.

r_update_step()

update the return variables of basic operations without using graph_update_step().

add_variable()

add variables from front objects to _variables of Backend.

add_backend_variable()

add variables according to the specified backend.

add_operation()

add basic operations from front objects to _operations of Backend.

register_standalone()

add standalone operations from front objects to _standalone_operations of Backend.

register_initial()

add initial operations from front objects to _initial_operations of Backend.

basic_operate = {}

param_init_operate = {}

backend_name = 'None'

set_batch_size(batch_size)

get_batch_size()

set_runtime(runtime)

register_basic_op(name, op_function)

register the function as basic operation to backend’s _basic_operation dict :param name: :param op_function:

Returns:

build_graph()

Build a computation graph before performing the calculation. Note that only the basic operations are redefiend into the _graph_operations list. The format of _graph_operations is as follows: [(dict_type, ret_var_name), operation_name, [(dict_type1, input_var_name1),(dict_type2, input_var_name2),…]]. Traverse all basic operations and add the corresponding keyword in the _graph_var_dicts as dict_type to each variable in basic operation.

get_dependency()

set_func_grad()

var_check(op, *args)

For specified operation, check the type or the shape of input variables.

graph_update_step_r()

get_place()

graph_update_step()

push_update_step()

fetch_update_step()

initial_step()

Initialize network variables.

clear_step()

Returns:

initial_continue_step()

Initialize network for continuous run.

update_step()

Update the return variables of standalone operations and basic operations and current backend time. :returns: tuple(self._variables.values())

update_time_steps()

r_update_step()

Update the return variables of basic operations without using graph_update_step(). :returns: tuple(self._variables.values())

reduce_sum_update(value)

get_varialble(name)

set_variable_value(name, value, is_parameter)

Set the backend value, in specific Backend :param name: :param value: :param is_parameter:

Returns:

add_variable(module, name: str, shape, value=None, is_parameter=False, is_sparse=False, init=None, init_param=None, min=None, max=None, is_constant=False, prefer_device=None)

Add variables from front objects to _variables of Backend and get copies to assign to _parameters_dict and _InitVariables_dict. :param module: the parent Module object the variable is belongs to :type module: spaic.BaseModule :param name: the name of the added variable :type name: str :param shape: the shape of the variable :type shape: list, int :param value: the value of the variable :type value: optional :param is_parameter: whether the variable is trainable :type is_parameter: bool, optional :param init: :type init: optional

has_variable(name: str)

add_delay(var_name, max_delay)

abstract add_backend_variable(module, name, shape, value=None, grad=False, is_sparse=False, init=None, init_param=None)

This method will be overwritten by different subclasses to add variables to _variables of specified backend. :param module: the parent Module object the variable is belongs to :type module: spaic.BaseModule :param name: the name of the added variable :type name: str :param shape: the shape of the variable :type shape: list, int :param value: the value of the variable :type value: optional :param is_parameter: whether the variable is trainable :type is_parameter: bool, optional :param init: :type init: optinal :param grad: whether to use grad :type grad: bool, optional

abstract sparse_to_dense(index_name, value_name, shape_name)

This method will be sparse matrix to dense matrix. :param index_name: :type index_name: str :param value_name: :type value_name: str :param shape_name: :type shape_name: str

add_operation(op)

Add basic operations from front objects to _operations of Backend.

register_standalone(op)

Add standalone operations from front objects to _standalone_operations of Backend.

register_initial(op)

Add initial operations from front objects to _initial_operations of Backend.. op = {output, func, input, owner, place, requires_grad}

store(name='default')

Store backend_name and _variables into _stored_states dictionary. :param name: the name of network state. :type name: str, optional

restore(name='default')

Restore network state from _stored_states dictionary. :param name: the name of network state. :type name: str

check_key(ckey, target_dict)

abstract to_nograd_func(func)

Define the function as with no_grad :param func:

Returns:

abstract to_grad_func(func)

Define the function as with enable_grad :param func:

Returns:

abstract threshold(v, v_th)

Parameters:

• v – membrane voltage

• v_th – threshold

Returns:

v> v_th

abstract reset(v, o)

Parameters:

• v – membrane voltage

• o – output spike

Returns:

if o>0, return 0 else reutn v

abstract cat(x, dim=1)

Joining data together along a dimension. Note that the total dimension of the data remains the same after cat. :param x: :type x: list :param dim: the dimension to cat. :type dim: int

Returns:

concat(x, dim)

abstract stack(x, dim=1)

Add new dimension when stack data. :param x: :type x: list :param dim: the dimension to stack. :type dim: int

Returns:

stack(x, dim)

abstract permute(x, permute_dim)

Parameters:

• input (x--->) –

• operation (permute_dim---> the dimension index of permute) –

abstract view(x, view_dim)

Parameters:

• input (x--->) –

• operation (view_dim---> the shape of view) –

assign(x)

Parameters:

• target (y--->) –

• input (x--->) –

• x (y =) –

abstract unsqueeze(x, dim)

Parameters:

• input (x--->) –

• operation (dim---> the dim of unsqueeze) –

abstract reduce_sum(x, *dim)

Reduce the dimensions of the data :param x: :type x: list :param dim: the dimension to reduce. :type dim: tuple(int)

Returns:

sum(x, dim)

abstract index_select(x, indices, dim=1)

Parameters:

• x –

• indices –

abstract scatter(x, indices)

Parameters:

• x –

• indices –

abstract conv1d(x, kernel)

Parameters:

• x –

• kernel –

abstract conv_2d_complex(x, kernel, stride, padding, dilation, groups, beta, delay=None)

Parameters:

• x –

• kernel –

• stride –

• padding –

• dilation –

• groups –

• beta –

Returns:

abstract conv_trans2d(x, kernel, stride, padding, dilation, groups)

transposed conv 2d :param x: :param kernel:

abstract conv_trans1d(x, kernel)

transposed conv 1d :param x: :param kernel:

abstract im2col_indices(x, kh, kw, padding, stride)

Parameters:

• x (4D array N, FH, FW, C_{in}) –

• kh (kernel_height) –

• kw (kernel_width) –

• stride –

• padding –

abstract conv2d_flatten(x)

Parameters:

• x (4D array (batch_size, out_channels, height, width)) –

• Returns –

• (batch_size (3D array) –

• out_channels –

• width) (height *) –

• ---------- –

abstract feature_map_flatten(x)

For RSTDP and STDP learning rules which is follwed with conv pre_layer :param x: :type x: 4D array (batch_size, out_channels, height, width) :param Returns: :param 2D array (batch_size: :param out_channels * height * width): :param ———-:

abstract add(x, y)

Add the tensor y to the input x and returns a new result. :param x: input :type x: Tensor :param y: the second input :type y: Tensor or Number

Returns:

x + y

abstract minus(x, y)

The first input minus the second input :param x: input :type x: Tensor :param y: the second input :type y: Tensor or Number

Returns:

x - y

abstract div(x, y)

The first input div the second input :param x: input :type x: Tensor :param y: the second input :type y: Tensor or Number

Returns:

x/y

abstract relu(x)

Rectified Linear :param x:

Returns:

x = x if x>0. else x = 0

abstract mat_mult_weight(A, X)

Matrix product. :param A: the first input to be multiplied :type A: Tensor :param X: the second input to be multiplied :type X: Tensor

Returns:

mat_mult_weight(A,X)

abstract mat_mult_weight_complex(A, X, beta, delay=None)

abstract mat_mult_weight_2complex(A, X, beta)

abstract mat_mult_pre(A, X)

Matrix product. :param A: the first input to be multiplied :type A: Tensor :param X: the second input to be multiplied :type X: Tensor

Returns:

mat_mult_pre(A,X)

abstract sigmoid(x)

Parameters:

x –

Returns:

abstract upsample(x, scale)

abstract mat_mult(A, X)

Matrix product. :param A: the first input to be multiplied :type A: Tensor :param X: the second input to be multiplied :type X: Tensor

Returns:

mat_mult(A,X)

abstract reshape_mat_mult(A, X)

Matrix product. :param A: the first input to be multiplied :type A: Tensor :param X: the second input to be multiplied :type X: Tensor

Returns:

abstract bmm(A, X)

Performs a batch matrix-matrix product. :param A: the first input to be multiplied [batch_size, n, m] :type A: Tensor :param X: the second input to be multiplied [batch_size, m, p] :type X: Tensor

Returns:

bmm(A,X) [batch_size, n, p]

abstract sparse_mat_mult_weight(A, X)

Sparse matrix product. :param A: the first input to be multiplied :type A: Tensor :param X: the second input to be multiplied :type X: Tensor

Returns:

sparse_mat_mult_weight(A,X)

abstract var_mult(A, X)

Parameters:

• A –

• X –

Returns:

A * X

abstract mult_sum_weight(A, X)

sum(A*X, dim=-2)

Parameters:

• A –

• X –

Returns:

abstract mat_linear(A, X, b)

Parameters:

• A –

• X –

• b –

Returns:

mat_mul(A,X)+b

abstract ger(A, X)

Parameters:

• A –

• X –

Returns:

ger(A,X)

abstract var_linear(A, X, b)

If A is matrix, then A and X should have the same shape, A*X is elemen-wise multiplication else A should be a scalar value. :returns: A*X +b

abstract to_numpy(data)

Args：

data

Returns:

data.numpy()

abstract to_tensor(data)

Parameters:

data –

Returns:

torch.tensor(data)

abstract clamp_(data, min, max)

in-place clamp the data

abstract clamp_max_(data, max)

in-place clamp the max of the data

abstract clamp_min_(data, min)

in-place clamp the min of the data

abstract uniform(data, a=0.0, b=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a (float) – the lower bound of the uniform distribution

• b (float) – the upper bound of the uniform distribution

Returns:

torch.nn.init.uniform_(data, a=0.0, b=1.0)

abstract normal(data, mean=0.0, std=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• mean (float) – the mean of the normal distribution

• std (float) – the standard deviation of the normal distribution

Returns:

torch.nn.init.normal_(data, mean=0.0, std=1.0)

abstract xavier_normal(data, gain=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• gain – an optional scaling factor

Returns:

torch.nn.init.xavier_normal_(data, gain=1.0)

abstract xavier_uniform(data, gain=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• gain – an optional scaling factor

Returns:

torch.nn.init.xavier_uniform_(data, gain=1.0)

abstract kaiming_normal(data, a=0, mode='fan_in', nonlinearity='leaky_relu')

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with ‘leaky_relu’)

• mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with ‘relu’ or ‘leaky_relu’ (default).

Returns:

torch.nn.init.kaiming_normal_(data, a=0, mode=’fan_in’, nonlinearity=’leaky_relu’)

abstract kaiming_uniform(data, a=0, mode='fan_in', nonlinearity='leaky_relu')

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with ‘leaky_relu’)

• mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with ‘relu’ or ‘leaky_relu’ (default).

Returns:

torch.nn.init.kaiming_uniform_(data, a=0, mode=’fan_in’, nonlinearity=’leaky_relu’)

abstract constant(data, constant_value=0.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• constant_value (float) – the value to fill the tensor with

Returns:

torch.nn.init.constant_(data, constant_value)

abstract to(data, device)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• constant_value (float) – the value to fill the tensor with

Returns:

torch.nn.init.constant_(data, constant_value)

abstract weight_norm(weight, amp)

w = g/||v|| * v :param weight: v :param amp: g

Returns: amp*weight/norm(weight)

exp(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

sin(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

cos(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

tan(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

log(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

log2(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

log10(x)

Parameters:

x (tensor) – an n-dimensional torch.Tensor

Returns:

return exp(x)

spaic.Backend.TF_Backend module

spaic.Backend.Torch_Backend module

Created on 2020/9/14 @project: SPAIC @filename: Torch_Backend @author: Hong Chaofei @contact: hongchf@gmail.com @description:

class spaic.Backend.Torch_Backend.Torch_Engine(graph_operations)

Bases: Module

forward(variables: Dict[str, Tensor])

training: bool

class spaic.Backend.Torch_Backend.Torch_Backend(device='cpu')

Bases: Backend

backend_name = 'pytorch'

build()

build_graph()

remove_tensor(inputs)

is_insert(inputs)

move_compute_and_assign_tensors(op)

graph_update_step_multigpu()

to_nograd_func(func)

to_grad_func(func)

add_backend_variable(module, name, shape, value=None, grad=False, is_sparse=False, init=None, init_param=None, prefer_device=None)

Parameters:

• name –

• shape –

• value –

• init –

set_variable_value(name, value, is_parameter)

sparse_to_dense(index_name, value_name, shape_name)

get_str(level)

threshold(x, v_th)

reset(v, o)

cat(x, dim=1)

stack(x, dim=1)

reduce_sum(x, *dim)

index_select(x, indices, dim=1)

permute(x, permute_dim)

view(x, view_dim)

scatter(x, indices)

conv1d(x, kernel)

conv_trans1d(x, kernel, bias=None)

conv_2d(x, kernel, stride, padding, dilation, groups, bias=None, padding_mode='constant')

conv_2d_complex(x, kernel, stride, padding, dilation, groups, beta, bias=None, delay=None)

conv_trans2d(x, kernel, stride=1, padding=0, dilation=0, groups=1)

conv_max_pool2d(x, kernel, pool_kernel, stride, pool_stride, padding, pool_padding, dilation, groups)

conv_avg_pool2d(x, kernel, pool_kernel, stride, pool_stride, padding, pool_padding, dilation, groups)

conv_add_bias(x, bias)

max_pool2d(x, pool_kernel, pool_stride, pool_padding)

post_max_pool2d_complex(x, pool_kernel, pool_stride, pool_padding)

avg_pool2d(x, pool_kernel, pool_stride, pool_padding)

batchnorm2d(x, num_features)

dropout(x, p, inplace=False)

reshape_mat_mult(A, X)

im2col_indices(x, kh, kw, padding, stride)

conv2d_flatten(x)

feature_map_flatten(x)

add(x, y)

minus(x, y)

div(x, y)

relu(x)

sigmoid(x)

mat_mult_weight(A, X)

Parameters:

• A--->preGroup (input) –

• X--->postGroup (weight) –

mat_mult_weight_complex(A, X, beta, delay=None)

Parameters:

• A--->preGroup (input) –

• X--->postGroup (weight) –

• postGroup (beta--->) –

mat_mult_weight_2complex(A, X, beta, delay=None)

mat_mult_pre(A, X)

Parameters:

• A--->preGroup (input) –

• X--->postGroup (weight) –

mat_mult(A, X)

Parameters:

• A--->preGroup (input) –

• X--->postGroup (weight) –

bmm(A, X)

Parameters:

• postGroup (A--->) –

• preGroup (X--->) –

ger(A, X)

Parameters:

• postGroup (A--->) –

• preGroup (X--->) –

sparse_mat_mult_weight(A, X)

Parameters:

• A--->preGroup (sparseWeight(post, pre)) –

• X--->postGroup (input(batch, pre)) –

var_mult(A, X)

mult_sum_weight(A, X)

mat_linear(A, X, b)

var_linear(A, X, b)

unsqueeze(X, dim)

to_numpy(data: Tensor)

to_tensor(data)

upsample(x, scale)

exp(x)

clamp_(data, min, max)

clamp_max_(data, max)

clamp_min_(data, min)

uniform(data, a=-0.0, b=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a (float) – the lower bound of the uniform distribution

• b (float) – the upper bound of the uniform distribution

Returns:

torch.nn.init.uniform_(data, a=0.0, b=1.0)

normal(data, mean=0.0, std=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• mean (float) – the mean of the normal distribution

• std (float) – the standard deviation of the normal distribution

Returns:

torch.nn.init.normal_(data, mean=0.0, std=1.0)

xavier_normal(data, gain=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• gain – an optional scaling factor

Returns:

torch.nn.init.xavier_normal_(data, gain=1.0)

xavier_uniform(data, gain=1.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• gain – an optional scaling factor

Returns:

torch.nn.init.xavier_uniform_(data, gain=1.0)

kaiming_normal(data, a=0, mode='fan_in', nonlinearity='leaky_relu')

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with ‘leaky_relu’)

• mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with ‘relu’ or ‘leaky_relu’ (default).

Returns:

torch.nn.init.kaiming_normal_(data, a=0, mode=’fan_in’, nonlinearity=’leaky_relu’)

kaiming_uniform(data, a=0, mode='fan_in', nonlinearity='leaky_relu')

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• a – the negative slope of the rectifier used after this layer (only used with ‘leaky_relu’)

• mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing ‘fan_in’ preserves the magnitude of the variance of the weights in the forward pass. Choosing ‘fan_out’ preserves the magnitudes in the backwards pass.

• nonlinearity – the non-linear function (nn.functional name), recommended to use only with ‘relu’ or ‘leaky_relu’ (default).

Returns:

torch.nn.init.kaiming_uniform_(data, a=0, mode=’fan_in’, nonlinearity=’leaky_relu’)

constant(data, constant_value=0.0)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• constant_value (float) – the value to fill the tensor with

Returns:

torch.nn.init.constant_(data, constant_value)

sparse(data, sparsity=0.1, std=0.01)

Parameters:

• data (tensor) – an n-dimensional torch.Tensor

• sparsity (float) – The fraction of elements in each column to be set to zero

• std (float) – the standard deviation of the normal distribution used to generate

• values (the non-zero) –

Returns:

torch.nn.init.sparse_(data, sparsity, std)

weight_norm(weight, amp)

to(x, device)

sin(x)

cos(x)

tan(x)

log(x)

log2(x)

log10(x)

Module contents

Created on 2020/8/11 @project: SPAIC @author: Hong Chaofei @contact: hongchf@gmail.com

@description: