2.5. Graph attention Levenberg-Marquardt algorithm

Graph convolutional neural networks aggregate the neighboring information of nodes with equal weights. To clarify the weight values of neighboring information of each node, Veličković et al. proposed a graph attention neural network based on the attention mechanism framework [38]. It can aggregate the neighboring information of nodes with a learned weights. The attention coefficients c is computed by performing self-attention on the nodes a shared attentional mechanism aT∈<!-- ∈ -->R2fl+1 . And further the LeakyReLU nonlinearity is applied to the attention coefficients as follows:

Here, .T represents transposition and || is the concatenation operation. W∈<!-- ∈ -->Rfl+1×<!-- × -->fl is a linear transformation. To make coefficients easily comparable across different nodes, the Softmax function is used to normalize Eq. (11). The coefficient α computed by the attention mechanism is expressed as:

The attention mechanism α<!-- α --> is a single-layer feedforward neural network. The learning process is stabilized through multi-head attention, which is performed on the final layer of the network by averaging and on other layers by concatenation.