BiLSTM layer

Long short-term memory (LSTM) is a specific type of recurrent neural network that models dependencies between elements in a sequence through recurrent connections. Here, we use one forward LSTM to compute a hidden state h→t=LSTMxth→t−1∈ℝd2 of the sequence X from left to right at the t-th time step, and the other backward LSTM to compute a hidden state h←t=LSTMxth←t+1∈ℝd2 of the same sequence in reverse, where d₂ is the dimension of the hidden state. Then, the two hidden states are concatenated to form the final output ht=h→th←t of the BiLSTM layer at the t-th time step.

After that, the output of BiLSTM h=h1...ht...hn∈ℝ2d2×n is fed to a two-layer fully-connected neural network (FC) with tanh activation to predict the confidence score for each possible label of the word, which can be written as follow:

where W∈ℝd2×2d2, V∈ℝk×d2 and b∈ℝd2×n are the parameters that need to be trained, k is the number of distinct labels.