Regular RNNs process input in sequence. When applied to a language modeling task, one tries to predict a word given the previous ones. For example, with the sentence The quick brown fox jumps over the lazy, a classical RNN will initialize and internal state \(s_0\) and process each word in sequence, starting from The and updating its internal state with each new word in order to make a final prediction.
A backward RNN reverses this idea, building it internal state starting from the last word lazy (the one we can expect to have the most information for prediction) and process the sentence backward. It should be simpler for such network to learn to properly select meaningful information rather than starting from the first word which might not be useful at all.
A variant of this idea, would be to use the same principle and apply it forward in a transformer-like language model training situation. To predict fox in The quick brown <?> jumps over the lazy dog, the network would start by looking at brown and jumps, and then quick and over, etc.