How to Develop a Character-Based Neural Language Model in Keras

Last Updated on September 3, 2020

A language model predicts the next word in the sequence based on the specific words that have come before it in the sequence.

It is also possible to develop language models at the character level using neural networks. The benefit of character-based language models is their small vocabulary and flexibility in handling any words, punctuation, and other document structure. This comes at the cost of requiring larger models that are slower to train.

Nevertheless, in the field of neural language models, character-based models offer a lot of promise for a general, flexible and powerful approach to language modeling.

In this tutorial, you will discover how to develop a character-based neural language model.

After completing this tutorial, you will know:

• How to prepare text for character-based language modeling.
• How to develop a character-based language model using LSTMs.
• How to use a trained character-based language model to generate text.

Kick-start your project with my new book Deep Learning for Natural Language Processing, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

• Update Feb/2018: Minor update to generation for API change in Keras 2.1.3.