Contributions to the joint segmentation and classification of sequences (My two cents on decoding and handwriting recognition)
Abstract
This work is focused on problems (like automatic speech recognition (ASR) and handwritten text recognition (HTR)) that: 1) can be represented (at least approximately) in terms of one-dimensional sequences, and 2) solving these problems entails breaking the observed sequence down into segments which are associated to units taken from a finite repertoire. The required segmentation and classification tasks are so intrinsically interrelated (“Sayre’s Paradox”) that they have to be performed jointly. We have been inspired by what some works call the “successful trilogy”, which refers to the synergistic improvements obtained when considering: - a good formalization framework and powerful algorithms; - a clever design and implementation taking the best profit of hardware; - an adequate preprocessing and a careful tuning of all heuristics. We describe and study “two stage generative models” (TSGMs) comprising two stacked probabilistic generative stages without reordering. This model not only includes Hidden Markov Models (HMMs, but also “segmental models” (SMs). “Two stage decoders” may be deduced by simply running a TSGM in reversed way, introducing non determinism when required: 1) A directed acyclic graph (DAG) is generated and 2) it is used together with a language model (LM). One-pass decoders constitute a particular case. A formalization of parsing and decoding in terms of semiring values and language equations proposes the use of recurrent transition networks (RTNs) as a normal form for Context Free Grammars (CFGs), using them in a parsing-as-composition paradigm, so that parsing CFGs result in a slight extension of regular ones. Novel transducer composition algorithms have been proposed that can work with RTNs and can deal with null transitions without resorting to filter-composition even in the presence of null transitions and non-idempotent semirings. A review of LMs is described and some contributions mainly focused on LM interfaces, LM representation and on the evaluation of Neural Network LMs (NNLMs) are provided. A review of SMs includes the combination of generative and discriminative segmental models and general scheme of frame emission and another one of SMs. Some fast cache-friendly specialized Viterbi lexicon decoders taking profit of particular HMM topologies are proposed. They are able to manage sets of active states without requiring dictionary look-ups (e.g. hashing). A dataflow architecture allowing the design of flexible and diverse recognition systems from a little repertoire of components has been proposed, including a novel DAG serialization protocol. DAG generators can take over-segmentation constraints into account, make use SMs other than HMMs, take profit of the specialized decoders proposed in this work and use a transducer model to control its behavior making it possible, for instance, to use context dependent units. Relating DAG decoders, they take profit of a general LM interface that can be extended to deal with RTNs. Some improvements for one pass decoders are proposed by combining the specialized lexicon decoders and the “bunch” extension of the LM interface, including an adequate parallelization. The experimental part is mainly focused on HTR tasks on different input modalities (offline, bimodal). We have proposed some novel preprocessing techniques for offline HTR which replace classical geometrical heuristics and make use of automatic learning techniques (neural networks). Experiments conducted on the IAM database using this new preprocessing and HMM hybridized with Multilayer Perceptrons (MLPs) have obtained some of the best results reported for this reference database. Among other HTR experiments described in this work, we have used over-segmentation information, tried lexicon free approaches, performed bimodal experiments and experimented with the combination of hybrid HMMs with holistic classifiers.
Keywords
- Pattern recognition
- Sequence classification
- Decoding
- Transducer composition
- Recurrent Transition Network
- Hidden Markov Model
- Hybrid HMM
- Segment Model
- Holistic classifier
- Language Modeling
- Neural Network Language Model
- Handwritten Text Recognition
- Slant Correction
- Text Size Normalization
Publications during the thesis (10)
- 2015 ConferenceA combined Convolutional Neural Network and Dynamic Programming approach for text line normalization
Joan Pastor Pellicer , Salvador España Boquera , María José Castro Bleda , Francisco Julian Zamora Martínez
International Conference on Document Analysis and Recognition (ICDAR)
- 2015 ConferenceInsights on the Use of Convolutional Neural Networks for Document Image Binarization
Joan Pastor Pellicer , Salvador España Boquera , Francisco Julian Zamora Martínez , María José Castro Bleda
International Work-Conference on Artificial Neural Networks (IWANN), 115-126
- 2014 ConferenceFirst steps towards Skipping NNLMs
Palacios-Corella, A. , Francisco Julian Zamora Martínez , Salvador España Boquera , María José Castro Bleda
International Conference IberSPEECH (Formerly Jornadas en Tecnología del Habla), 111-118
- 2014 ConferenceHandwriting Normalization by Zone Estimation using HMM/ANNs
Joan Pastor Pellicer , Salvador España Boquera , Francisco Julian Zamora Martínez , María José Castro Bleda
International Conference on Frontiers in Handwriting Recognition (ICFHR), 633-638
- 2014 JournalHuman or Computer Assisted Interactive Transcription: Automated Text Recognition, Text Annotation, and Scholarly Edition in the Twenty-First Century
María José Castro Bleda , Torres, Juan Miguel Vilar , Salvador España Boquera , Llorens, D. , Marzal Varó, Andrés , Prat, F. , Francisco Julian Zamora Martínez
Mirabilia (Online), 18(1), 247-253
- 2014 JournalNeural network language models for off-line handwriting recognition
Francisco Julian Zamora Martínez , Frinken, V. , Salvador España Boquera , María José Castro Bleda , Fischer, A. , Bunke, Horst
PATTERN RECOGNITION, 47(4), 1642-1652
- 2013 JournalF-Measure as the Error Function to Train Neural Networks
Joan Pastor Pellicer , Francisco Julian Zamora Martínez , Salvador España Boquera , María José Castro Bleda
Lecture Notes in Computer Science, 7902, 376-384
- 2013 JournalNeural network language models to select the best translation
Khalilov, Maxim , Fonollosa, José A.R. , Francisco Julian Zamora Martínez , María José Castro Bleda , Salvador España Boquera
Computational Linguistics in the Netherlands Journal, 3, 217-233
- 2012 ConferenceLong-Short Term Memory Neural Networks Language Modeling for Handwriting Recognition
Frinken, Volkmar , Francisco Julian Zamora Martínez , Salvador España Boquera , María José Castro Bleda , Fischer, Andreas , Bunke, Horst
International Conference on Pattern Recognition (ICPR), 701-704
- 2012 ConferenceCache Neural Network Kanguage Models based on Long-Distance Dependencies for a Spoken Dialog System
Francisco Julian Zamora Martínez , Salvador España Boquera , María José Castro Bleda , Mori, R. De
IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 4993-4996