2016 15th International Conference on Frontiers in Handwriting Recognition

A Candidate Lattice Refinement Method

for Online Handwritten Japanese Text Recognition

Jianjuan Liang, Bilan Zhu and Masaki Nakagawa

Department of Computer and Information Sciences
Tokyo University of Agriculture and Technology
Tokyo, 184-8588, Japan
[email protected], {zhubilan, nakagawa}@cc.tuat.ac.jp

Abstract— This paper presents a candidate lattice refinement method has widely been used to overcome the ambiguity of
method for online handwritten Japanese text recognition. In character segmentation [1], which is classified into the
the integrated segmentation-recognition framework, we first segmentation-free method and the over-segmentation-based
over-segment a character string pattern into primitive method [2]. The over-segmentation-based method can utilize
segments at least at their true boundaries so that each the character shapes into recognition, therefore, it can be
primitive segment may compose a single character or a part of successfully applied into handwritten Chinese/Japanese text
a character. Then a candidate lattice is constructed based on recognition [3] [4].
the primitive segments. We search within the candidate lattice In the over-segmentation-based recognition framework, a
to obtain the optimal path as recognition result. In striving for
handwritten text (character string) is first over-segmented
high recognition accuracy, however, the approach must
generate many candidate lattice nodes, which ultimately
into primitive segments, consecutive segments are
increase the recognition time. To solve this problem, we refine concatenated into candidate character patterns. Each
the candidate lattice to eliminate unnecessary nodes before candidate pattern is associated with a number of candidate
path search and text recognition. For the refinement, we classes and confidence scores by a character classifier. Then,
evaluate all segmentation hypotheses by combining the a segmentation-recognition candidate lattice is constructed
probability of a character verifier using noncharacter samples, by combining all candidate patterns and character classes.
the class-independent unary and binary geometric context, as Finally, by integrating the scores of character recognition,
well as character segmentation. We retain N-best paths by geometric context and linguistic context, an optimal
beam search to reduce the complexity of the candidate lattice. segmentation-recognition path is searched from the
Experiments on horizontal text lines extracted from the constructed lattice by Viterbi search.
Kondate database show that the proposed method keeps In over-segmentation, we usually keep all possible
recognition accuracy while reducing recognition time to half. segmentation points to guarantee a high recognition rate,
though it complicates the lattice and consequently burdens
Keywords-online handwritten text recognition; integrated the computation of string recognition, especially for the case
segmentation and recognition; lattice refinement; noncharacter of partial overlapped characters [5]. As a result, there are
patterns. many candidate character patterns. Due to thousands of
Japanese characters, Zhu et al. [6] proposed a coarse
I. INTRODUCTION classifier for candidate patterns before the fine classification
Online handwritten text recognition has been receiving to speed up the online handwritten Japanese characters
large attention due to the development and growing recognition.
popularity of pen-based input devices with large writing Since candidate patterns include true character and
areas, such as tablet PCs, electronic whiteboards and digital noncharacter patterns in the lattice, the character classifier or
pens (e.g., Anoto pen). Using these devices, people tend to path evaluation should be resistant to noncharacter patterns.
write text continuously with little constraints. To meet the Liu et al. [1] evaluated several classifiers for candidate
needs of real applications, however, there is still further work patterns with noncharacter training in the context of
needed to improve the performance of handwritten text handwritten numeral strings, and showed that training with
recognition. noncharacter samples improves neural classifiers and support
Due to the variable space between characters and the fact vector classifiers. However, noncharacter training shows
that many characters are composed of multiple radicals with limited influence on the performance of discriminative
internal gaps in the languages of Chinese origin, online density models: the learning vector quantization classifier
handwritten Japanese text recognition faces the difficulty of and the discriminative learning quadratic discriminant
character segmentation compared to isolated character function classifier.
recognition. The integrated segmentation and recognition Moreover, Li et al. [7] proposed a probabilistic model to
evaluate segmentation hypotheses of a text line by

2167-6445/16 $31.00 © 2016 IEEE 216

DOI 10.1109/ICFHR.2016.46
combining an isolated character recognizer and a character II. OVERVIEW OF OUR RECOGNITION SYSTEM
verifier. The character verifier evaluates candidate patterns For online handwritten Japanese text recognition, our
based on the modified quadratic discriminant function recognition system has four major steps: over-segmentation,
(MQDF) classifier using noncharacter samples. This method candidate lattice construction, candidate lattice refinement
works well for handwritten offline Chinese text line and path search, as shown in Fig. 1. The input is a
recognition. handwritten text line composed of a sequence of strokes.
On the other hand, many studies have focused on the We define an off-stroke is a vector from the last point of
postprocessing of geometric context to reject noncharacter a previous stroke to the first point of the next stroke. Each
patterns. The geometric context, such as character size, processing step will be described in detail in the following
character unary and binary position, disambiguates the subsections.
uncertainty of character segmentation. The geometric context
models, which are integrated with a character classifier and A. Over-segmentation
linguistic context, improve the recognition performance in In over-segmentation, we use two-stage classification
handwritten Japanese text recognition [8], in handwritten scheme to minimize misclassification of each off-stroke.
Chinese text recognition [9] and in handwritten numeral A segmentation point separates two characters at the off-
string recognition [10]. stroke, while a non-segmentation point indicates the off-
In this study, based on the system of Zhu et al. [3] stroke is within a character. An undecided point is
(hereafter we call it the initial system), we focus on refining interpreted as either segmentation point or non-segmentation
the constructed candidate lattice for handwritten Japanese point.
text recognition to improve recognition speed. The proposed The group of consecutive strokes between two adjacent
recognition method can be resistant to noncharacter patterns. segmentation/undecided points is a primitive segment, and
In the lattice refinement, we evaluate segmentation paths in one or more consecutive primitive segments form a
the lattice by combining the information of the candidate candidate character pattern.
pattern verifier and class-independent geometric context, as In the first stage, each off-stroke is classified into two
well as character segmentation. A candidate pattern is classes: non-segmentation point and hypothetical
verified whether it is true character or not by an SVM segmentation point, according to geometric features. To let
classifier using noncharacter samples. Then, we maintain N- hypothetical segmentation points include true segmentation
best segmentation paths by beam search for normal text points as much as possible, we adjust the threshold. We can
recognition. We search within the refined candidate lattice to get strict and loose over-segmentation, as shown in Fig. 2.
obtain an optimal segmentation-recognition path by Viterbi We generally choose loose over-segmentation to take the
search. Experiments on horizontal Japanese text lines accuracy unless speed is required.
extracted from the Kondate database [11], demonstrate the In the second stage, we select some hypothetical
effectiveness of our lattice refinement. segmentation points as segmentation points using an SVM
The rest of this paper is organized as follows: Section ϩ classifier on 11 geometric features extracted from an off-
gives an overview of our handwritten Japanese text stroke, as shown in Table Ϩ. The remaining hypothetical
recognition system. Section Ϫ describes the lattice segmentation points are set as undecided points.
refinement. Section ϫ shows the results. Finally, section Ϭ Table ϩ shows terminologies to derive geometric
concludes with our remarks. features, where acs is estimated by measuring the longer side
length of the bounding box of each stroke, sorting the lengths
Inputatext line of all the strokes and taking the average of the larger 1/3 of
them.
Oversegmentation
segmentation point
undecided point

Candidatelatticeconstruction

Characterverifier
Candidatelatticerefinement Classindependent
geometriccontext (a)
Characterrecognition
Pathsearchandrecognition Linguisticcontext
Classdependent
geometriccontext (b)
Outputtheoptimalresultstring
Figure 2. Over-segmentation of the same text line: (a) strict case, (b)
Figure 1. Flow of handwritten Japanse text recognition system. loose case.

217
 TABLE I. FEATURES EXTRACTED FROM AN OFF-STROKE. s1 s2 s3s4 s5 s6 s7 s8 s9 s10 s11 s12

No. Definition
f1 DBx / acs s1 s2 s3s4s5 s6 s7 s8 s9 s10 s11 s12
Start
f2 , f3 Dbx / acs, Dby / acs
f4 , f5 Dbsx / acs, Dbsy / acs
f6 Dbs / acs
f7 Loff / acs
f8 , f9 sine(Loff), cosine(Loff)
f10 DBx / the maximu DBx in text line
f11 D / acs

End
TABLE II. TERMS TO DERIVE FEATURES.

Term. Description Figure 4. Refined lattice for the loose case of the candidate lattice as
shown in Fig. 3(b).
acs Average character size of text line
Bbp Bounding box of immediately preceding stroke
Bbs Bounding box of immediately succeeding stroke C. Candidate lattice refinement
BBp_all Bounding box of all preceding strokes
Due to many noncharacter patterns in the constructed
BBs_all Bounding box of all succeeding strokes
candidate lattice, especially for the loose over-segmentation,
Bounding box of primitive segment which includes
BSp
immediately preceding stroke
noncharacters complicate the computation of text recognition.
Bounding box of primitive segment which includes To speed up the text recognition, we keep only N-best
BSs segmentation paths by beam search with combining the
immediately succeeding stroke
DBx Distance between BBp_all and BBs_all in x-axis probability of a character verifier using noncharacter samples,
Dbx Distance between Bbp and Bbs in x-axis the class-independent unary and binary geometric context, as
Dby Distance between Bbp and Bbs in y-axis well as character segmentation. We use the maintained N-
Dbsx Distance between centers of Bbp and Bbs in x-axis best segmentation paths to reconstruct a downsized candidate
Dbsy Distance between centers of Bbp and Bbs in y-axis lattice with few candidate character patterns, as shown in Fig.
Dbs Distance between centers of Bbp and Bbs 4, that is refined from the lattice as shown in Fig. 3 (b) where
D Distance between in BSp and BSs in x-axis the thickly marked path is the correct segmentation path. Sec.
Loff Length of off-stroke Ϫdescribes the processing of lattice refinement in detail.

For SVM classification, we set the target value of D. Path search and recognition
segmentation point as 1 and that of non-segmentation point Each candidate character pattern in the refined candidate
as -1, and train it using training patterns of off-strokes. lattice, is associated with a number of candidate classes with
confidence scores by character classification. Then, all
B. Candidate lattice construction retained segmentation paths and recognition candidate
The combination of all possible candidate character classes are represented by a segmentation-recognition
patterns is represented by a candidate lattice, as shown in Fig. candidate lattice.
3, where each node (rectangle) denotes a candidate character We utilize the path evaluation criterion proposed by Zhu
pattern. Each path from start segmentation point to end one et al. [3] to re-evaluate paths in the refined candidate lattice
in the lattice denotes a segmentation hypothesis for a and search for the optimal string result by the Viterbi
handwritten text line. algorithm. This criterion combines the scores of character
s1 s2 s3 s4 s5 s6 s7 s8 s9 s1 s2 s3s4 s5 s6 s7 s8 s9 s10 s11 s12
recognition, character size, inner gap, single-character
position, pair-character position, and linguistic context, as
s1 s2 s3s4 s5 s6 s7 s8 s9 s1 s2 s3s4s5 s6 s7 s8 s9 s10 s11 s12
Start Start well as character segmentation, with weighting parameters
estimated by the genetic algorithm.
III. CANDIDATE LATTICE REFINEMENT
This section describes the process of candidate lattice
refinement.
End Given a handwritten text pattern, which is over-
segmented into a sequence of candidate character
patterns X x1x2 ,, xn . Using the following evaluation
End
(a) (b) criterion, we can get N-best segmentation paths in the
candidate lattice.
Figure 3. Candidate lattice based on the over-segmentation as shown in
Fig. 2: (a) strict case, (b) loose case.

218
 § ·
¨ O1ki log P(true | xi )  O2 log P ( pi | xi )  ¸
u
n
¨ ¸
  
f ( X) ¦ ¨ O3 log P ( pi | xi 1 , xi )  O4 log P ( g ji | Sb)  ¸  On
b

i 1
¨ ¸
¨ O5 ¦ ji j i1 log P ( g j | Sw)
j  k 1
¸
© i ¹

where n is the number of candidate character patterns in a

path, ki is the number of primitive segments within a
candidate pattern xi , O h (h=1~5) and  are weighting
parameters.
The term piu and pib denotes the unary and binary
geometric feature vector, respectively. To distinguish the
geometric features used in the path search and recognition
step, here, we call them class-independent unary and binary
geometric feature, respectively. Figure 5. Examples of noncharacter patterns in handwritten Japanese text.
The term g j denotes a geometric feature vector
concerning a segmentation point, which is extracted from an The class-independent binary geometric model measures
whether a segmentation point between two adjacent
off-stroke.
primitive segments is a valid between-character gap or not.
The following subsections will describe each probability We also use another SVM classifier trained with true and
in Eq. (1). invalid between-character gap samples.
A. Candidate character verification For these two models, we transform the outputs of the
SVM classifiers to posterior probability by the similar
P(true | xi ) is the probability that a candidate pattern xi is method described in [3].
a true character pattern. We obtain this probability using an
SVM classifier, since there are only two classes: true C. Segmentation of candidate character patterns
character patterns and noncharacter patterns. Both of them P ( g j | Sb) is the probability that the spacing between
i
are extracted from real handwritten text patterns. Figure 5
shows some examples of noncharacter patterns. candidate character patterns (Sb) appears as g ji and P( g j | Sw)
For training the SVM classifier, we extract directional is the probability that spacing within a candidate character
features described in [12]. Since the extracted directional
pattern (Sw) appears as g j .
feature has 512 dimensions, we further use principal
component analysis (PCA) to reduce the feature dimension. Here, a segmentation point is always treated as Sb and an
In SVM classification, we set the target value of true non-segmentation point is always treated as Sw. undecided
character pattern as 1 and that of noncharacters as -1. We use point is interpreted as either Sb or Sw. When it is within a
the similar method described in [3] to transform the output of character pattern, it is treated as Sw, when it is between
the SVM classifier to probability values. character patterns, it is treated as Sb.
B. Class-independent geometric features We approximate these two probabilities using an SVM
classifier, and transform the output of the SVM classifier to
P ( piu | xi ) and P ( pib | xi 1, xi ) are the probability of a probability values. The two probabilities P( g j | Sb) and
i
candidate pattern is a true character, and that of an over-
segmentation gap is a valid between-character gap, P( g j | Sw) are also applied for the path evaluation in the
respectively. refined candidate lattice search to obtain the optimal path as
For class-independent unary geometric model, we extract recognition result.
12 geometric features from a candidate pattern. For class-
independent binary geometric model, we extract 14 IV. EXPERIMENTS
geometric features from two adjacent primitive segments. We evaluate our proposed handwritten Japanese text
These features are the same as in [9].
recognition method on horizontal Japanese text lines
The class-independent unary geometric model measures
whether a candidate pattern is a true character or not. For this extracted from the Kondate database [11], which is collected
two-class problem, we use an SVM classifier trained with from 100 people. We extract text lines from no. 1 to no. 75
the true character and noncharacter samples. people as the training set, and that from no. 76 to no. 100
people as the testing set. In the testing set, the number of
characters in a text line is greater than 5. Table Ϫ shows the
statistics of text lines in the training and testing sets.

219
 TABLE III. STATISTICS OF TEXT LINES. 2
F
1/R  1/P
Text Character Character Characters number of correctly detected segmentation points (2)
Dataset R
lines patterns classes per line number of true segmentation points
Training set 10,174 104,093 1,106 10.23 number of correctly detected segmentation points
P
number of detected segmentation points (including false)
Testing set 2,058 26,863 686 13.05

B. Result and analysis

A. Settings
We compare the proposed system for online handwritten
For the over-segmentation which is used for both the path Japanese text recognition with the initial system on the same
evaluation in the lattice refinement search and that in the testing set. The difference between them is that whether it
refined candidate lattice search, we collect the samples of uses the processing of candidate lattice refinement or not.
two classes (segmentation points and non-segmentation Table ϫ shows the results.
points) from handwritten Japanese text lines of the training From the results, we can see that the character
set as training data, and train the SVM classifier for off- recognition rate of the proposed method can be close to that
strokes using the selected 11 features. of the initial system, with higher speed by keeping 70-best
For the lattice refinement, we collect samples of two segmentation paths. Moreover, the proposed method can
classes (true character patterns and noncharacter patterns) achieve higher character recognition rate of 93.50% by
from handwritten Japanese text lines in the training set. The keeping 100-best segmentation paths.
collection process is as follows: we mark the ground-truth
boundaries of characters in text lines, then, we use the over- TABLE IV. RESULTS OF TEXT LINE RECOGNITION ON THE TESTING SET.
segmentation step to over-segment each text line so that a
Performance
candidate lattice is constructed. For each candidate pattern in Method
N-best Cr (%) F Tav_rec_tl
this lattice, if its boundary is the same as the ground-truth, it
Initial system — 93.30 0.9918 0.267 (s)
is treats as true character pattern, otherwise, it is regarded as
a noncharacter pattern. 1 81.72 0.9470 0.070 (s)
Since there are hundreds of thousands of noncharacter
samples, we use a part of them to train the candidate 3 86.91 0.9668 0.072 (s)
character verifier, and a small part of them to train the class- 5 88.90 0.9745 0.080 (s)
independent unary geometric model. For the class-
independent binary geometric model, we use a small part of 10 90.76 0.9820 0.090 (s)
text lines of the training set to train it. For the character Proposed
30 92.39 0.9882 0.105 (s)
verifier, we reduce the feature dimension from 512 to 10 by
PCA to balance the accuracy and speed. Moreover, we adjust 50 92.92 0.9902 0.114 (s)
empirically the weighting parameters in the Eq. (1).
For the path evaluation in the refined candidate lattice 70 93.01 0.9905 0.122 (s)
search, the isolated character recognizer combines on-line 100 93.50 0.9923 0.126 (s)
and off-line character recognizers by a linear function [13],
where the combining parameters are trained by Nakayosi
database [14], we keep top 10 candidate character classes for
every candidate character pattern. We also use Nakayosi
database to train class-dependent geometric feature
functions: unary and binary position features, size features 䠌䠐䠑䠉䠐䠓䠌䠉䠍䠎䠋䠒
and inner-gap features. (䠌䠐䠑䠉䠐䠓䠌䠉䠍䠎䠍䠒)

As for the linguistic context, we use tri-gram model. It is

trained on the year 1993 volume of the ASAHI newspaper
䠄᭶䠅䠍䠐䠖䠌䠌䛻ᡂ⏣䝍䞊䝭䝘䝹ฟ䝻䠝䛻䛶䛚ᚅ䛱
and the year 2002 volume of the NIKKEI newspaper. We 䠄䠄᭶䠅䠍䠐䠖䠌䠌䛻ᡂ⏣䝍䞊䝭䝘䝹ฟཱྀ䠝䛻䛶䛚ᚅ䛱䠅
estimate the smoothing parameters by Nakayosi database.
As for comparison with the initial system, we use the
same terms to evaluate the performance of text line 䛈䠌䠏䠕䠉䠌䠑䠌䠎㟷᳃┴୕ᡞ㒆▼ᕝ⏫኱Ꮠୗ䝍䝻ஂ஭Ꮠ㟷ᮌ༲䝏␒ᆅ䠅
䠄䛈䠌䠏䠕䠉䠌䠑䠌䠎㟷᳃┴୕ᡞ㒆ྡᕝ⏫኱Ꮠୗྡஂ஭Ꮠ㟷ᰗ䠐␒ᆅ䠍䠅
recognition as follows: the character recognition rate ( Cr ),
which is the percentage of characters which are both
correctly segmented and recognized, the segmentation
measure ( F ) defined in Eq. (2), which combines recall and ᅗ䠑䜸䞁䝷䜲䞁ᯟ䛺䛧ᩥᏐㄆ㆑䞁䝇䝔䝮䝣኱䛝䛛䛺ὶ䜜
䠄ᅗ䠑䜸䞁䝷䜲䞁ᯟ䛺䛧ᩥᏐㄆ㆑䝅䝇䝔䝮䛾኱䜎䛛䛺ὶ䜜䠅
precision rates, and the average recognition time of a text
line ( Tav _ rec _ tl ). Figure 6. Examples of misrecognition of handwritten Japanese text, the
result is under the hanwritten text, and the ground truth is in brackets.
The experiments are made on a PC with Intel(R) Core™
i7 3.40GHz CPU and 8GB RAM.

220
 Figure 6 shows some misrecognition examples of [3] B. Zhu, X.-D. Zhou, C.-L. Liu and M. Nakagawa, “A Robust
handwritten Japanese text. The recognition errors are mainly Model for On-line Handwritten Japanese Text Recognition,”
due to the following reasons: First, the character classifier International Journal on Document Analysis and Recognition
(IJDAR), vol.13, no. 2, pp.121-131, 2010.
cannot give the correct class in the top 10 candidates, Second,
[4] D.-H. Wang, C.-L. Liu, and X.-D. Zhou, “An approach for
the correct segmentation-recognition path cannot be found in real-time recognition of online Chinese handwritten
the path search and text recognition, although the refined sentences,” Pattern Recognition, vol. 45, no. 10, pp.3661–
lattice includes the correct segmentation path. Third, the 3675, 2012.
correct segmentation path cannot be found in N-best paths of [5] J. Liang, B. Zhu, T. Kumagai and M. Nakagawa, “Character-
the lattice refinement. Position-Free On-line Handwritten Japanese Text Recognition
by Two Segmentation Methods,” IEICE Trans. on Information
V. CONCLUSION and Systems, vol. E99-D, no. 4, pp.1172-1181, 2016.
[6] B. Zhu and M. Nakagawa, “A Robust Method for Coarse
In this paper, we proposed a method to refine the lattice Classifier Construction from a Large Number of Basic
for online handwritten Japanese text recognition. In the Recognizers for On-line Handwritten Chinese/Japanese
lattice refinement, we evaluate segmentation hypotheses by Character Recognition,” Pattern Recognition, 47, pp. 685-693,
combining the scores of the character verifier using 2014.
noncharacter samples, the class-independent unary and [7] N.X. Li and L.W. Jin, “A Bayesian-based method of
unconstrained handwritten offline Chinese text Line
binary geometric context, as well as character segmentation. recognition,” International Journal on Document Analysis and
The experiments on horizontal Japanese text lines showed Recognition (IJDAR), vol. 16, no. 1, pp.17-31, 2013.
the effect of the candidate lattice refinement. [8] X.-D. Zhou, C.-L. Liu, and M. Nakagawa, “Online
In the future work, we will do the following works to handwritten Japanese character string recognition using
make our proposed text recognition method robust. First, we conditional random fields,” in Proc. 10th ICDAR, pp. 521-525,
will optimize the parameters in the evaluation criterion of 2009.
lattice refinement by the genetic algorithm. Second, to better [9] F. Yin, Q.-F. Wang, and C.-L. Liu, “Transcript mapping for
verify candidate character patterns, we cluster true characters handwritten Chinese documents by integrating character
recognition model and geometric context,” Pattern
into several super-classes according to some geometric Recognition, vol. 46, no. 10, pp. 2807-2818, 2013.
features such as size and position, then classify them with [10] Y.-C. Wu, F. Yin and C.-L. Liu, “Evaluation of geometric
noncharacters using another classifier such as MQDF. context models for handwritten numeral string recognition,” in
Proc. 14th ICFHR, pp.193-198, 2014.
ACKNOWLEDGMENT [11] T. Matsushita and M. Nakagawa, “A database of on-line
This research is being partially supported by Grant-in-aid for handwritten mixed objects named “Kondate”,” in Proc. 14th
ICFHR, pp.369-374, 2014.
Scientific Research C-15K00225 and NEDO under the
[12] C.-L. Liu and X.-D. Zhou, “Online Japanese character
number 27J1103. recognition using trajectory-based normalization and direction
feature extraction,” in Proc. 10th ICFHR, pp. 217-222, 2006.
[13] B. Zhu, J. Gao and M. Nakagawa, “Objective function design
REFERENCES for MCE-based combination of on-line and off-line character
recognizers for on-line handwritten Japanese text recognition,”
[1] C.-L. Liu, H. Sako and H. Fujisawa, “Effects of classifier in Proc. 11th ICDAR, pp.594-599, 2011.
structures and training regimes on integrated segmentation and [14] M. Nakagawa and K. Matsumoto, “Collection of on-line
recognition of handwritten numeral strings,” IEEE Trans. handwritten Japanese character pattern databases and their
Pattern Analysis and Machine Intelligence, vol. 26, no. 11, analysis,” International Journal on Document Analysis and
pp.1395-1407, 2004. Recognition (IJDAR), vol.7, no.1, pp.69-81, 2004.
[2] B. Zhu and M. Nakagawa, “Online handwritten
Chinese/Japanese character recognition,” Advance in
Character Recognition, InTech, Chapter 3, pp.51-68, 2012.

221