181. Using dependency analysis to improve question classification tài liệu, giáo án, bài giảng , luận văn, luận án, đồ á...
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/266908659 Using Dependency Analysis to Improve Question Classification Conference Paper · October 2014 DOI: 10.1007/978-3-319-11680-8_52 CITATION READS 229 1 author: Le-Hong Phuong Vietnam National University, Hanoi 33 PUBLICATIONS 166 CITATIONS SEE PROFILE All content following this page was uploaded by Le-Hong Phuong on 15 October 2014 The user has requested enhancement of the downloaded file All in-text references underlined in blue are added to the original document and are linked to publications on ResearchGate, letting you access and read them immediately Using Dependency Analysis to Improve Question Classification Phuong Le-Hong1 , Xuan-Hieu Phan2 , and Tien-Dung Nguyen3 University of Science Vietnam National University, Hanoi phuonglh@vnu.edu.vn University of Engineering and Technology Vietnam National University, Hanoi hieupx@vnu.edu.vn FPT Research, FPT Corp., Hanoi, Vietnam dungnt64@fpt.com.vn Abstract Question classification is a first necessary task of automatic question answering systems Linguistic features play an important role in developing an accurate question classifier This paper proposes to use typed dependencies which are extracted automatically from dependency parses of questions to improve accuracy of classification Experiment results show that with only surface typed dependencies, one can improve the accuracy of a discriminative question classifier by over 8.0% on two benchmark datasets Introduction Question answering (QA) has been an important line of research in natural language processing in general and in human-machine interface in particular The ultimate goal of a QA system is to provide a concise and exact answer to a question asked in a natural language For example, the answer to the question “What French city has the largest population? ” should be “Paris” Open-domain QA is a challenging task because the research and validation of a precise answer to a question require a good understanding of the question itself and of the text containing the potential answer to the question Typically, we need to carry out both syntactic and semantic analysis in order to fully understand a question and pinpoint an answer This is much more difficult than common information retrieval task, where one only needs to present a ranked list of documents in response to a question, which can be efficiently performed by available search engines The first step of understanding a question is to perform question analysis Question classification is an important task of question analysis which detects the answer type of the question Question classification helps not only filter out a wide range of candidate answers but also determine answer selection strategies In the example question above, if one knows that the answer type is city, one Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen can restrict candidate answers as cities instead of consider every noun phrase of a document providing the answer At first glance, one may think that question classification can be framed as a text classification task However, there exists characteristics of question classification that distinguish it from the common task First, a question is relatively short and contains less word-based information than an entire text Second, a short question needs a deeper-level analysis to reveal its hidden semantics Therefore, application of text classification algorithms per se to question classification could not result in a good result Furthermore, natural language is inherently ambiguous; the question classification is not trivial, especially for what and which type questions For example, “What is the capital of France? ” is of location (city) type, while “What is the Internet of things? ” is of definition type Consider also examples: (1) What tourist attractions are there in Reims? (2) What most tourists visit in Reims? (3) What are the names of the tourist attractions in Reims? (4) What attracts tourists to Reims? (5) What is worth seeing in Reims? [1]; all these questions are of the same answer type location Different wording and syntactic structures make it difficult for classification [2] This paper focuses on the question classification task The main contribution of the paper is to show that accuracy of question classification can be improved by using dependency analysis of questions: even with two simple typed dependencies extracted from dependency parses, we can improve the accuracy of question classifiers on two different question classification datasets by over 8.0%, a significant improvement, given that no hand-crafted rules are required as in many existing works The rest of this paper is structured as follows Section gives an overview of existing works on question classification Section presents briefly some backgrounds for the work, including two classification models in use, namely naive Bayes models and maximum entropy models, which are typical instances of generative and discriminative classifiers, respectively; and dependency parsing of natural languages Section describes datasets, experimental results and discussions Finally, Section concludes the paper Related Works Early works in question classification used rule-based approaches to map a question to a type, which require expert labor for manually constructing rules This makes rule-based approaches not only inefficient in maintain but also difficult to upgrade or port to different domains With the increasing popularity of statistical approaches to natural language processing in general and to question classification in particular, recent years have seen many machine learning approaches which have been applied to question classification problem The main advantage of machine learning approaches is that one can learn a statistical model using useful features extracted from a sufficiently large set of labeled questions and then use it to automatically Using Dependency Analysis to Improve Question Classification classify new questions In this section, we summarize existing machine learning approaches to question classification and their results Li and Roth [3] developed the first machine learning approach to question classification which uses the SNoW learning architecture They have created the UIUC question classification dataset4 containing 5,952 manually labeled questions of coarse-grained classes and 50 fine-grained classes (see Table 2) Using the feature set of lexical words, part-of-speech tags, chunks and named entities, they achieved 78.8% of accuracy for 50 fine-grained classes When augmented with a hand-built dictionary of semantically related words, they were able to reach 84.2% of accuracy.5 Table Accuracy of question classifiers on the UIUC dataset Model classes 50 classes Li and Roth, SNoW – 78.8% Zhang and Lee, Linear SVM 87.4% 79.2% Zhang and Lee, Tree SVM 90.0% – Hacioglu and Ward, SVM+ECOC – 82.0% Krishnan et al., SVM+CRF 93.4% 86.2% Nguyen et al., ST-Boost+ME 91.2% 83.6% Huang et al., Linear SVM 93.4% 89.2% Huang et al., MaxEnt 93.6% 89.0% The UIUC dataset has inspired many follow-up works on question classification Zhang and Lee [4] used linear support vector machines (SVM) with all question n-grams and obtained 79.2% of accuracy Hacioglu and Ward [5] used linear SVM with question bigrams and error-correcting codes and achieved 82.0% of accuracy Krishnan et al [6] also used linear SVM with contiguous subsequence of question words detected by a Conditional Random Field (CRF) and achieved 86.2% of accuracy on the UIUC dataset over fine-grained question types Li and Roth [1] used more syntactic and semantic features including chunks, named entities, WordNet senses, class-specific related words and distributional similarity and obtained 89.3% of accuracy.6 Nguyen et al [7] used maximum entropy and boosting models and achieved 83.6% of accuracy Most recently, Huang et al [2] used SVM and maximum entropy models with question head words and their hypernyms and obtained 89.2% of accuracy, which is the highest reported accuracy on this dataset Table shows the summary of classification accuracy of all models which were tested on the UIUC dataset Available at http://cogcomp.cs.illinois.edu/Data/QA/QC/ However, follow-up works did not use this dictionary Nevertheless, their model was applied on a larger dataset comprising of 21, 500 training questions and 1, 000 test questions 4 Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen Table Distribution of question types in the UIUC dataset Category # Train # Test Category # Train # Test ABBREVIATION 86 term 93 abb 16 vehicle 27 exp 70 word 26 DESCRIPTION 1162 138 HUMAN 1223 65 definition 421 123 group 47 274 individual 189 55 description manner 276 title 962 reason 191 description 25 ENTITY 1250 94 LOCATION 835 81 animal 112 16 city 129 18 body 16 country 155 40 10 mountain 21 color creative 207 other 464 50 currency state 66 dis.med 103 NUMERIC 896 113 event 56 code food 103 count 363 instrument 10 date 218 47 lang 16 distance 34 16 money 71 letter other 217 12 order plant 13 other 52 12 product 42 period 27 percent 75 religion sport 62 speed substance 41 15 size 13 11 temp symbol technique 38 weight 11 Background In this study, we employ both generative and discriminative learning approaches for question classification, typified by two common machine learning models: naive Bayes model and maximum entropy model [8] For ease of exposition, in this section, we first briefly present these two classification models We then give a short introduction of dependency analysis of natural languages for those who are not familiar with this particular topic 3.1 Naive Bayes Classifier Generative models in general and Naive Bayes (NB) models in particular learn a model of the joint probability P (x, y) of the observation x ∈ X and the label y ∈ Y, and make their predictions by using Bayes rules to calculate the posterior P (y| x), then choosing the most probable label y Let ={0, 1}D be Using Dependency Analysis to Improve Question Classification the D-dimensional input space, let the output labels Y = {1, 2, , K} and let D = {(x1 , y1 ), , (xN , yN )} be a training set of N independent and identically distributed examples, where xi = (xi1 , xi2 , , xiD ) ∈ X The generative Bayes classifier is parameterized as follows:7 θk = P (y = k), ∀k = 1, 2, , K θj|k = P (xj = 1|y = k), ∀j = 1, 2, , D; ∀k = 1, 2, , K It uses D to calculate the maximum likelihood estimates θˆj|k and θˆk as follows:8 N i=1 δ(yi = k) , N N i=1 δ(xij = and yi = k) + α θˆk = θˆj|k = N i=1 δ(yi = k) + αK , where δ(·) is the identity function δ(b) = 1, 0, if b = true, if b = false Using the naive Bayes assumption of independent features, the posterior probability is computed as P (y = k| x) = = P (x |y = k)P (y = k) P (x) D j=1 P (xj = 1|y = k)P (y = k) P (x) = D j=1 θj|k θk P (x) Since y does not depend on P (x), the classification rule for an observation x is simply D y = arg max P (y = k| x) = arg max k=1,2, ,K Or, by using the logarithmic transformation: D y = arg max k=1, ,K θj|k θk k=1,2, ,K j=1 j=1 log θj|k + log θk In the question classification problem, each question is represented by an observation x, which is a binary feature vector representing the presence or absence of particular features, for instance the presence or absence of words (unigram features) This is indeed the Bernoulli NB model, not a multinomial NB model We use Laplace smoothing technique with constants fixed at α = 6 Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen 3.2 Maximum Entropy Classifier Maximum Entropy (ME) models (a.k.a multinomial logistic regression model) is a general purpose discriminative learning method for classification and prediction which has been succesfully applied to many problems of natural language processing In contrast to generative classifiers, discriminative classifiers model the posterior P (y| x) directly One of the main advantages of discriminative models is that one can integrate many heterogeneous features for prediction, which are not necessarily independent Each feature corresponds to a constraint on the model In ME models, the conditional probability of a label y given an observation x is defined as P (y| x) = exp(θ · f (x, y)) , y∈Y exp(θ · f (x, y)) where f (x, y) ∈ RD is a real-valued feature vector9 , Y is the set of labels and θ ∈ RD is the parameter vector to be estimated from training data This form of distribution corresponds to the maximum entropy probability distribution satisfying the constraint that the empirical expectation of each feature is equal to its true expectation in the model: ˆ j (h, t)) = E(fj (h, t)), E(f ∀j = 1, 2, , D The parameter θ ∈ RD can be estimated using iterative scaling algorithms or some more efficient gradient-based optimization algorithms like conjugate gradient or quasi-Newton methods [9] In this paper, we use the L-BFGS optimization algorithm and L2 -regularization technique to estimate the parameters of the ME models, with smooth term is fixed at 3.3 Dependency Analysis Constituency structure and dependency structure are two types of syntactic representation of a natural language sentence While a constituency structure represents a nesting of multi-word constituents, a dependency structure represents dependencies between individual words of a sentence The syntactic dependency represents the fact that the presence of a word is licensed by another word which is its governor In a typed dependency analysis, grammatical labels are added to the dependencies to mark their grammatical relations, for example subject or indirect object Recently, there have been many published works on syntactic dependency analysis both for well-studied languages, such as English [10] or French [11], and for less-studied ones like Vietnamese [12,13] It has been shown that syntactic dependencies are useful for semantic dependency analysis, where semantic dependencies are understood in terms of predicates and their arguments, which aimed at natural language understanding applications However, feature vectors of large scale ME models are typically sparse binary ones, indicating the presence or absence of corresponding features Using Dependency Analysis to Improve Question Classification root pobj nsubj attr det det Who was the prep prophet amod of the Muslim people Fig Dependency analysis of an English sentence Our motivation of using dependency analysis in question classification stems from the idea that a question can be classified more exactly if the meaning of the question can be determined at some level, even at a surface one Recently, it has been shown that dependency structure of a sentence can be used to automatically learn its semantics [14] Figure shows the dependency analysis of a question in the UIUC dataset “Who was the prophet of the Muslim people? ”, represented by using the Stanford Dependency scheme [15] Intuitively, if we know that the subject of the question is prophet and its semantically associated word people – the head of the following prepositional phrase, then it is easier to classify the question Similarly, consider the question “What U.S state lived under six flags? ” whose dependency structure is shown in Figure 2; knowledge of the subject U.S state and the prepositional object of the question flags is useful for prediction of its category Experimental Results 4.1 Datasets The experiments reported in this work were conducted on two datasets: the UIUC question classification dataset presented previously and a Vietnamese question classification dataset created by FPT Research10 in an ongoing larger project whose aim is to develop an open domain question answering system For the UIUC dataset, we use the standard split of training and test set as was used in previous studies: the training set is composed of 5,500 questions and the test set is composed of 500 questions from TREC 10 [16] The FPT question classification dataset is currently composed of 1,000 questions containing 13 coarse categories which focus on questions about the FPT Corporation The complete list of categories is: ACTION (action), CONC (concept), DESC (description), DTIME (datetime), EVT (event), HUM (human), LOC (location), NET (internet), NUM (number), ORG (organization), OTHER (other), THG (thing), YESNO (yes/no) Since each coarse-grained category can contain an overlapping set of fined-grained categories, in this study we report only the results of this dataset on the coarse-grained categories Since this dataset has a relatively small size, we not split it into fix training and test sets but 10 http://tech.fpt.com.vn/ Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen root det pobj nn What U.S nsubj state prep lived under num six flags Fig Dependency analysis of another English sentence use a 5-fold cross-validation technique for evaluation The average accuracy is then computed 4.2 Feature Sets In this subsection, we present the binary feature sets that are used in the experiments, namely question wh-word, unigrams and typed dependencies Question wh-words The wh-word feature is the question wh-word in a given question For example, the wh-word of question Which city has the oldest relationship as a sister city with Los Angeles? is what We have used question wh-words for English, including what, who, when, which, where, why and how, and 12 question wh-words for Vietnamese, including g`ı, ai, , bao nhiˆeu Unigrams Unigrams are bag-of-word features which take into account all single words of a question N-gram features in general and unigram features in particular play an important role in question classification since they provide word sense for question.11 We have adopted the same unigram selection method as presented in [17] as they can give better performance while reducing the size of the feature space In particular, – All numbers are converted to the same feature, say, for example “1960 ”, “1972 ”, “401 ” are changed to “1000 ”; – Auxiliary verbs are changed to their infinitive forms More precisely, all the tokens “am”, “is”, “are”, “was”, “were”, and “been” are converted to “be”; all the tokens “does”, “did ”, “don”, “done” are converted to “do”; and “has”, “had ” are converted to “have” Typed dependencies As discussed in the previous section, we propose to use typed dependencies extracted from the dependency analysis of a question as features We are interested in subjects and prepositional objects which can be automatically extracted using nsubj and pobj dependencies For instance, the typed dependency features of the question in Figure are nsubj=prophet 11 Note that stop words are also important for question classification, in contrast to other problems like text categorization Using Dependency Analysis to Improve Question Classification Table Accuracy using individual feature sets on the UIUC dataset Feature Set class NB ME wh-words 46.20 46.20 unigrams 71.20 85.00 50 class NB ME 46.80 46.80 16.00 77.00 and pobj=people and those of the question in Figure are nsubj=state and pobj=flags To generate all typed dependencies of a sentence, we adopt the Stanford Parser [18] to parse all the questions of the UIUC dataset and use a modified version of vnLTAGParser [13] to parse all the questions of the FPT dataset In addition, Vietnamese questions are tokenized using vnTokenizer, a highly accurate word segmentation tool of Vietnamese [19] 4.3 Results We first compute the accuracy of NB and ME models using individual feature sets for coarse and 50 fine classes on the UIUC dataset All the models are trained on the training set of 5,500 questions and tested on the test set of 500 questions The results are shown in Table Some interesting remarks can be drawn from these results First, we can achieve a reasonable accuracy by using only wh-words There is no difference in accuracy between NB models and ME models They give 46.20% and 46.80% of accuracy on the coarse-grained and fine-grained categories respectively Also, the wh-word feature set produces very compact models which have only features Second, unigrams give significantly higher accuracy over wh-words, nevertheless the improvement is far better in coarse-grained classification than in fine-grained classification Third, discriminative models are much more superior than generative models on the unigram feature set, which includes a large number of features Table Accuracy using incremental feature sets on the UIUC dataset Feature Set class NB ME wh-words + deps 49.00 59.20 unigrams + deps 66.80 87.60 50 class NB ME 17.80 61.60 11.40 78.40 Table shows the accuracy of the models when typed dependency features are included It is shown that typed dependencies are very informative features for question classification Integration of these features helps improve largely the accuracy of the ME classifier on both of the feature sets, in coarse-grained or fine-grained classification, with a net average improvement of 7.5% for coarsegrained categories and 8.1% for fine-grained respectively However, in general, 10 Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen Table Accuracy using different feature sets on the FPT dataset Feature Set wh-words unigrams wh-words + deps unigrams + deps NB 47.50 57.60 59.70 58.80 ME 51.50 78.40 69.60 80.50 typed dependencies make NB models perform worse than without using them This fact can be explained by the assumption of independence of features in NB models, which would treat additional features as noisy information, especially when the domain dimension of the classification problem at hand is large, as in the case of using unigram features Table reports the accuracy of the models on the FPT dataset using individual feature sets (first half) and incremental feature sets (second half) It seems that the improvement when integrating typed dependencies is more significant for Vietnamese than for English The ME models give an average improvement of 10.1% and that of naive Bayes models is about 5.5% These results also demonstrate a net benefit of typed dependencies in the question classification for Vietnamese, a language of different family from English 4.4 Discussion The idea of using syntactic parsing to improve the performance of question classification is not new For example, Nguyen et al [7] proposed to use subtrees extracted from the constituency parses of questions in a boosting model with ME classifier and achieved the accuracy of 91.2% and 83.6% for coarse-grained and fine-grained categories respectively However, this approach not only makes use of very rich feature space (the generated subtrees can be extremely numerous) but also employs a sophisticated technique for subtree selection for boosting In contrast, we propose a compact yet effective feature set of typed dependencies extracted from dependency parses of questions to achieve a good improvement of classification accuracy Huang et al [2] proposed to use head words and their hypernyms as important features to achieve the accuracy of 89.00% over the UIUC dataset Head word is one single word specifying the object that the question seeks For example, in the example question: What is a group of turkeys called?, the head word is turkeys To obtain the head word, they have to go through a quite complicated procedure First the Berkeley parser is used to get the constituency parse of a question; then a modified version of Collins head rules is used to extract the semantic head word; finally, a set of manually compiled tree patterns and regular expression patterns is used to re-assign the head word resulting from the previous step so as to fix it if necessary For example, the initial head word extracted from the question What is the proper name for a female walrus? is name should be fixed to walrus since it matches a pre-compiled tree pattern It is interesting to note that our Using Dependency Analysis to Improve Question Classification 11 approach of using typed dependencies is able to automatically identify the correct head word without having to go through the complex procedure as discussed above In particular, the dependencies extracted from the questions above are pobj=turkeys; and nsubj=name, pobj=walrus respectively Our approach is thus more general than that of Huang et al and provides a principled, fully automatic way to identify semantic features that are informative for question classification In a recent paper, Tran [20] et al has built a Vietnamese question classification system which employed some machine learning techniques In addition to the bag-of-word features, they use a keyword corpus extracted from the Web Their experiments are carried out on a translated version the UIUC dataset to Vietnamese, which are essentially different from our Vietnamese corpus Therefore, the reported results are not comparable It is not our goal to argue against the use of manually compiled features in high-performance question classification It has been demonstrated that head words and their hypernyms are useful in resolving classes of questions, and a question classifier should make use of such information where possible We focus here on not only using automatically extracted features, but also trying to improve the performance of question classification for less-resource languages like Vietnamese where semantic information source like WordNet is not currently available We see this investigation as only one part of the foundation for state-of-the-art question classification We believe that the integration of typed dependencies, manually compiled regular expressions and word hypernym features will result in high-performance question classifiers for natural languages This is a direction of our future work Finally, semi-supervised learning for question classification has been shown to provide promising results [17] We plan to investigate this approach with additional dependency features and a boosting technique to improve further the results This line of research is another direction for our future work Conclusion In contrast to many existing approaches for question classification which make use of very rich feature space or hand-crafted rules, we propose a compact yet effective feature set In particular, we propose to use typed dependencies as semantic features We have shown that by integrating only two simple dependencies of type nominal subject and prepositional object, one can improve the accuracy of question classification by over 8.0% using common statistical classifiers over two benchmark datasets, the UIUC dataset for English and a recently introduced FPT question dataset for Vietnamese With unigram feature and typed dependency feature, one can obtain accuracy of 87.6% and 80.5% using maximum entropy for the UIUC and FPT question dataset respectively Acknowledgment This work is partly supported by the FPT Technology Research Institute We are grateful to anonymous reviewers for helpful comments on the draft 12 Phuong Le-Hong, Xuan-Hieu Phan, and Tien-Dung Nguyen References Li, X., Roth, D.: Learning question classifiers: the role of semantic information Natural Language Engineering 12(3) (2006) 229–249 Huang, Z., Thint, M., Qin, Z.: Question classification using head words and their hypernyms In: Proceedings of the 2008 Conference on EMNLP (2008) 927–936 Li, X., Roth, D.: Learning question classifiers In: Proceedings of COLING (2002) 556–569 Zhang, D., Lee, W.S.: Question classification using support vector machines In: Proceedings of the 26th ACM SIGIR (2003) 26–32 Hacioglu, K., Ward, W.: Question classification with support vector machines and error correcting codes In: Proceedings of NAACL/HLT (2003) 28–30 Krishnan, V., Das, S., Chakrabarti, S.: Enhanced answer type inference from questions using sequential models In: Proceedings of EMNLP (2005) 315–322 Minh, N.L., Thanh, N.T., Akira, S.: Subtree mining for question classification problem In: Proceedings of IJCAI (2007) 1695–1700 Ng, A.Y., Jordan, M.I.: On discriminative vs generative classifiers: A comparison of logistic regression and naive Bayes In: Advances in NIPS (2001) Andrew, G., Gao, J.: Scalable training of l1 -regularized log-linear models In: ICML (2007) 33–40 10 Kbler, S., McDonald, R., Nivre, J.: Dependency Parsing Morgan & Claypool Publishers (2009) 11 Candito, M., Crabb, B., Denis, P., Gurin, F.: Analyse syntaxique du franais : des constituants aux dpendances In: Actes de TALN, Senlis, France (2009) 12 Nguyen, T.L., Ha, M.L., Nguyen, V.H., Nguyen, T.M.H., Le-Hong, P.: Building a treebank for Vietnamese dependency parsing In: The 10th IEEE RIVF, Hanoi, Vietnam, IEEE (2013) 147–151 13 Le-Hong, P., Nguyen, T.M.H., Azim, R.: Vietnamese parsing with an automatically extracted tree-adjoining grammar In: Proceedings of the IEEE RIVF, HCMC, Vietnam (2012) 14 Liang, P., Jordan, M.I., Klein, D.: Learning dependency-based compositional semantics Computational Linguistics 39(2) (2012) 389–446 15 de Marneffe, M.C., MacCartney, B., Manning, C.D.: Generating typed dependency parses from phrase structure parses In: Proceedings of LREC 2006, Italy (2006) 16 Voorhees, E.M.: Overview of TREC 2001 question answering track In: Proceedings of The Tenth Text REtrieval Conference, NIST (2001) 17 Thanh, N.T., Minh, N.L., Akira, S.: Using semi-supervised learning for question classification Journal of Natural Language Processing 3(1) (2008) 112–130 18 Klein, D., Manning, C.D.: Accurate unlexicalized parsing In: Proceedings of ACL, Sapporo, Japan (July 2003) 423–430 19 Le-Hong, P., Nguyen, T.M.H., Roussanaly, A., Ho, T.V.: A hybrid approach to word segmentation of Vietnamese texts In: Proceedings of LATA, LNCS 5196, Springer (2008) 20 Tran, D.H., Chu, C.X., Pham, S.B., Nguyen, M.L.: Learning based approaches for Vietnamese question classification using keywords extraction from the web In: Proceedings of the IJCNLP, Nagoya, Japan (2013) 740–746 View publication stats ... from a sufficiently large set of labeled questions and then use it to automatically Using Dependency Analysis to Improve Question Classification classify new questions In this section, we summarize... is to show that accuracy of question classification can be improved by using dependency analysis of questions: even with two simple typed dependencies extracted from dependency parses, we can improve. .. should be fixed to walrus since it matches a pre-compiled tree pattern It is interesting to note that our Using Dependency Analysis to Improve Question Classification 11 approach of using typed dependencies