VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY LE HOANG QUYNH MACHINE LEARNING-BASED EXTRACTION OF SEMANTIC RELATIONS FROM BIOMEDICAL LITERATURE DOCTOR OF PHILOSOPHY IN INFORMATION TECHNOLOGY DISSERTATION Hanoi, 2022 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY LE HOANG QUYNH MACHINE LEARNING-BASED EXTRACTION OF SEMANTIC RELATIONS FROM BIOMEDICAL LITERATURE Major: Information Systems Code: 9480104.01 DOCTOR OF PHILOSOPHY IN INFORMATION TECHNOLOGY DISSERTATION SUPERVISORS: Prof Dr Nigel Collier Dr Dang Thanh Hai Hanoi, 2022 Declaration I hereby declare that this Doctoral Dissertation was carried out by me for the degree of Doctor of Philosophy under the guidance and supervision of my supervisors This dissertation is my own work and includes nothing, which is the outcome of work done in collaboration except as specified in the text It is not substantially the same as any I have submitted for a degree, diploma or other qualification at any other university; and no part has already been, or is currently being submitted for any degree, diploma or other qualification Hanoi , January 2022 Author Le Hoang Quynh iii Table of Contents DECLARATION iii TABLE OF CONTENTS iv ABBREVIATIONS viii LIST OF FIGURES xi LIST OF TABLES xiii PREFACE 1 INTRODUCTION TO BIOMEDICAL RELATION EXTRACTION 11 1.1 Problem statement 11 1.1.1 Semantic relation extraction 11 1.1.2 Biomedical named entity recognition 12 1.1.3 Biomedical relation classification 15 1.2 Literature review 19 1.2.1 Literature review of biomedical named entity recognition 19 1.2.2 Literature review of biomedical relation extraction 24 1.2.3 Related doctoral dissertations 29 1.3 Related resources 30 1.3.1 Datasets for named entity recognition experiments 31 1.3.2 Datasets for relation classification experiments 32 1.4 Evaluation metrics 34 1.4.1 Evaluation metrics 34 1.4.2 Named entity recognition evaluation 35 1.4.3 Relation classification evaluation 36 1.5 Summary 37 iv AN END-TO-END PIPELINE MODEL FOR BIOMEDICAL RELATION EXTRACTION 38 2.1 Distant supervision learning with silverCID corpus 39 2.2 Proposed UET-CAM system 42 2.2.1 Joint model of named entity recognition and normalization (DNER) 43 2.2.2 Coreference resolution 49 2.2.3 Intra-sentence relation classification with support vector machine 52 2.3 Experimental results and discussion 54 2.3.1 Choosing the combining manner of SSI and skip-gram for named entity normalization results 54 2.3.2 Named entity recognition and normalization results 55 2.3.3 CID relation classification results 57 2.3.4 Discussion 58 2.4 Summary 62 AN IMPROVED CRF-BILSTM MODEL FOR BIOMEDICAL NAMED ENTITY RECOGNITION 64 3.1 Introduction to deep learning for named entity recognition 65 3.2 Proposed D3NER model 67 3.2.1 Data pre-processing 67 3.2.2 The TPAC embeddings layer 68 3.2.3 Context representing biLSTM layer 71 3.2.4 Project layer 72 3.2.5 Conditional random fields layer 72 3.3 Experimental results and discussion 72 3.3.1 Experimental environment and model settings 73 3.3.2 Comparative models 75 3.3.3 The performance of D3NER model and comparisons 76 3.3.4 Contribution of the model components 80 3.3.5 Error analysis 82 3.4 Summary 86 HYBRID, ATTENTION-BASED AND ENSEMBLE DEEP LEARNING MODELS FOR BIOMEDICAL RELATION CLASSIFICATION 87 4.1 The shortest dependency path 89 4.1.1 Dependency tree 89 v 4.1.2 The shortest dependency path 90 4.1.3 Dependency Unit 91 4.2 A hybrid adaptive deep learning model for biomedical relation extraction 91 4.2.1 Proposed MASS model 92 4.2.2 Experimental corpora and comparative models 98 4.2.3 Experimental environment and model settings 100 4.2.4 Experimental results and discussion 100 4.3 An attentive augmented deep learning model for biomedical relation extraction 106 4.3.1 Richer-but-smarter SDP 106 4.3.2 Proposed RbSP model 107 4.3.3 Experimental environment and model settings 114 4.3.4 Experimental results and discussion 114 4.4 A multi-fragment ensemble deep learning model for biomedical relation extraction 118 4.4.1 Over-fitting problem of deep learning-based models 118 4.4.2 Bagging with bootstrap training data 119 4.4.3 Proposed multi-fragment ensemble architecture 121 4.4.4 Experimental results and discussion 124 4.5 Summary 129 GRAPH-BASED INTER-SENTENCE RELATION CLASSIFICATION IN BIOMEDICAL TEXT 131 5.1 Inter-sentence relations classification problem 132 5.2 Proposed graph-based inter-sentence relation classification model 134 5.2.1 Model overview 134 5.2.2 Document sub-graph construction 135 5.2.3 Paths finding, merging and choosing 138 5.2.4 Shared-weight convolutional neural network 140 5.3 Experimental results and discussion 143 5.3.1 Experimental environment and model settings 143 5.3.2 Contribution of the added virtual edges in document sub-graph 144 5.3.3 Different sliding window size w for training and testing 145 5.3.4 Contribution of the model components 146 5.3.5 Comparison to comparative model 148 5.4 Discussion 150 vi 5.5 Summary 152 CONCLUSION 156 LIST OF PUBLICATIONS 158 BIBLIOGRAPHY 158 vii ABBREVIATIONS Acc Accuracy Adam Adaptive Moment Estimation ANN Artificial Neural Network bagging Bootstrap Aggregating BB3 Bacteria Biotope Task BC5 CDR corpus BioCreative V Chemical-Disease relation corpus BERT Bidirectional Encoder Representations from Transformers biLSTM Bidirectional Long Short-term Memory CBOW Continuous Bag-of-words CDR Chemical Disease Relation CID Chemical-induced Disease CNN Convolutional Neural Network CRF Conditional Random Fields CTD Comparative Toxicogenomics Database DDI Drug-drug Interaction DNER Disease Named Entity Recognition DNN Deep Neural Network DU Dependency Unit ELMO Embeddings from Language Models FN False Negative viii FP False Positive FSU-PRGE The FSU PRotein GEne Corpus GD Gradient Descent HAScO Human-Aware Science Ontology HHEAR Human Health Exposure Analysis Resource HMM Hidden Markov Model IAA Inter-annotator Agreement IE Information Extraction KB Knowledge-base LSTM Long Short-term Memory MASS Man for All SeasonS MESH Medical Subject Headings mf Multi-fragment MLP Multilayer Perceptron MUC Message Understanding Conferences NCBI National Center for Biotechnology Information NCIT National Cancer Institute Thesaurus NE Named Entity NEN Named Entity Normalization NER Named Entity Recognition NLP Natural Language Processing OOV Out-Of-Vocabulary OWL Orthology Ontology P Precision PMC Pubmed Central ix POS Part-of-speech R Recall RbSP Richer-but-Smarter Shortest Dependency Path RC Relation Classification RE Relation Extraction ReLU Rectified Linear Unit REP Replacement RGO Radiology Gamuts Ontology RNN Recurrent Neural Network SDP The Shortest Dependency Path SilverCID A Silver-standard Corpus for Chemicalinduced Disease Relation Extraction SNOMED Systematized Nomenclature of Medicine SSI Supervised Semantic Indexing stdev Standard Deviation SVM Suport Vector Machine swCNN Shared-weight Convolutional Neural Network TN True Negative TP True Positive TPAC the Token-POS tag-Abbrviation-Character Embeedings UMLS Unified Medical Language System w/o REP With out Replacement x [57] J Hakenberg, Mining relations from the biomedical literature, Ph.D dissertation, Humboldt-Universităat zu Berlin, Mathematisch-Naturwissenschaftliche Fakultăat II, 2010 [58] D Hanisch, K Fundel, H.-T Mevissen, R Zimmer, and J Fluck, “Prominer: rulebased protein and gene entity recognition,” BMC bioinformatics, vol 6, no 1, p S14, 2005 [59] H He and E A Garcia, “Learning from imbalanced data,” IEEE Trans on Knowl and Data Eng., vol 21, no 9, pp 1263–1284, 2009 [60] M Herrero-Zazo, I Segura-Bedmar, P Mart´ınez, and T Declerck, “The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions,” Journal of biomedical informatics, vol 46, no 5, pp 914–920, 2013 [61] M Herrero-Zazo, I Segura-Bedmar, P Mart´ınez, and T Declerck, “The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions,” Journal of Biomedical Informatics, vol 46, no 5, pp 914–920, 2013 [62] S Higashiyama, “Cost-sensitive structured perceptron incorporating category hierarchy for named entity recognition,” Journal of Information and Communication Technology, vol 14, pp 1–20, 2020 [63] S Hochreiter and J Schmidhuber, “Long short-term memory,” Neural Computation, vol 9, no 8, pp 1735–1780, 1997 [64] C.-C Huang and Z Lu, “Community challenges in biomedical text mining over 10 years: success, failure and the future,” Briefings in bioinformatics, vol 17, no 1, pp 132–144, 2015 [65] L Huang, S Fayong, and Y Guo, “Structured perceptron with inexact search,” in Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Association for Computational Linguistics, 2012, pp 142–151 [66] Z Huang, “Biomedical information extraction: Mining disease associated genes from literature,” Ph.D dissertation, Drexel University, 2014 [67] S Ioffe and C Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International Conference on Machine Learning, 2015, pp 448–456 165 [68] R Islamaj Dogan, G C Murray, A N´ev´eol, and Z Lu, “Understanding pubmed® user search behavior through log analysis,” Database, vol 2009:bap018, 2009 [69] R Islamaj Do˘gan, D C Comeau, L Yeganova, and W J Wilbur, “Finding abbreviations in biomedical literature: three bioc-compatible modules and four biocformatted corpora,” Database, vol 2014:bau044, 2014 [70] R Javed, S Farhan, and S Humdullah, “A hybrid approach based on pattern recognition and bionlp for investigating drug-drug interaction,” Current Bioinformatics, vol 10, no 3, pp 315–322, 2015 [71] F Jenhani, M S Gouider, and L B Said, “A hybrid approach for drug abuse events extraction from twitter,” Procedia computer science, vol 96, pp 1032– 1040, 2016 [72] J Jiang and C Zhai, “A systematic exploration of the feature space for relation extraction,” in Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, 2007, pp 113–120 [73] Z Jiang, L Jin, L Li, M Qin, C Qu, J Zheng, and D Huang, “A crd-wel system for chemical-disease relations extraction,” in Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, 2015, pp 317–326 [74] L.-p Jin and J Dong, “Ensemble deep learning for biomedical time series classification,” Computational intelligence and neuroscience, vol 2016, p 6212684, 2016 [75] N Kambhatla, “Minority vote: at-least-n voting improves recall for extracting relations,” in Proceedings of the COLING/ACL on Main conference poster sessions Association for Computational Linguistics, 2006, pp 460–466 [76] M A Khalid, V Jijkoun, and M De Rijke, “The impact of named entity normalization on information retrieval for question answering,” in European Conference on Information Retrieval Springer, 2008, pp 705–710 [77] R Khare, R Leaman, and Z Lu, “Accessing biomedical literature in the current information landscape,” in Biomedical Literature Mining 11–31 166 Springer, 2014, pp [78] H Kilicoglu and S Bergler, “Adapting a general semantic interpretation approach to biological event extraction,” in Proceedings of the BioNLP Shared Task 2011 Workshop Association for Computational Linguistics, 2011, pp 173–182 [79] J.-j Kim and D Rebholz-Schuhmann, “Improving the extraction of complex regulatory events from scientific text by using ontology-based inference,” Journal of biomedical semantics, vol 2, no 5, p S3, 2011 [80] Y Kim, “Convolutional neural networks for sentence classification,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 2014, pp 1746–1751 [81] D P Kingma and J Ba, “Adam: A method for stochastic optimization,” CoRR, vol abs/1412.6980, 2014 [82] K Kowsari, K Jafari Meimandi, M Heidarysafa, S Mendu, L Barnes, and D Brown, “Text classification algorithms: A survey,” Information, vol 10, no 4, p 150, 2019 [83] M Krallinger, F Leitner, C Rodriguez-Penagos, and A Valencia, “Overview of the protein-protein interaction annotation extraction task of biocreative ii,” Genome biology, vol 9, no 2, p S4, 2008 [84] M Krallinger, F Leitner, O Rabal, M Vazquez, J Oyarzabal, and A Valencia, “CHEMDNER: the drugs and chemical names extraction challenge,” J Cheminformatics, vol 7, no S-1, p S1, 2015 [85] A Krogh and P Sollich, “Statistical mechanics of ensemble learning,” Physical Review E, vol 55, no 1, p 811, 1997 [86] J Lafferty, A McCallum, and F C Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in Proceedings of the 18th International Conference on Machine Learning 2001 (ICML 2001), 2001, pp 282–289 [87] G Lample, M Ballesteros, S Subramanian, K Kawakami, and C Dyer, “Neural architectures for named entity recognition,” in HLT-NAACL, K Knight, A Nenkova, and O Rambow, Eds guistics, 2016, pp 260–270 167 The Association for Computational Lin- [88] R Leaman and G Gonzalez, “Banner: An executable survey of advances in biomedical named entity recognition.” in Pacific Symposium on Biocomputing, R B Altman, A K Dunker, L Hunter, T Murray, and T E Klein, Eds World Scientific, 2008, pp 652–663 [89] R Leaman and Z Lu, “Taggerone: joint named entity recognition and normalization with semi-markov models,” Bioinformatics, vol 32, no 18, p 2839–2846, 2016 [90] R Leaman, R Islamaj Do˘gan, and Z Lu, “Dnorm: disease name normalization with pairwise learning to rank,” Bioinformatics, vol 29, no 22, pp 2909–2917, 2013 [91] R Leaman, C.-H Wei, and Z Lu, “tmchem: a high performance approach for chemical named entity recognition and normalization,” Journal of cheminformatics, vol (Suppl 1), no S3, 2015 [92] Y LeCun, B Boser, J S Denker, D Henderson, R E Howard, W Hubbard, and L D Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation, vol 1, no 4, pp 541–551, 1989 [93] Y LeCun, L Bottou, Y Bengio, and P Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol 86, no 11, pp 2278– 2324, 1998 [94] H Lee, Y Peirsman, A Chang, N Chambers, M Surdeanu, and D Jurafsky, “Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task,” in Proceedings of the fifteenth conference on computational natural language learning: Shared task Association for Computational Linguistics, 2011, pp 28–34 [95] H.-C Lee, Y.-Y Hsu, and H.-Y Kao, “An enhanced crf-based system for disease name entity recognition and normalization on biocreative v dner task,” in Proceedings of the fifth biocreative challenge evaluation workshop, 2015, pp 226–233 [96] J Y Lee, “Information extraction with neural networks,” Ph.D dissertation, Massachusetts Institute of Technology, 2017 [97] J Lee, W Yoon, S Kim, D Kim, S Kim, C H So, and J Kang, “Biobert: a pretrained biomedical language representation model for biomedical text mining,” Bioinformatics, vol 36, no 4, pp 1234–1240, 2020 168 [98] S Lee, Y Song, M Choi, and H Kim, “Bagging-based active learning model for named entity recognition with distant supervision,” in 2016 International Conference on Big Data and Smart Computing (BigComp) IEEE, 2016, pp 321–324 [99] J Lever and S J Jones, “Verse: Event and relation extraction in the bionlp 2016 shared task,” in Proceedings of the the 4th BioNLP Shared Task Workshop, 2016, pp 42–49 [100] C Li, “Biological network evaluation and relation discovery from scientific literature,” Ph.D dissertation, University of Cambridge, 2014 [101] F Li, M Zhang, G Fu, and D Ji, “A neural joint model for entity and relation extraction from biomedical text,” BMC bioinformatics, vol 18, no 1, p 198, 2017 [102] G Li, K E Ross, C N Arighi, Y Peng, C H Wu, and K Vijay-Shanker, “mirtex: a text mining system for mirna-gene relation extraction,” PLoS computational biology, vol 11, no 9, p e1004391, 2015 [103] H Li, M Yang, Q Chen, B Tang, X Wang, and J Yan, “Chemical-induced disease extraction via recurrent piecewise convolutional neural networks,” BMC medical informatics and decision making, vol 18, no 2, p 60, 2018 [104] J Li, Y Sun, R J Johnson, D Sciaky, C Wei, R Leaman, A P Davis, C J Mattingly, T C Wiegers, and Z Lu, “Annotating chemicals, diseases, and their interactions in biomedical literature,” in Proceedings of the Fifth BioCreative challenge evaluation workshop, 2015, pp 173–182 [105] J Li, Y Sun, R J Johnson, D Sciaky, C.-H Wei, R Leaman, A P Davis, C J Mattingly, T C Wiegers, and Z Lu, “Biocreative v cdr task corpus: a resource for chemical disease relation extraction,” Database Oxford, vol 2016:baw068, 2016 [106] J Li, A Ritter, C Cardie, and E Hovy, “Major life event extraction from twitter based on congratulations/condolences speech acts,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp 1997–2007 [107] L Li, L Jin, Z Jiang, D Song, and D Huang, “Biomedical named entity recognition based on extended recurrent neural networks,” in 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) 652 169 IEEE, 2015, pp 649– [108] L Li, J Zheng, and J Wan, “Dynamic extended tree conditioned lstm-based biomedical event extraction,” International Journal of Data Mining and Bioinformatics, vol 17, no 3, pp 266–278, 2017 [109] Q Li and H Ji, “Incremental joint extraction of entity mentions and relations,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2014, pp 402–412 [110] Z Liao and H Wu, “Biomedical named entity recognition based on skip-chain crfs,” in Industrial Control and Electronics Engineering (ICICEE), 2012 International Conference on IEEE, 2012, pp 1495–1498 [111] S Lim, K Lee, and J Kang, “Drug drug interaction extraction from the literature using a recursive neural network,” PloS one, vol 13, no 1, p e0190926, 2018 [112] Y Lin, S Shen, Z Liu, H Luan, and M Sun, “Neural relation extraction with selective attention over instances,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp 2124–2133 [113] W Ling, C Dyer, A W Black, I Trancoso, R Fermandez, S Amir, L Marujo, and T Luis, “Finding function in form: Compositional character models for open vocabulary word representation,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp 1520–1530 [114] B Liu, Q Cui, T Jiang, and S Ma, “A combinational feature selection and ensemble neural network method for classification of gene expression data,” BMC bioinformatics, vol 5, no 1, p 136, 2004 [115] J Liu, A Li, and S Seneff, “Automatic drug side effect discovery from online patient-submitted reviews: Focus on statin drugs,” in Proceedings of First International Conference on Advances in Information Mining and Management (IMMM), Barcelona, Spain Citeseer, 2011, pp 23–29 [116] X Liu, M Zhou, F Wei, Z Fu, and X Zhou, “Joint inference of named entity recognition and normalization for tweets,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume Association for Computational Linguistics, 2012, pp 526–535 170 [117] Y Liu, Z Shi, and A Sarkar, “Exploiting rich syntactic information for relation extraction from biomedical articles,” in Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers Association for Computational Linguistics, 2007, pp 97–100 [118] Y Lou, Y Zhang, T Qian, F Li, S Xiong, and D Ji, “A transition-based joint model for disease named entity recognition and normalization,” Bioinformatics, vol 33, no 15, pp 2363–2371, 2017 [119] D M Lowe, N M O’boyle, and R A Sayle, “Leadmine: Disease identification and concept mapping using wikipedia,” in Proceedings of the Fifth BioCreative Challenge Evaluation Workshop, 2015, pp 240–246 [120] D Lukovnikov, A Fischer, J Lehmann, and S Auer, “Neural network-based question answering over knowledge graphs on word and character level,” in Proceedings of the 26th international conference on World Wide Web International World Wide Web Conferences Steering Committee, 2017, pp 1211–1220 [121] L Luo, Z Yang, P Yang, Y Zhang, L Wang, H Lin, and J Wang, “An attentionbased bilstm-crf approach to document-level chemical named entity recognition,” Bioinformatics, vol 34, no 8, pp 1381–1388, 2018 [122] X Ma and E Hovy, “End-to-end sequence labeling via bi-directional lstm-cnnscrf,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics, 2016, pp 1064–1074 [123] R Maclin and D Opitz, “An empirical evaluation of bagging and boosting,” AAAI/IAAI, vol 1997, pp 546–551, 1997 [124] P S Madhyastha and R Jain, “On model stability as a function of random seed,” in Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), 2019, pp 929–939 [125] T Mavropoulos, D Liparas, S Symeonidis, S Vrochidis, and I Kompatsiaris, “A hybrid approach for biomedical relation extraction using finite state automata and random forest-weighted fusion,” in International Conference on Computational Linguistics and Intelligent Text Processing 171 Springer, 2017, pp 450–462 [126] R McDonald, K Hall, and G Mann, “Distributed training strategies for the structured perceptron,” in Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics Association for Computational Linguistics, 2010, pp 456–464 [127] M L McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica: Biochemia medica, vol 22, no 3, pp 276282, 2012 [128] F Mehryary, J Bjăorne, S Pyysalo, T Salakoski, and F Ginter, “Deep learning with minimal training data: Turkunlp entry in the bionlp shared task 2016,” in Proceedings of the the 4th BioNLP Shared Task Workshop Association for Computational Linguistics, 2016, pp 73–81 [129] C Mih˘ail˘a and S Ananiadou, “Semi-supervised learning of causal relations in biomedical scientific discourse,” Biomedical engineering online, vol 13, no 2, p S1, 2014 [130] T Mikolov, I Sutskever, K Chen, G Corrado, and J Dean, “Distributed representations of words and phrases and their compositionality,” in Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, ser NIPS’13, 2013, pp 3111–3119 [131] M Mintz, S Bills, R Snow, and D Jurafsky, “Distant supervision for relation extraction without labeled data,” in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, 2009, pp 1003– 1011 [132] T Mitsumori, M Murata, Y Fukuda, K Doi, and H Doi, “Extracting proteinprotein interaction information from biomedical text with svm,” IEICE Transactions on Information and Systems, vol 89, no 8, pp 2464–2466, 2006 [133] M Miwa and M Bansal, “End-to-end relation extraction using lstms on sequences and tree structures,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol 1, 2016, pp 1105– 1116 [134] M Miwa and Y Sasaki, “Modeling joint entity and relation extraction with table representation,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp 1858–1869 172 [135] M Miwa, R Sætre, J.-D Kim, and J Tsujii, “Event extraction with complex event classification using rich features,” Journal of bioinformatics and computational biology, vol 8, no 01, pp 131–146, 2010 [136] A S A.-H A Mohammed and F O F T Bagash, “A biomedical named entity recognition using machine learning classifiers and rich feature set,” IJCSNS, vol 17, no 1, p 170, 2017 [137] H Moses III and J B Martin, “Biomedical research and health advances,” 2011 [138] D Nadeau and S Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol 30, no 1, pp 3–26, 2007 [139] F Nargesian, H Samulowitz, U Khurana, E B Khalil, and D S Turaga, “Learning feature engineering for classification.” in IJCAI, 2017, pp 2529–2535 [140] V Ng, “Unsupervised models for coreference resolution,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing Association for Computational Linguistics, 2008, pp 640–649 [141] T Ohta, Y Tateisi, and J.-D Kim, “The genia corpus: An annotated research abstract corpus in molecular biology domain,” in Proceedings of the second international conference on Human Language Technology Research Morgan Kaufmann Publishers Inc., 2002, pp 82–86 [142] S C ONYE, A AKKELES¸, and N DIMILILER, “Review of biomedical relation extraction,” European International Journal of Science and Technology, no 6, p 1, 2017 [143] N C Panyam, K Verspoor, T Cohn, and K Ramamohanarao, “Exploiting graph kernels for high performance biomedical relation extraction,” Journal of biomedical semantics, vol 9, no 1, p 7, 2018 [144] N Peng, H Poon, C Quirk, K Toutanova, and W.-t Yih, “Cross-sentence n-ary relation extraction with graph lstms,” Transactions of the Association for Computational Linguistics, vol 5, pp 101–115, 2017 [145] Y Peng, “A study of relation extraction for biomedical text,” Ph.D dissertation, University of Delaware, 2016 173 [146] Y Peng, M Torii, C H Wu, and K Vijay-Shanker, “A generalizable nlp framework for fast development of pattern-based biomedical relation extraction systems,” BMC bioinformatics, vol 15, no 1, p 285, 2014 [147] Y Peng, C.-H Wei, and Z Lu, “Improving chemical disease relation extraction with rich features and weakly labeled data,” Journal of cheminformatics, vol 8, no 1, p 53, 2016 [148] J Pennington, R Socher, and C Manning, “Glove: Global vectors for word representation,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp 1532–1543 [149] M E Peters, M Neumann, M Iyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer, “Deep contextualized word representations,” in Proceedings of NAACLHLT, 2018, pp 2227–2237 [150] S Pradhan, N Elhadad, B R South, D Martinez, L Christensen, A Vogel, H Suominen, W W Chapman, and G Savova, “Evaluating the state of the art in disorder recognition and normalization of the clinical narrative,” Journal of the American Medical Informatics Association, vol 22, no 1, pp 143–154, 2014 [151] S Pyysalo, F Ginter, H Moen, T Salakoski, and S Ananiadou, “Distributional semantics resources for biomedical text processing,” in Proceedings of LBM 2013, 2013, pp 39–44 [152] P Qin, W Xu, and J Guo, “An empirical convolutional neural network approach for semantic relation classification,” Neurocomputing, vol 190, pp 1–9, 2016 [153] C Quan, M Wang, and F Ren, “An unsupervised text mining method for relation extraction from biomedical literature,” PloS one, vol 9, no 7, p e102039, 2014 [154] C Quan, L Hua, X Sun, and W Bai, “Multichannel convolutional neural network for biological relation extraction,” BioMed Research International, vol 2016, pp 1–10, 01 2016 [155] C Quirk and H Poon, “Distant supervision for relation extraction beyond the sentence boundary,” in Proceedings of the Fifteenth Conference on European chapter of the Association for Computational Linguistics, vol Volume 1, Long Papers Association for Computational Linguistics, 2017, pp 1171—-1182 174 [156] A Raihani and N Laachfoubi, “A rich feature-based kernel approach for drugdrug interaction extraction,” International Journal of Advanced Computer Science and Applications, vol 8, no 4, pp 324–330, 2017 [157] L Ratinov and D Roth, “Design challenges and misconceptions in named entity recognition,” in Proceedings of the Thirteenth Conference on Computational Natural Language Learning Association for Computational Linguistics, 2009, pp 147–155 [158] C K Reddy and C C Aggarwal, Healthcare data analytics CRC Press, 2015, vol 36 [159] J A Reyes-Ortiz, B A Gonz´alez-Beltr´an, and L Gallardo-L´opez, “Clinical decision support systems: a survey of nlp-based approaches from unstructured data,” in 2015 26th International Workshop on Database and Expert Systems Applications (DEXA) IEEE, 2015, pp 163–167 [160] S Riedel, L Yao, A McCallum, and B M Marlin, “Relation extraction with matrix factorization and universal schemas,” in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013, pp 74–84 [161] F Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain.” Psychological review, vol 65, no 6, p 386, 1958 [162] D E Rumelhart, G E Hinton, and R J Williams, “Learning representations by back-propagating errors,” nature, vol 323, no 6088, p 533, 1986 [163] S K Sahu, F Christopoulou, M Miwa, and S Ananiadou, “Inter-sentence relation extraction with document-level graph convolutional neural network,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, p 4309–4316 [164] I Segura-Bedmar, P Mart´ınez, and M H Zazo, “Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013),” in Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), vol 2, 2013, pp 341–350 175 [165] I Segura-Bedmar, P Mart´ınez, and M H Zazo, “Lessons learnt from the ddiextraction-2013 shared task,” Journal of Biomedical Informatics, vol 51, pp 152–164, 2014 [166] S Sharma, S Srivastava, A Kumar, and A Dangi, “Multi-class sentiment analysis comparison using support vector machine (svm) and bagging technique-an ensemble method,” in 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE) IEEE, 2018, pp 1–6 [167] M S Simpson and D Demner-Fushman, “Biomedical text mining: a survey of recent progress,” in Mining text data Springer, 2012, pp 465–517 [168] S Sohn, D C Comeau, W Kim, and W J Wilbur, “Abbreviation definition identification based on automatic precision estimates,” BMC bioinformatics, vol 9, no 1, p 402, 2008 [169] A J Soto, P Przybyła, and S Ananiadou, “Thalia: semantic search engine for biomedical abstracts,” Bioinformatics, vol 35, no 10, pp 1799–1801, 2019 [170] N Srivastava, G Hinton, A Krizhevsky, I Sutskever, and R Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” J Mach Learn Res., vol 15, no 1, pp 1929–1958, 2014 [171] M Surdeanu, J Tibshirani, R Nallapati, and C D Manning, “Multi-instance multi-label learning for relation extraction,” in Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning Association for Computational Linguistics, 2012, pp 455–465 [172] P Thomas, “Robust relationship extraction in the biomedical domain,” Ph.D dissertation, Humboldt-Universităat zu Berlin, Mathematisch-Naturwissenschaftliche Fakultăat, 2015 [173] D V Tung, “Classification and prediction of genes related to disease using network-based algorithmsc,” Ph.D dissertation, Posts and Telecommunications Institute of Technology, Vietnam, 2017 [174] Y Usami, H.-C Cho, N Okazaki, and J Tsujii, “Automatic acquisition of huge training data for bio-medical named entity recognition,” in Proceedings of BioNLP 2011 Workshop Association for Computational Linguistics, 2011, pp 65–73 176 [175] P Verga, D Belanger, E Strubell, B Roth, and A McCallum, “Multilingual relation extraction using compositional universal schema,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp 886–896 [176] P Verga, E Strubell, and A McCallum, “Simultaneously self-attending to all mentions for full-abstract biological relation extraction,” in Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), 2018, pp 872–884 [177] L Verwimp, J Pelemans, H V hamme, and P Wambacq, “Character-word lstm language models,” in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, long papers, 2017, pp 417– 427 [178] N T Vu, H Adel, P Gupta, and H Schăutze, Combining recurrent and convolutional neural networks for relation classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp 534–539 [179] Z Wang, Y Qu, L Chen, J Shen, W Zhang, S Zhang, Y Gao, G Gu, K Chen, and Y Yu, “Label-aware double transfer learning for cross-specialty medical named entity recognition,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), vol 1, 2018, pp 1–15 [180] C.-H Wei, Y Peng, R Leaman, A P Davis, C J Mattingly, J Li, T C Wiegers, and Z Lu, “Overview of the biocreative v chemical disease relation (cdr) task,” in Proceedings of the fifth BioCreative challenge evaluation workshop, 2015, pp 154–166 [181] Q Wei, T Chen, R Xu, Y He, and L Gui, “Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks,” Database, vol 2016, no baw140, 2016 [182] X Wei, Q Zhu, C Lyu, K Ren, and B Chen, “A hybrid method to extract triggers in biomedical events,” Journal of Digital Information Management, vol 13, no 4, p 299, 2015 177 [183] Y Won and P D Gader, “Morphological shared-weight neural network for pattern classification and automatic target detection,” in Proceedings of ICNN’95International Conference on Neural Networks, vol IEEE, 1995, pp 2134– 2138 [184] Y Wu, M Jiang, J Xu, D Zhi, and H Xu, “Clinical named entity recognition using deep learning models,” in AMIA Annual Symposium Proceedings, vol 2017 American Medical Informatics Association, 2017, p 1812 [185] J Xu, Y Wu, Y Zhang, J Wang, R Liu, Q Wei, , and H Xu, “Uth-ccb@ biocreative v cdr task: identifying chemical-induced disease relations in biomedical text,” in Proceedings of the 5th BioCreative Challenge Evaluation Workshop, 2015, pp 254–259 [186] J Xu, Y Wu, Y Zhang, J Wang, H.-J Lee, and H Xu, “Cd-rest: a system for extracting chemical-induced disease relation in literature,” Database, vol 2016, 2016 [187] K Xu, Y Feng, S Huang, and D Zhao, “Semantic relation classification via convolutional neural networks with simple negative sampling,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015, pp 536–540 [188] S.-J Yen and Y.-S Lee, “Under-sampling approaches for improving prediction of the minority class in an imbalanced dataset,” in Proceedings of Intelligent Control and Automation, 2006, pp 731–740 [189] L Yu and H Liu, “Efficient feature selection via analysis of relevance and redundancy,” Journal of Machine Learning Research, vol 5, pp 1205–1224, 12 2004 [190] S Zhang and N Elhadad, “Unsupervised biomedical named entity recognition: Experiments with clinical and biological texts,” Journal of biomedical informatics, vol 46, no 6, pp 1088–1098, 2013 [191] X Zhang, F Chen, and R Huang, “A combination of rnn and cnn for attentionbased relation classification,” Procedia computer science, vol 131, pp 911–917, 2018 [192] Y Zhang, Y Xin, Q Li, J Ma, S Li, X Lv, and W Lv, “Empirical study of seven data mining algorithms on different characteristics of datasets for biomed178 ical classification applications,” Biomedical engineering online, vol 16, no 1, p 125, 2017 [193] Y Zhang and S Clark, “Joint word segmentation and pos tagging using a single perceptron,” Proceedings of ACL-08: HLT, pp 888–896, 2008 [194] Z Zhao, Z Yang, L Luo, H Lin, and J Wang, “Drug drug interaction extraction from biomedical literature using syntax convolutional neural network,” Bioinformatics, vol 32, no 22, pp 3444–3453, 2016 [195] J Zheng, W W Chapman, R S Crowley, and G K Savova, “Coreference resolution: A review of general methodologies and applications in the clinical domain,” Journal of biomedical informatics, vol 44, no 6, pp 1113–1122, 2011 [196] W Zheng, H Lin, Z Li, X Liu, Z Li, B Xu, Y Zhang, Z Yang, and J Wang, “An effective neural model extracting document level chemical-induced disease relations from biomedical literature,” Journal of biomedical informatics, vol 83, pp 1–9, 2018 [197] D Zhou, L Miao, and Y He, “Position-aware deep multi-task learning for drug– drug interaction extraction,” Artificial intelligence in medicine, vol 87, pp 1–8, 2018 [198] H Zhou, H Deng, L Chen, Y Yang, C Jia, and D Huang, “Exploiting syntactic and semantics information for chemical–disease relation extraction,” Database, vol 2016:baw048, 2016 179 ... UNIVERSITY OF ENGINEERING AND TECHNOLOGY LE HOANG QUYNH MACHINE LEARNING-BASED EXTRACTION OF SEMANTIC RELATIONS FROM BIOMEDICAL LITERATURE Major: Information Systems Code: 9480104.01 DOCTOR OF PHILOSOPHY... extraction 1.1.1 Semantic relation extraction First of all, we present the definition of semantic relation extraction in Definition 1.1 Definition 1.1 Semantic relations (or semantic relationships)... 19 1.2.1 Literature review of biomedical named entity recognition 19 1.2.2 Literature review of biomedical relation extraction 24 1.2.3 Related doctoral