Google Summer of Code (GSoC) is a global program focused on introducing students to open source software development. In Greece, the Open Technologies Alliance participates in GSOC 2019 with more than 25 open source projects. The Hellenic OCR Team has provided a highly intresting project proposal, which is presented below. We invite students who wish to participate to contact us in order to be guided through the process.
Hellenic OCR Team 2019 GSoC Proposal
Development of a Tool for Extracting Quantitative Text Profiles
Brief Explanation
Quantitative text analysis is the basis of nearly every computational approach to text management and processing. All advanced Natural Language Processing (NLP) tasks including information retrieval, sentiment analysis, computational stylistics etc. involve the quantification of texts across a huge number of linguistic features and transform text into vectors. In many programming languages, e.g. R, Python, Java etc., there are numerous open source scripts, tools, packages and libraries that can transform texts to vectors of word frequencies, character and word n-gram frequencies, stylometric features etc. However, each of these tools covers only a restricted subset of the possible linguistic features. Moreover, the available tools are written in different languages and require considerable efforts to be combined so that the user can extract a unified file of results. Due to the fragmentary nature of the programing environments and the highly technical skills that are required to operate the tools and combine their results, they can’t be used by large communities of scientists with humanities and sociopolitical background. For the above reasons, we envisage the development of a user-friendly Graphical User Interface (GUI) based tool that shall provide integrated access to existing open NLP software. The new tool shall support the quantitative analysis of multilingual texts and produce quantitativetext profiles that can be used as input for further analysis, visualization, machine learning and other advanced computational processing. Such a tool does not exist to date and it will boost research in all scientific areas that require computational processing of large amounts of text.
Expected Results
The outcome of this project would be an open-source software with the following specifications:
- User-friendly GUI that can guide intuitively its users to select the features they want to count in their text collections
- Large set of linguistic features that include at least:
- Most frequent words of the texts analyzed
- User-specified word lists
- Word and Character n-grams of arbitrary length
- Different stylometric features such as vocabulary diversity indices, readability indices, quantitative linguistic indices
- UTF-8 support
- Corpus management features using text metadata
Related repositories
- https://github.com/gmikros/Author_Multilevel_Ngram_Profiles
- https://github.com/quanteda/quanteda
- https://github.com/unDocUMeantIt/koRpus
- https://miroslavkubat.webnode.cz/software/
- https://github.com/bnosac/udpipe
- https://github.com/explosion/spaCy
Knowledge Prerequisites
Good knowledge of the languages R, Java, Python and skills for GUI interfaces development. Good understanding of NLP concepts and tools.
Mentors