Indicatements that character language models learn English morpho-syntactic units and regularities (1809.00066v1)

Abstract: Character LLMs have access to surface morphological patterns, but it is not clear whether or how they learn abstract morphological regularities. We instrument a character LLM with several probes, finding that it can develop a specific unit to identify word boundaries and, by extension, morpheme boundaries, which allows it to capture linguistic properties and regularities of these units. Our LLM proves surprisingly good at identifying the selectional restrictions of English derivational morphemes, a task that requires both morphological and syntactic awareness. Thus we conclude that, when morphemes overlap extensively with the words of a language, a character LLM can perform morphological abstraction.