KLL (Kotlin Lexer Library)

A lightweight, extensible lexical analysis (lexer/tokenizer) toolkit written in Kotlin. It lets you describe how to recognize tokens, optionally minify token streams, and run simple pre-syntax validations before parsing.

Overview

KLL provides a small, well-documented façade (Lexer) to configure a pipeline made of:

• Tokenizers (a sequence of TokenMatcher implementations)
• Optional Minifiers (MinifierMatcher) to remove or transform non-essential tokens
• Optional Pre-syntax validators (SyntaxRule) to run adjacency checks between tokens

The project ships a std package with common token types, matchers and minifiers that serve as a good starting point for many simple grammars.

Importing

Add the JitPack repository to your root build.gradle file:

repositories {
    maven { url 'https://jitpack.io' }
}

Then add the dependency to your module's build.gradle file:

dependencies {
    implementation 'com.github.RafaPear:KLL:Tag'
}

Replace Tag with the desired release tag, branch name, or commit hash.

Quick start

A minimal setup using the standard library components shipped in this repo:

fun demo() {
    val lexer = pt.rafap.kll.Lexer()

    // Logging helpers:
    // lexer.shutTheFuckUp()   // Level.OFF
    lexer.speakTheFuckUp()     // Level.INFO
    // lexer.speakLouder()     // Level.DEBUG

    lexer.addMatchers(
        pt.rafap.kll.std.matchers.WhitespaceMatcher,
        pt.rafap.kll.std.matchers.KeyWordMatcher(
            listOf("add", "sub", "and", "or", "b", "b."), pt.rafap.kll.std.types.ReservedType
        ),
        pt.rafap.kll.std.matchers.StringLiteralMatcher(listOf('"')),
        *pt.rafap.kll.std.types.DefaultSymbols.SYMBOL_MATCHERS,
        pt.rafap.kll.std.matchers.NumberMatcher,
        pt.rafap.kll.std.matchers.IdentifierMatcher,
    )

    lexer.addMinifiers(*pt.rafap.kll.std.minifier.DefaultMinifiers.ALL_MINIFIERS)
    lexer.addRules(*pt.rafap.kll.std.rules.DefaultRules.ALL_RULES)

    val tokens = lexer.lexFile("path/to/file.s", verify = true)
    lexer.printTokensAsFile(tokens, "out.s")
}

Packages

• pt.rafap.kll — Public façade (Lexer) and top-level helpers
• pt.rafap.kll.components — Internal pipeline components (Tokenizer, Minifier, PreSyntaxValidator)
• pt.rafap.kll.token — Core token model and matching primitives (Token, TokenMatcher, TokenType, etc.)
• pt.rafap.kll.minify — Minifier extension point and result types
• pt.rafap.kll.rules — Pre-syntax rule interfaces and results
• pt.rafap.kll.std — Standard token types, matchers and minifiers shipped with the project

Building & Testing

Use the provided Gradle wrapper to build and run tests:

./gradlew build
./gradlew test

(Windows users can run gradlew.bat.)

License

MIT — see the repository license file for details.