• iamthetot@piefed.ca
    link
    fedilink
    English
    arrow-up
    17
    ·
    edit-2
    1 day ago

    Worth noting that a token is not necessarily a word, though can be. One word could also take multiple tokens. It can also vary from LLM to LLM and their tokenization methods.