What Exactly Is a Word?

 When we talk or write, we use words without thinking much about them. But when computers process language — like when you chat with an AI — figuring out what counts as a “word” isn’t as simple as it seems.

In everyday life, a word is something that carries meaning on its own — like cat, happy, or jumped. But computers can’t “see” meaning the way we do. They only see strings of letters and symbols. To teach a computer what a word is, we have to give it clear rules for where one word ends and the next begins.

For example, in the sentence:

They played by the river.

Most people would say there are five words. But what about punctuation? Should the period at the end count as a word too? Some computer programs say yes; others say no, depending on what they’re trying to do.

Things get even more complicated in spoken language. People often say things like “uh” or “um” when they speak. Are those words? Linguists call them fillers, and whether they count as words depends on the purpose of the analysis.

Then there’s the challenge of different languages. English uses spaces to separate words, but languages like Chinese or Thai don’t — they write characters continuously without gaps. In those languages, deciding what a “word” is becomes a tricky problem.

Even in English, we have gray areas. Is I’m one word or two (I and am)? What about New York — one name or two words? The answer depends on the context and on what we’re trying to teach the computer.

So, defining a “word” is one of the first and most important steps in teaching computers to understand language. Once we have a clear sense of what words are, we can move on to the next step — turning them into tokens, the building blocks that computers use to read and generate text.