What is natural language processing?
Put simply, Natural Language Processing (or NLP) is the automatic processing of human language.
NLP is a type of artificial intelligence based on machine learning that examines patterns in data to derive meaning from human (natural) language input. NLP makes use of linguistic concepts such as grammatical structures and parts of speech.
What is natural language processing used for?
NLP is usually used to determine who did what to whom, when, where, how, and why.
NLP performs analysis on text at different levels:
1. Determine the meaning of the word in the context it is in by examining characteristics of the word (prefixes, suffixes, roots, and parts of speech)
2. Determine the relationship between words by analysing grammatical structure
3. Determine the possible meaning of the sentence by examining word order, sentence structure, and disambiguating words
4. Determine the meaning of the text beyond sentence level
How does it work?
How does a computer know which of these posts are relevant to an employer when they both include key words?
Come up with dictionary of phrases
Run through an engram (a software that adapts and learns from past patterns.) This comes up with a huge list of words that could be related.
Put this through a filter system e.g. if you are looking at employment references and keyword is ‘working’, you automatically void irrelevant terms e.g. ‘working hard at the gym’
Eventually, the system learns what would should be associated and what words shouldn’t be associated (Machine Learning).
Basically, machines learn what words are relevant in what context, much like humans do, and eventually apply rules that result in accurate categorisation and classification.
Check out our blog on Bayesian Belief Networks