[ How to keep certain entities as one word using nltk tokenize in python? ]

How does one keep certain strings together in the following? For example,

sentence = "?!a# .see"
tokens = nltk.word_tokenize(sentence)  
tokens 

gives

['!', '?', 'a', '#', '.see'] rather than keeping '?!a#' as one entity.

Answer 1

Seems like what you want to do is to split the string with whitespace. So just calling split would suffice:

>>> sentence.split()
['?!a#', '.see']

However if you really want to use a tokenizer, you can use a Regexp tokenizer:

>>> word_tokenizer = RegexpTokenizer('[\S]+') 
>>> word_tokenizer.tokenize(sentence)
['?!a#', '.see']

'\S' matches any non-whitespace character.