Tutorial

Now that you have PyNLPIR installed, let’s look at how to use it.

There are two ways to use PyNLPIR: directly using the ctypes interface provided by PyNLPIR or using PyNLPIR’s helper functions. The ctypes interface is more extensive, but more rigid. The helper functions are easy-to-use, but don’t provide access to every NLPIR function. You can also use a mixture of the two methods. First, let’s look at the helper functions.

PyNLPIR Helper Functions

The helper functions are located in PyNLPIR’s __init__.py file, so they are accessible by importing pynlpir directly.

Initializing NLPIR

Importing PyNLPIR loads the NLPIR API library automatically:

import pynlpir

Once it’s imported, call open() to tell NLPIR to open the data files and initialize the API. See open()‘s documentation for information on specifying a different data directory.

pynlpir.open()

By default, input is assumed to be unicode or UTF-8 encoded. If you’d like to use a different encoding (e.g. GBK or BIG5), use the encoding keyword argument when calling open():

pynlpir.open(encoding='big5')

Tip

No matter what encoding you’ve specified, you can always pass unicode strings to pynlpir functions.

PyNLPIR’s helper functions always return unicode strings.

Once you’ve initialized NLPIR, you can start segmenting and analyzing text.

Segmenting Text

Let’s segment a lengthy sentence:

s = 'NLPIR分词系统前身为2000年发布的ICTCLAS词法分析系统，从2009年开始，为了和以前工作进行大的区隔，并推广NLPIR自然语言处理与信息检索共享平台，调整命名为NLPIR分词系统。'
pynlpir.segment(s)

# Sample output: [('NLPIR', 'noun'), ('分词', 'verb'), ('系统', 'noun'), ('前身', 'noun'), ('为', 'preposition'), ('2000年', 'time word'), ('发布', 'verb'), . . . ]

If you don’t want part of speech tagging, call segment() with pos_tagging set to False:

pynlpir.segment(s, pos_tagging=False)

# Sample output: ['NLPIR', '分词', '系统', '前身', '为', '2000年', '发布', . . . ]

You can also customize how the part of speech tags are shown. By default, only the most generic part of speech name is used, i.e. the parent (for example, 'noun' instead of 'transcribed toponym'). If you’d like the most specific part of speech name instead, i.e. the child, set pos_names to 'child':

pynlpir.segment(s, pos_names='child')

If you want even more information about the part of speech tags, you can set pos_names to 'all' and a part of speech hierarchy is returned (for example, 'noun:toponym:transcribed toponym'):

pynlpir.segment(s, pos_names='all')

By default, part of speech tags are returned in English. If you’d like to see Chinese instead (e.g. '名词' instead of 'noun'), set pos_english to False:

pynlpir.segment(s, pos_english=False)

Getting Key Words

Another useful function is get_key_words():

pynlpir.get_key_words(s, weighted=True)
[('NLPIR', 2.08), ('系统', 1.74)]

get_key_words() analyzes the given Chinese text string and returns words that NLPIR considers key words. If weighted is True, then the key word’s weight is also returned as a float.

Closing the API

Now that we’ve looked at a brief introduction to PyNLPIR’s helper functions, let’s look at how to close the API.

When you’re done using PyNLPIR, you can free up allocated memory by calling close():

pynlpir.close()

ctypes NLPIR Interface

pynlpir.nlpir provides access to NLPIR’s C functions via ctypes. You can call them directly without bothering with the helper functions above. These functions work almost exactly the same as their C counterparts.

pynlpir.nlpir includes the module-level constants exported by NLPIR that are needed for calling many of its functions (e.g. encoding and part of speech constants). See the API page on pynlpir.nlpir for more information.

The sections below do not provide a comprehensive explanation of how to use NLPIR. NLPIR has its own documentation. The section below provides basic information about how to get started with PyNLPIR assuming you are familiar with NLPIR. If you’re not, be sure to check out the documentation linked to below.

Initializing and Exiting the API

Before you can call any other NLPIR functions, you need to initialize the NLPIR API. This is done by calling Init(). You have to specify where NLPIR’s Data directory is. PyNLPIR ships with a copy and it’s found in the top-level of the package directory. So, you can use the module-level constant PACKAGE_DIR when calling Init():

from pynlpir import nlpir

nlpir.Init(nlpir.PACKAGE_DIR)

NLPIR defaults to using GBK encoding. If you don’t plan on passing around GBK-encoded strings, you’ll want to change the encoding when calling Init():

nlpir.Init(nlpir.PACKAGE_DIR, nlpir.UTF8_CODE)

Once NLPIR is initialized, you can begin using the rest of the NLPIR functions. When you’re finished, it’s good to call Exit() in order to exit the NLPIR API and free the allocated memory:

nlpir.Exit()

The Rest of the NLPIR Functions

For a complete list of NLPIR functions that pynlpir.nlpir includes, check out the PyNLPIR API. NLPIR’s documentation is included in a PDF file in the NLPIR/ICTCLAS 2016 download. Consult it for detailed information on how to use NLPIR.

What’s Next

Now that you’ve finished the tutorial, you should be able to perform basic tasks using PyNLPIR. If you need more information regarding a module, constant, or function, be sure to check out the PyNLPIR API. If you need help, spot a bug, or have a feature request, then please visit PyNLPIR’s GitHub Issues page.