Pygments

Builtin Tokens

« Back To Index

Keyword Tokens
Name Tokens
Literals
Operators
Punctuation
Comments
Generic Tokens

Inside the pygments.token module, there is a special object called Token that is used to create token types.

You can create a new token type by accessing an attribute of Token:

>>> from pygments.token import Token
>>> Token.String
Token.String
>>> Token.String is Token.String
True

Note that tokens are singletons so you can use the is operator for comparing token types.

As of Pygments 0.7 you can also use the in operator to perform set tests:

>>> from pygments.token import Comment
>>> Comment.Single in Comment
True
>>> Comment in Comment.Multi
False

This can be useful in filters and if you write lexers on your own without using the base lexers.

You can also split a token type into a hierarchy, and get the parent of it:

>>> String.split()
[Token, Token.Literal, Token.Literal.String]
>>> String.parent
Token.Literal

In principle, you can create an unlimited number of token types but nobody can guarantee that a style would define style rules for a token type. Because of that, Pygments proposes some global token types defined in the pygments.token.STANDARD_TYPES dict.

For some tokens aliases are already defined:

>>> from pygments.token import String
>>> String
Token.Literal.String

Inside the pygments.token module the following aliases are defined:

Text	Token.Text	for any type of text data
Whitespace	Token.Text.Whitespace	for specially highlighted whitespace
Error	Token.Error	represents lexer errors
Other	Token.Other	special token for data not matched by a parser (e.g. HTML markup in PHP code)
Keyword	Token.Keyword	any kind of keywords
Name	Token.Name	variable/function names
Literal	Token.Literal	Any literals
String	Token.Literal.String	string literals
Number	Token.Literal.Number	number literals
Operator	Token.Operator	operators (`+`, `not`...)
Punctuation	Token.Punctuation	punctuation (`[`, `(`...)
Comment	Token.Comment	any kind of comments
Generic	Token.Generic	generic tokens (have a look at the explanation below)

The Whitespace token type is new in Pygments 0.8. It is used only by the VisibleWhitespaceFilter currently.

Normally you just create token types using the already defined aliases. For each of those token aliases, a number of subtypes exists (excluding the special tokens Token.Text, Token.Error and Token.Other)

The is_token_subtype() function in the pygments.token module can be used to test if a token type is a subtype of another (such as Name.Tag and Name). (This is the same as Name.Tag in Name. The overloaded in operator was newly introduced in Pygments 0.7, the function still exists for backwards compatiblity.)

With Pygments 0.7, it's also possible to convert strings to token types (for example if you want to supply a token from the command line):

>>> from pygments.token import String, string_to_tokentype
>>> string_to_tokentype("String")
Token.Literal.String
>>> string_to_tokentype("Token.Literal.String")
Token.Literal.String
>>> string_to_tokentype(String)
Token.Literal.String

Keyword Tokens

Keyword: For any kind of keyword (especially if it doesn't match any of the subtypes of course).
Keyword.Constant: For keywords that are constants (e.g. None in future Python versions).
Keyword.Declaration: For keywords used for variable declaration (e.g. var in some programming languages like JavaScript).
Keyword.Namespace: For keywords used for namespace declarations (e.g. import in Python and Java and package in Java).
Keyword.Pseudo: For keywords that aren't really keywords (e.g. None in old Python versions).
Keyword.Reserved: For reserved keywords.
Keyword.Type: For builtin types that can't be used as identifiers (e.g. int, char etc. in C).

Name Tokens

Name: For any name (variable names, function names, classes).
Name.Attribute: For all attributes (e.g. in HTML tags).
Name.Builtin: Builtin names; names that are available in the global namespace.
Name.Builtin.Pseudo: Builtin names that are implicit (e.g. self in Ruby, this in Java).
Name.Class: Class names. Because no lexer can know if a name is a class or a function or something else this token is meant for class declarations.
Name.Constant: Token type for constants. In some languages you can recognise a token by the way it's defined (the value after a const keyword for example). In other languages constants are uppercase by definition (Ruby).
Name.Decorator: Token type for decorators. Decorators are synatic elements in the Python language. Similar syntax elements exist in C# and Java.
Name.Entity: Token type for special entities. (e.g.   in HTML).
Name.Exception: Token type for exception names (e.g. RuntimeError in Python). Some languages define exceptions in the function signature (Java). You can highlight the name of that exception using this token then.
Name.Function: Token type for function names.
Name.Label: Token type for label names (e.g. in languages that support goto).
Name.Namespace: Token type for namespaces. (e.g. import paths in Java/Python), names following the module/namespace keyword in other languages.
Name.Other: Other names. Normally unused.
Name.Tag: Tag names (in HTML/XML markup or configuration files).
Name.Variable: Token type for variables. Some languages have prefixes for variable names (PHP, Ruby, Perl). You can highlight them using this token.
Name.Variable.Class: same as Name.Variable but for class variables (also static variables).
Name.Variable.Global: same as Name.Variable but for global variables (used in Ruby, for example).
Name.Variable.Instance: same as Name.Variable but for instance variables.

Literals

Literal: For any literal (if not further defined).
Literal.Date: for date literals (e.g. 42d in Boo).
String: For any string literal.
String.Backtick: Token type for strings enclosed in backticks.
String.Char: Token type for single characters (e.g. Java, C).
String.Doc: Token type for documentation strings (for example Python).
String.Double: Double quoted strings.
String.Escape: Token type for escape sequences in strings.
String.Heredoc: Token type for "heredoc" strings (e.g. in Ruby or Perl).
String.Interpol: Token type for interpolated parts in strings (e.g. #{foo} in Ruby).
String.Other: Token type for any other strings (for example %q{foo} string constructs in Ruby).
String.Regex: Token type for regular expression literals (e.g. /foo/ in JavaScript).
String.Single: Token type for single quoted strings.
String.Symbol: Token type for symbols (e.g. :foo in LISP or Ruby).
Number: Token type for any number literal.
Number.Float: Token type for float literals (e.g. 42.0).
Number.Hex: Token type for hexadecimal number literals (e.g. 0xdeadbeef).
Number.Integer: Token type for integer literals (e.g. 42).
Number.Integer.Long: Token type for long integer literals (e.g. 42L in Python).
Number.Oct: Token type for octal literals.

Operators

Operator: For any punctuation operator (e.g. +, -).
Operator.Word: For any operator that is a word (e.g. not).

Punctuation

New in Pygments 0.7.

Punctuation: For any punctuation which is not an operator (e.g. [, (...)

Comments

Comment: Token type for any comment.
Comment.Multiline: Token type for multiline comments.
Comment.Preproc: Token type for preprocessor comments (also <?php/<% constructs).
Comment.Single: Token type for comments that end at the end of a line (e.g. # foo).
Comment.Special: Special data in comments. For example code tags, author and license informations etc.

Generic Tokens

Generic tokens are for special lexers like the DiffLexer that doesn't really highlight a programming language but a patch file.

Generic: A generic, unstyled token. Normally you don't use this token type.
Generic.Deleted: Marks the token value as deleted.
Generic.Emph: Marks the token value as emphasized.
Generic.Error: Marks the token value as an error message.
Generic.Heading: Marks the token value as headline.
Generic.Inserted: Marks the token value as inserted.
Generic.Output: Marks the token value as program output (e.g. for python cli lexer).
Generic.Prompt: Marks the token value as command prompt (e.g. bash lexer).
Generic.Strong: Marks the token value as bold (e.g. for rst lexer).
Generic.Subheading: Marks the token value as subheadline.
Generic.Traceback: Marks the token value as a part of an error traceback.