Syntax of Regular Expressions

 Top  Previous 

Introduction

 

Regular Expressions are a widely-used method of specifying patterns of text to search for. Special metacharacters allow you to specify, for instance, that a particular string You are looking for occurs at the beginning or end of a line, or contains n recurrences of a certain character.

 

Regular expressions look ugly for novices, but really they are very simple (well, usually simple ;) ), handy and powerful tool. Let's start our learning trip!

 

Simple matches

 

Any single character matches itself, unless it is a metacharacter with a special meaning described below.

 

A series of characters matches that series of characters in the target string, so the pattern "bluh" would match "bluh'' in the target string. Quite simple, eh ?

 

You can cause characters that normally function as metacharacters or escape sequences to be interpreted literally by 'escaping' them by preceding them with a backslash "\", for instance: metacharacter "^" match beginning of string, but "\^" match character "^", "\\" match "\" and so on.

 

Examples:

 foobar             matches string 'foobar'

 \^FooBarPtr     matches '^FooBarPtr'

 

Escape sequences

 

Characters may be specified using a escape sequences syntax much like that used in C and Perl: "\n'' matches a newline, "\t'' a tab, etc. More generally, \xnn, where nn is a string of hexadecimal digits, matches the character whose ASCII value is nn. If You need wide (Unicode) character code, You can use '\x{nnnn}', where 'nnnn' - one or more hexadecimal digits.

 

•\xnn         char with hex code nn
•\x{nnnn}  char with hex code nnnn (one byte for plain text and two bytes for Unicode)
•\t             tab (HT/TAB), same as \x09
•\n            newline (NL), same as \x0a
•\r             car.return (CR), same as \x0d
•\f             form feed (FF), same as \x0c
•\a            alarm (bell) (BEL), same as \x07
•\e            escape (ESC), same as \x1b

 

Examples:

 foo\x20bar   matchs 'foo bar' (note space in the middle)

 \tfoobar      matchs 'foobar' predefined by tab

 

Character classes

 

You can specify a character class, by enclosing a list of characters in [], which will match any one character from the list.

 

If the first character after the "['' is "^'', the class matches any character not in the list.

 

Examples:

•foob[aeiou]r    finds strings 'foobar', 'foober' etc. but not 'foobbr', 'foobcr' etc.
•foob[^aeiou]r  find strings 'foobbr', 'foobcr' etc. but not 'foobar', 'foober' etc.

 

Within a list, the "-'' character is used to specify a range, so that a-z represents all characters between "a'' and "z'', inclusive.

 

If You want "-'' itself to be a member of a class, put it at the start or end of the list, or escape it with a backslash. If You want ']' you may place it at the start of list or escape it with a backslash.

Also, you can use predefined character classes here.

 

Examples:

•[-az]        matches 'a', 'z' and '-'
•[az-]        matches 'a', 'z' and '-'
•[a\-z]       matches 'a', 'z' and '-'
•[a-z]        matches all twenty six small characters from 'a' to 'z'
•[\n-\x0D]  matches any of #10,#11,#12,#13.
•[\d-t]       matches any digit, '-' or 't'.
•[]-a]        matches any char from ']'..'a'.

 

 

Metacharacters

 

Metacharacters are special characters which are the essence of Regular Expressions. There are different types of metacharacters, described below.

 

Metacharacters - line separators

 

•^      start of line
•$      end of line
•\A     start of text
•\Z     end of text
•.       any character in line

 

Examples:

•^foobar      matches string 'foobar' only if it's at the beginning of line
•foobar$       matches string 'foobar' only if it's at the end of line
•^foobar$     matches string 'foobar' only if it's the only string in line
•foob.r         matches strings like 'foobar', 'foobbr', 'foob1r' and so on

 

The "^" metacharacter by default is only guaranteed to match at the beginning of the input string/text, the "$" metacharacter only at the end. Embedded line separators will not be matched by "^'' or "$''.

You may, however, wish to treat a string as a multi-line buffer, such that the "^'' will match after any line separator within the string, and "$'' will match before any line separator. You can do this by switching On the modifier /m.

The \A and \Z are just like "^'' and "$'', except that they won't match multiple times when the modifier /m is used, while "^'' and "$'' will match at every internal line separator.

 

The ".'' metacharacter by default matches any character, but if you switch Off the modifier /s, then '.' won't match embedded line separators.

 

TLMDRegExpr works with line separators as recommended at unicode.org:

 

•"^" is at the beginning of a input string, and, if modifier /m is On, also immediately following any occurrence of \x0D\x0A or \x0A or \x0D (if You are using Unicode version of TLMDRegExpr, then also \x2028 or \x2029 or \x0B or \x0C or \x85). Note that there is no empty line within the sequence \x0D\x0A.
•"$" is at the end of a input string, and, if modifier /m is On, also immediately preceding any occurrence of \x0D\x0A or \x0A or \x0D (if You are using Unicode version of TLMDRegExpr, then also \x2028 or \x2029 or \x0B or \x0C or \x85). Note that there is no empty line within the sequence \x0D\x0A.
•"." matchs any character, but if You switch Off modifier /s then "." doesn't match \x0D\x0A and \x0A and \x0D (if You are using Unicode version of TLMDRegExpr, then also \x2028 and \x2029 and \x0B and \x0C and \x85).

 

Note that "^.*$" (an empty line pattern) does not match the empty string within the sequence \x0D\x0A, but matchs the empty string within the sequence \x0A\x0D.

 

Multiline processing can be easily tuned for your own purpose with help of  TLMDRegExpr properties LineSeparators and LinePairedSeparator, You can use only Unix style separators \n or only DOS/Windows style \r\n or mix them together (as described above and used by default) or define Your own line separators!

 

Metacharacters - predefined classes

 

•\w     an alphanumeric character (including "_")
•\W    a nonalphanumeric
•\d      a numeric character
•\D     a non-numeric
•\s      any space (same as [ \t\n\r\f])
•\S     a non space

 

Also, in Unicode builds we support Unicode categories:

 

•\p{L}: any kind of letter from any language.
•\p{Ll}: a lowercase letter that has an uppercase variant.
•\p{Lu}: an uppercase letter that has a lowercase variant.
•\p{Lt}: a letter that appears at the start of a word when only the first letter of the word is capitalized.
•\p{Lm} : a special character that is used like a letter.
•\p{Lo} : a letter or ideograph that does not have lowercase and uppercase variants.
•\p{M} : a character intended to be combined with another character (e.g. accents, umlauts, enclosing boxes, etc.).
•\p{Mn}: a character intended to be combined with another character that does not take up extra space (e.g. accents, umlauts, etc.).
•\p{Mc}: a character intended to be combined with another character that takes up extra space (vowel signs in many Eastern languages).
•\p{Me}: a character that encloses the character is is combined with (circle, square, keycap, etc.).
•\p{Z} : any kind of whitespace or invisible separator.
•\p{Zs}: a whitespace character that is invisible, but does take up space.
•\p{Zl}: line separator character U+2028.
•\p{Zp}: paragraph separator character U+2029.
•\p{S} : math symbols, currency signs, dingbats, box-drawing characters, etc..
•\p{Sm}: any mathematical symbol.
•\p{Sc}: any currency sign.
•\p{Sk}: a combining character (mark) as a full character on its own.
•\p{So}: various symbols that are not math symbols, currency signs, or combining characters.
•\p{N} : any kind of numeric character in any script.
•\p{Nd}: a digit zero through nine in any script except ideographic scripts.
•\p{Nl}: a number that looks like a letter, such as a Roman numeral.
•\p{No}: a superscript or subscript digit, or a number that is not a digit 0..9 (excluding numbers from ideographic scripts).
•\p{P} : any kind of punctuation character.
•\p{Pd}: any kind of hyphen or dash.
•\p{Ps}: any kind of opening bracket.
•\p{Pe}: any kind of closing bracket.
•\p{Pi}: any kind of opening quote.
•\p{Pf}: any kind of closing quote.
•\p{Pc}: a punctuation character such as an underscore that connects words.
•\p{Po}: any kind of punctuation character that is not a dash, bracket, quote or connector.
•\p{C} : invisible control characters and unused code points.
•\p{Cc}: an ASCII 0x00..0x1F or Latin-1 0x80..0x9F control character.
•\p{Cf}: invisible formatting indicator.
•\p{Co}: any code point reserved for private use.
•\p{Cs}: one half of a surrogate pair in UTF-16 encoding.
•\p{Cn}: any code point to which no character has been assigned.

 

•\P{Category} means same as \p{Category}, with one difference: it means any character NOT in given category.

 

You may use \w, \d and \s within custom character classes. Also, in Unicode builds of TLMDRegExpr you can use \p and \P there too.

 

Examples:

•foob\dr     matches strings like 'foob1r', ''foob6r' and so on but not 'foobar', 'foobbr' and so on
•foob[\w\s]r matches strings like 'foobar', 'foob r', 'foobbr' and so on but not 'foob1r', 'foob=r' and so on

 

TLMDRegExpr uses properties SpaceChars and WordChars to define character classes \w, \W, \s, \S, so You can easily redefine it.

 

Metacharacters - word boundaries

 

•\b     Match a word boundary
•\B     Match a non-(word boundary)

 

A word boundary (\b) is a spot between two characters that has a \w on one side of it and a \W on the other side of it (in either order), counting the imaginary characters off the beginning and end of the string as matching a \W.

 

Metacharacters - iterators

 

Any item of a regular expression may be followed by another type of metacharacters - iterators. Using this metacharacters You can specify number of occurrences of previous character, metacharacter or subexpression.

 

•*      zero or more ("greedy"), similar to {0,}
•+      one or more ("greedy"), similar to {1,}
•?      zero or one ("greedy"), similar to {0,1}
•{n}    exactly n times ("greedy")
•{n,}   at least n times ("greedy")
•{n,m}  at least n but not more than m times ("greedy")
•*?     zero or more ("non-greedy"), similar to {0,}?
•+?     one or more ("non-greedy"), similar to {1,}?
•??     zero or one ("non-greedy"), similar to {0,1}?
•{n}?   exactly n times ("non-greedy")
•{n,}?  at least n times ("non-greedy")
•{n,m}? at least n but not more than m times ("non-greedy")

 

So, digits in curly brackets of the form {n,m}, specify the minimum number of times to match the item n and the maximum m. The form {n} is equivalent to {n,n} and matches exactly n times. The form {n,} matches n or more times. There is no limit to the size of n or m, but large numbers will chew up more memory and slow down r.e. execution.

 

If a curly bracket occurs in any other context, it is treated as a regular character.

 

Examples:

foob.*r           matches strings like 'foobar', 'foobalkjdflkj9r' and 'foobr'

foob.+r           matches strings like 'foobar', 'foobalkjdflkj9r' but not 'foobr'

foob.?r            matches strings like 'foobar', 'foobbr' and 'foobr' but not 'foobalkj9r'

fooba{2}r       matches the string 'foobaar'

fooba{2,}r      matches strings like 'foobaar', 'foobaaar', 'foobaaaar' etc.

fooba{2,3}r    matches strings like 'foobaar', or 'foobaaar' but not 'foobaaaar'

 

A little explanation about "greediness". "Greedy" takes as many as possible, "non-greedy" takes as few as possible. For example, 'b+' and 'b*' applied to string 'abbbbc' return 'bbbb', 'b+?' returns 'b', 'b*?' returns empty string, 'b{2,3}?' returns 'bb', 'b{2,3}' returns 'bbb'.

 

You can switch all iterators into "non-greedy" mode (see the modifier /g).

 

Metacharacters - alternatives

 

You can specify a series of alternatives for a pattern using "|'' to separate them, so that fee|fie|foe will match any of "fee'', "fie'', or "foe'' in the target string (as would f(e|i|o)e). The first alternative includes everything from the last pattern delimiter ("('', "['', or the beginning of the pattern) up to the first "|'', and the last alternative contains everything from the last "|'' to the next pattern delimiter. For this reason, it's common practice to include alternatives in parentheses, to minimize confusion about where they start and end.

Alternatives are tried from left to right, so the first alternative found for which the entire expression matches, is the one that is chosen. This means that alternatives are not necessarily greedy. For example: when matching foo|foot against "barefoot'', only the "foo'' part will match, as that is the first alternative tried, and it successfully matches the target string. (This might not seem important, but it is important when you are capturing matched text using parentheses.)

Also remember that "|'' is interpreted as a literal within square brackets, so if You write [fee|fie|foe] You're really only matching [feio|].

 

Examples:

•foo(bar|foo)  matches strings 'foobar' or 'foofoo'.

 

 

Metacharacters - subexpressions

 

The bracketing construct ( ... ) may also be used for define r.e. subexpressions (after parsing You can find subexpression positions, lengths and actual values in MatchPos, MatchLen and Match properties of TLMDRegExpr, and substitute it in template strings by TLMDRegExpr.Substitute).

 

Subexpressions are numbered based on the left to right order of their opening parenthesis.

First subexpression has number '1' (whole r.e. match has number '0' - You can substitute it in TLMDRegExpr.Substitute as '$0' or '$&').

 

Examples:

 (foobar){8,10}  matches strings which contain 8, 9 or 10 instances of the 'foobar'

 foob([0-9]|a+)r matches 'foob0r', 'foob1r' , 'foobar', 'foobaar', 'foobaar' etc.

 

 

Subexpressions and substitution

 

Subexpressions are numbered based on the left to right order of their opening parenthesis.

So, you can do advanced replace with substitution subexpression names with its values in replace pattern.

See syntax used by TLMDRegExpr.Substitute:

•$0 or $& - means whole regexp text found.
•$1 - means first subexpression in regexp text found.
•$N - means Nth subexpression in regexp text found. Here, valid N range is 1..16.
•If you want place into template raw '$' or '\', use prefix '\'.
•Escape sequences, same as regexp search procedures use.

 

See code example:

 

LMDReplaceRegExpr('(\d+) \s+ (\d+) ', '10  20', '$1 + $2 = $1$2\nJoke!', true);

 

It will replace string '10 20' to '10+20=1020

Joke!'

 

Metacharacters - backreferences

 

Metacharacters \1 through \9 are interpreted as backreferences. \<n> matches previously matched subexpression #<n>.

 

Examples:

•(.)\1+         matches 'aaaa' and 'cc'.
•(.+)\1+        also match 'abab' and '123123'
•(['"]?)(\d+)\1 matches '"13" (in double quotes), or '4' (in single quotes) or 77 (without quotes) etc

 

Modifiers

         

Modifiers are for changing behaviour of TLMDRegExpr. There are many ways to set up modifiers.

Any of these modifiers may be embedded within the regular expression itself using the (?...) construct.

Also, you can assign to appropriate TLMDRegExpr properties (ModifierX for example to change /x, or ModifierStr to change all modifiers together).

 

•i:        Do case-insensitive pattern matching (using installed in you system locale settings), see also TLMDRegExpr.InvertCase.
•m:        Treat string as multiple lines. That is, change "^'' and "$'' from matching at only the very start or end of the string to the start or end of any line anywhere within the string, see also Line separators.
•s:        Treat string as single line. That is, change ".'' to match any character whatsoever, even a line separators (see also Line separators), which it normally would not match.        
•g:        Non standard modifier. Switching it Off You'll switch all following operators into non-greedy mode (by default this modifier is On). So, if modifier /g is Off then '+' works as '+?', '*' as '*?' and so on        
•x:        Extend your pattern's legibility by permitting whitespace and comments (see explanation below).

 

The modifier /x itself needs a little more explanation. It tells the TLMDRegExpr to ignore whitespace that is neither backslashed nor within a character class. You can use this to break up your regular expression into (slightly) more readable parts. The # character is also treated as a metacharacter introducing a comment, for example:

 

 (  

 (abc) # comment 1

   |   # You can use spaces to format r.e. - TLMDRegExpr ignores it

 (efg) # comment 2

 )

 

This also means that if you want real whitespace or # characters in the pattern (outside a character class, where they are unaffected by /x), that you'll either have to escape them or encode them using octal or hex escapes. Taken together, these features go a long way towards making regular expressions text more readable.

                 

Perl extensions

 

•(?imsxr-imsxr) You may use it into r.e. for modifying modifiers by the fly. If this construction inlined into subexpression, then it effects only into this subexpression

 

Examples:

•(?i)Saint-Petersburg       matches 'Saint-petersburg' and 'Saint-Petersburg'
•(?i)Saint-(?-i)Petersburg  matches 'Saint-Petersburg' but not 'Saint-petersburg'
•(?i)(Saint-)?Petersburg    matches 'Saint-petersburg' and 'saint-petersburg'
•((?i)Saint-)?Petersburg    matches 'saint-Petersburg', but not 'saint-petersburg'

 

•(?#text) A comment, the text is ignored. Note that TLMDRegExpr closes the comment as soon as it sees a ")", so there is no way to put a literal ")" in the comment.

 

•(?:  ...   ) Non-capturing group: match of this group will not occur in TLMDRegExpr.Matches collection, and will not participate in \1..\9 backreference expressions.
•(?=  .... ) Positive lookahead: q(?=u) matches a q that is followed by a u, without making the u part of the match. The positive lookahead construct is a pair of round brackets, with the opening bracket followed by a question mark and an equals sign. Note: now TLMDRegexpr not supports such complex iterators for lookahead group as: (?= .. ){min, max}, (?= ...)+? (non-greedy), (?= ...)*? (non-greedy). Only constructs we support now are (?= ...)+ or (?= ...)* - simple greedy iterators.  To use complex iterators, you should write something like this: (  (?= ....)  ){1, 2}?
This will be fixed in future versions of TLMDRegExpr

 

Special syntax for LMD SyntaxEdit parser

 

In <RegexBlock> and <SyntaxBlock>, in end attribute, you can use syntax $0..$9 as reference to correspondent start group match. See example here, and details here.Also, in <SyntaxBlock> for regexps of syntax tokens, you should use limited number of constructs as described here