Featured image of post Mastering Regular Expressions and Leaving the Beginner Stage Behind

Mastering Regular Expressions and Leaving the Beginner Stage Behind

Regular expressions are a powerful method for text matching and retrieval when processing text

Preface

Regular Expression is a powerful language for describing text patterns. It uses concise combinations of symbols to describe complex string-matching rules. It is widely used in data validation, text extraction, log analysis, and other fields, and plays an important role in our daily work.

In the software industry, whether you have mastered regular expressions is often an important marker that separates a rookie from a veteran. In this article, we walk through the application of regular expressions with concrete examples so you can leave the beginner stage behind as soon as possible.

Core Syntax of Regular Expressions

To learn regular expressions, a very intuitive online validation platform is recommended here, https://regex101.com , which is very helpful for practicing and mastering regex syntax.

Ordinary Characters and Special Characters

A regular expression is a matching rule. Naturally, it can first match all kinds of ordinary characters, mainly Chinese and English words, letters, and digits. In addition, regular expressions reserve many special characters, such as * ? + . () [] $ ^ \ |, which represent special matching rules. When we need to treat these special characters as ordinary characters to be matched, we need to use the \ symbol to escape them, so \ is also called the escape character. For example, \? will match a ? character in the target text, and \\ will match a \ character in the target text

Quantifiers

Quantifiers are symbols used to control the number of matches:

  • * zero or more times (e.g. a* can match “”, “a”, “aa”)
  • + one or more times (a+ must have at least one a)
  • ? zero or one time (a? can match "" or a)
  • {n} matches exactly n times ({4} matches 4 times)
  • {m,n} matches m to n times (inclusive)
  • {m,} matches m or more times
  • {,n} matches zero to n times

In addition, there are two matching modes in regular expressions

  • Greedy mode: this is the default mode, which matches the longest possible run of characters that fit the rule in the target text
  • Lazy mode: if a ? is added after a quantifier, it marks a lazy match, that is, matching as few characters as possible that fit the rule

For example:

浮云长长长长长长长消

This couplet contains 7 长 characters. The regular expression 长{2,3}, in the default greedy mode, will match 长长长 2 times, while 长{2,3}? matches as few characters as possible and will match 长长 3 times

Metacharacters

Besides quantifiers, there is another class of metacharacters, the fundamental special symbols reserved to build regular expressions:

  • \d matches a digit (equivalent to [0-9])
  • \D matches a non-digit
  • \w matches a word character, including letters, digits, and underscore (equivalent to [A-Za-z0-9_])
  • \W matches a non-word character
  • \s matches a whitespace character (space/tab/newline)
  • \S matches a non-whitespace character
  • . matches any single character (by default it does not match a newline)
  • | means logical “or”

Position Anchors

Used to precisely locate the matching position:

  • \b matches a word boundary (e.g. \bcat\b matches a whole word)
  • \B matches a non-word-boundary
  • ^ matches the start of the target text, and in multiline mode also matches the start of a line
  • $ matches the end of the target text, and in multiline mode also matches the end of a line

For example: In tester is good at testing, using ^test will only match the test at the start of the line, but the test in testing will not be matched; and if test\b is used, it still matches nothing, because in this text test is never a standalone word and there is no word boundary after test

Character Classes

Character classes are used to batch-match a specific set of characters:

  • [abc] matches any one of a/b/c
  • [^abc] matches characters not in the set
  • Range notation: [a-z] (lowercase letters), [A-Z] (uppercase letters), [0-9] (digits)

In the following example, [b-f] matches only the d,e in the target string

Groups

Grouping is an important technique in regular expressions. It groups matching rules so that matched content can be extracted in subsequent text processing. A group is denoted by enclosing a pattern in () within the expression.

For example, specify a string

1
this is a "testtest" string to demo regex group! 

Groups and Their References

Groups are numbered starting from 1 by default, that is, when there are multiple groups, they can be back-referenced in order with \1 \2

In this example, we specify two groups with (is)(\s)\1\2, group 1 matches “is”, group 2 matches a whitespace character (here a space), and then \1\2 references these two groups respectively; in effect it matches the case where "is"+space appears twice in a row

Marking a Group as Non-Capturing

When we group, if we do not want a certain group to be referenced later, we can use (?:pattern) to mark it, specifying that the current group will not be referenced later. This group is then ignored in the default capture numbering and is not assigned a number.

In the above example, if we specify that the second group is not to be referenced

1
(is)(?:\s)\1\2 

Then the regular expression in the above example actually has a syntax error and cannot complete the match, because the second group is ignored while \2 references it, which causes a pattern error due to its absence

Adding Comments to Groups

For some complex regular expressions, we can also add comments to a group to explain the regular expression (?#comment)(other group)

Naming Groups

Besides referencing groups by their default numeric index, you can also name a group: (?P<name>pattern). Then reference it with (?P=<name>)

Based on the above, use the following regex as an illustration

1
(?#这是一个嵌套分组示例)(?P<demo>t(?:\w*)t)(?P=demo)

Assertions

Assertions are mainly used for judgment, to match the position of characters that fit a pattern. Because they mainly judge positions and do not consume characters, they are generally also called zero-width assertions

Assertions can be divided into 4 categories, still using the text from the above example

1
this is a "testtest" string to demo regex group! 

Positive Lookahead Assertion

Judges that the characters after the matching position fit the pattern, asserted with (?=pattern)

Example:

Negative Lookahead Assertion

Judges that the characters after the matching position do not fit the given pattern, asserted with (?!pattern)

Example:

Positive Lookbehind and Negative Lookbehind Assertions

Opposite to lookahead, lookbehind assertions match whether the characters before the expected position fit a pattern, and also come in positive and negative forms. Adding a < after the ? indicates lookbehind, namely (?<=pattern) and (?<!pattern)

Matching Modes

Besides the main matching rules of regular expressions above, regular expressions also have some matching modes. They are represented by \ plus some special letters for different modes

  • \g indicates global matching, that is, it matches all qualifying substrings in the given text; if not specified, it only matches the first substring that fits the pattern
  • \m multiline mode, mainly specifies whether ^ $ match the start and end of a line
  • \i ignore case when matching
  • \x whether to ignore whitespace in our pattern text when matching, useful when writing more complex rules
  • \s specifies that . includes the newline character when matching characters
  • \u matches unicode characters, such as Chinese
  • \a matches only ascii characters

Practical Cases

For the following text:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
用户注册信息如下:
联系电话:+86 138-0013-8000(北京) / 400-800-8888(全国客服)
电子邮箱:support@example.com / service@sub.domain.org.cn / user.name+tag+sorting@example.co.uk
网站链接:
https://www.example.com/page1.html?query=123&sort=desc#section2
http://sub.example.co.jp:8080/admin/dashboard
ftp://ftp.example.org/upload.zip
产品序列号:SKU-20240309-001234(生产日期:2024-03-09)
INV-2023-Winter-Sale-0456(有效期至:2024-01-31)
活动时间:2024年第一季度(2024-01-01 至 2024-03-31)
特殊字符测试:!@#$%^&*()_+-=[]{}|;':",./<>?`~
测试用密码:Password123!@#(正确) / password123(错误) / Passw0rd!(错误)
日志条目:ERROR[2024-03-09T15:40:22+08:00] 用户登录失败
IP地址记录:
192.168.1.1 / 203.0.113.255 / ::1 / IPv6-Test-Address::FFFFF:1234

Matching a Complete URL (Including Protocol, Domain, Port, and Path)

1
^(https?:\/\/)([\w\-\.]+\.[a-z]{2,6})(:[0-9]{1,5})?(\/\S*)?$

Matching result:

Extracting Phone Numbers

1
\b(\+?\d{2,3})[-\s]+?(?:\d{3,4}[- ]?)+

Explanation: first \b matches a word boundary, a group extracts the country code (2-3 digits, compatible with the + case), followed by - or a space, in lazy mode, then matches the case where 3-4 digits and - appear consecutively multiple times

Matching result:

  • +86 138-0013-8000
  • 400-800-8888

Validating Email Addresses

1
[-.\w]+@[-.\w]+\.[-.\w]+\b

Matching result:

  • support@example.com
  • service@sub.domain.org.cn
  • sorting@example.co.uk

Capturing Product Serial Numbers (Including Date and Unique Code)

1
SKU-\d{8}-\d{6}|INV-\d{4}[-A-Za-z]+-\d{4}

Matching result:

  • SKU-20240309-001234
  • INV-2023-Winter-Sale-0456

Matching Chinese Characters

1
[\u4e00-\u9fa5]

Parsing the Timestamp in a Log Entry

1
ERROR\[(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2})\]

Group extraction result: 2024-03-09T15:40:22+08:00

Extracting IP Addresses

1
(?:\d{1,3}\.){3}\d{1,3}|(?:(?:[A-Fa-f0-9]{1,4}:){7}[A-Fa-f0-9]{1,4})

Matching result:

  • 192.168.1.1
  • 203.0.113.255

Conclusion

The above is a summary of regular expressions and a demonstration of some cases. For more on systematically improving your testing skills, follow my course and reply with outline to get the detailed syllabus.

Built with Hugo
Theme Stack designed by Jimmy