Featured image of post A Plain-Language Guide to the HTTPS Protocol

A Plain-Language Guide to the HTTPS Protocol

Why is HTTPS more secure than HTTP?

When you access the network and browse web pages these days, if you pay attention, you will notice that almost every URL starts with the https:// prefix. This prefix indicates that the website uses the HTTPS protocol. So how exactly should we understand HTTPS? In this article, we will try to explain clearly the protocol that is currently the most widely used of all.

Understanding the HTTP Protocol

To explain the HTTPS protocol, we first need to understand the HTTP protocol. The HTTP protocol can be said to be the foundation of the modern Internet. In the OSI seven-layer model of modern networks, HTTP is a typical Layer 7 application-layer protocol. That is to say, it is a practical application built on network communication; for example, a browser rendering web page content is a concrete application of network communication.

The following figures show the protocol definition formats of the HTTP request message and the response message respectively:

A request to Baidu is parsed as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
GET https://www.baidu.com/ HTTP/1.1
#Request method, URL, HTTP protocol version
Host: www.baidu.com
#Requested server address
#The following are the message header fields
Connection: keep-alive
#Connection method: persistent connection
Cache-Control: max-age=0
#Request cache control; confirm whether the requested content has been modified
Upgrade-Insecure-Requests: 1
#Supports the HTTPS protocol
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
#Requesting client, browser version
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
#Supported response content types
Accept-Encoding: gzip, deflate, br
#Supported encoding types
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
#Supported languages
Cookie: BAIDUID=C0A2...
#Cookie information carried
#No request message body is carried; body is empty

The response is parsed as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
HTTP/1.1 200 OK
#Protocol version, status code, status description
Bdpagetype: 2
Bdqid: 0x8707d7d80001f34e
#Custom message headers
Cache-Control: private
#Cache control policy
Connection: Keep-Alive
#Persistent connection
Content-Type: text/html;charset=utf-8
#Response content type
Date: Sat, 22 Dec 2018 08:54:51 GMT
#Response time
Expires: Sat, 22 Dec 2018 08:54:51 GMT
#Expiration time
Server: BWS/1.1
#Server system and version
Set-Cookie: BDSVRTM=372; path=/
Set-Cookie: BD_HOME=1; path=/
Set-Cookie: path=/; domain=.baidu.com
#Set client cookie
Strict-Transport-Security: max-age=172800
#Strict transport security, validity period
X-Ua-Compatible: IE=Edge,chrome=1
#Compatible browser versions
Content-Length: 191722
#Message body length
#The following is the message body content
<!Doctype html>
<html xmlns=http://www.w3.org/1999/xhtml><head>
...

For a detailed explanation of the HTTP protocol, you can refer to the section on HTTP protocol basics in my earlier blog post API Testing Tool Postman - Basics

But here the requested URL is actually https://www.baidu.com, which uses the HTTPS protocol. So why do we still say it is the HTTP protocol? What exactly is the relationship between the two?

The HTTPS Protocol, TLS, and SSL

In fact, the HTTPS protocol stands for HyperText Transfer Protocol over Secure Socket Layer, that is, the HTTP protocol layered on top of SSL.

SSL (Secure Socket Layer) refers to the Secure Sockets Layer, a standard protocol proposed by Netscape, a pioneer company of the Internet, to provide security and privacy protection for network communications. After the protocol evolved to version SSL 3.0, the standards organization IETF standardized the SSL protocol and renamed it TLS (Transport Layer Security). It has currently been updated to version TLS 1.3.

So SSL and TLS actually come from the same lineage; the two can be regarded as the same protocol at different points in time. And the HTTPS protocol is essentially:

HTTPS = HTTP + SSL/TLS

Why do we need the HTTPS protocol? Because with the HTTP protocol itself, the information transmitted during transfer is essentially in plain text, and any node along the transmission path can actually see the content of the message body being transmitted.

Therefore, for security reasons, there arose a need to encrypt the transmitted content. The role of the SSL/TLS protocol is precisely to protect the security of this content.

As can also be seen from the naming of the TLS protocol, it is in fact a protocol that adds security features on top of the TCP protocol (Transmission Control Protocol).

The HTTPS protocol is based on the HTTP protocol, so it still belongs to the application layer, while TLS is built on the TCP protocol at Layer 4 and is generally considered to belong to Layer 5, the session layer.

Through TLS, the encryption of communication data during network transmission can be achieved

About Encryption

So to understand the HTTPS protocol, what we need to figure out here is how TLS completes the encryption of communication data.

Symmetric Encryption

Our usual understanding of encryption is that, through a rule agreed upon by both parties—namely an encryption algorithm—the original string is transformed into ciphertext known only to the two communicating parties. In this way, even if others know the communication content, they cannot know the original information.

For example, in spy thrillers, passing intelligence usually involves an important item called a codebook. Here the codebook is essentially the encryption algorithm, that is, the key.

In this encryption method, the two parties are on an equal footing; the same key is used to encrypt and decrypt the communication content, hence it is called symmetric encryption

However, this encryption method has one biggest problem, namely the codebook here. In spy thrillers, intelligence agents risk their lives to protect the codebook; otherwise, once it is leaked, the intelligence is at risk of being exposed.

The same applies to network communication: the biggest problem here is how to prevent the key from being leaked. But as long as the key itself also needs to be transmitted over network communication, it is still not secure.

This process is analogous to:

1
2
3
4
A: 跟你说个秘密,我下面的话你不要直接听,我全部说第几页第几行第几列,你去《红楼梦》找到对应的字连起来读。你回话也告诉我对应字的位置就行
B: 好嘞,《红楼梦》已摆好,你说吧
。。。。
C:我偷听到啦,赶快买本《红楼梦》去...

Asymmetric Encryption

Besides symmetric encryption, there is another encryption method called asymmetric encryption. This approach uses some special encryption algorithms to generate two keys, divided into a public key and a private key. Content encrypted with one of the keys can only be decrypted with the other key. Moreover, it is very difficult for this algorithm to be reverse-engineered, deduced, or forged through brute-force attacks so as to reproduce the other key.

This is similar to the ancient tiger tally used for deploying troops: only when the two halves fit together perfectly can it be verified as authentic.

With this mechanism, the private key can be kept on one’s own side and never disclosed, while the public key can be announced to the world and transmitted normally over the network, so that anyone may obtain it. In this way, when the other party wants to send me encrypted information, they only need to encrypt it with the public key, and decryption can only be completed on my own side.

So the general process of asymmetric encryption is:

1
2
3
4
5
6
 A: 我有点秘密想跟你说, 不能被其他人听到....
 B:稍等,我给你个公钥,你上个锁以后再说给我听。拿到公钥先念下公钥上的字给我看看对不对
 A: 收到公钥,上面写着“我是王大锤”
 B: 对,你收到的就是我给你的公钥,用它上锁吧
 A:公钥加密上锁,打包发给B
 B:取出私钥,打开包裹,阅读内容...

Compared with symmetric encryption, asymmetric encryption is clearly more secure, but its drawback is that it involves a large amount of computation, the encryption and decryption process is inefficient, and the speed is relatively slow. Therefore, in practical applications, the two methods are usually used in combination.

That is, the TLS encryption process:

The two parties first negotiate a symmetric key for communication through an asymmetric encryption process; the generation of this symmetric key is completed locally on each side. After the key negotiation is complete, both parties each have an identical symmetric key, and subsequent symmetric encrypted communication can be carried out based on this same key.

Because this symmetric key is obtained through asymmetric encryption, it is not transmitted during the communication between the two parties. In this way, both security and efficiency are achieved.

Signatures and Certificates

However, there is still a vulnerability when using asymmetric encryption.

That is, during communication between the server and the client, the communication actually passes through a large number of intermediate links. If a malicious hijacker exists, after receiving the server’s information it wants to impersonate the server itself, tamper with the information, and then send it to the client.

In this case it can actually disguise itself as the server, issue its own public key to the client, and after obtaining the client’s information, decrypt it with its own private key, then re-encrypt it with the real server’s public key and send it to the server.

This risk is actually the man-in-the-middle attack vulnerability in the security field.

To avoid this problem, the server needs to prove its own identity, and the client must also be able to verify that identity. Hence signatures and certificates came into being.

A signature is the server’s way of asserting its identity; a client that obtains the public key can use the signature to determine that the sender is the correct server. Here the signature and the public key are actually both stored in a single file, namely the server’s signed certificate.

To prevent a man-in-the-middle from forging certificates, CA (Certificate Authority) is introduced, that is, an authoritative third-party certificate issuance organization. These authoritative third-party issuance organizations encrypt the server’s signature with their private key, so that the client can use the organization’s public key to verify whether the signature corresponds to the intended server. Once verification succeeds, the public key carried in the certificate can be used to complete the subsequent communication negotiation with the server.

The HTTPS Protocol Interaction Process

With the above foundations, let us take a concrete look at the specific communication process of accessing an HTTPS website.

Because TLS has different versions and supports a variety of key exchange and encryption algorithms, before communication it must first complete processes such as deciding which encryption algorithm to use and which key generation method to use. And since TLS is based on the TCP protocol, after the typical TCP three-way handshake completes the connection establishment, the TLS handshake and connection establishment process takes place:

  • For example, when a client accesses a website from a browser, the browser first sends a Client Hello message to the server. This message mainly contains the cipher suites supported by the client, the TLS version information, and the client random number. (transmitted in plain text)
  • After receiving the message, the server returns several items, including its supported cipher suites, the TLS version, its own digital certificate, and a random number generated by the server. (transmitted in plain text)
  • The client verifies the received digital certificate. After successful verification, it generates a new key, namely the pre-master key, and uses the public key carried in the certificate to encrypt the pre-master key and send it to the server. (at this point it is transmitted using asymmetric encryption)
  • The server receives the ciphertext from the client, decrypts it with its private key, and obtains the pre-master key. (transmitted using asymmetric encryption)
  • Up to this point, both the server and the client have three sets of numbers: the client random number, the server random number, and the pre-master key. Although the first two random numbers are transmitted in plain text and carry a risk of leakage, the pre-master key is not transmitted during communication; it is a secure key obtained by computation. Therefore the three are combined and, using the specific encryption algorithm negotiated earlier, another key is generated. This key is called the shared secert, which is the subsequent session key and the key used for the subsequent symmetric encryption.
  • After the client computes the symmetric encryption key, it uses that key to symmetrically encrypt the communication content and informs the server that this key will be used for symmetric encryption from then on. (transmitted using symmetric encryption)
  • After receiving the ciphertext, the server uses the previously computed key to perform symmetric decryption. Once decryption succeeds, it likewise uses that key for symmetric encrypted communication, and informs the client that the key is confirmed correct and can be used for communication. (transmitted using symmetric encryption)

At this point, the entire TLS handshake process is complete, and symmetric encrypted communication can begin.

If you capture packets with WireShark, the process usually looks like this:

For the specific interaction process, refer to the diagram below (image from the web


The above is an overall rundown of the HTTPS protocol communication process. For more articles on testing and technology, feel free to follow Chengxia Qiucao’s Blog or the WeChat official account (Qiucao on Testing)

Built with Hugo
Theme Stack designed by Jimmy