○–○––○–○ blog

Percent-Encoding Guide

According to STD 66 RFC 3986¹, the characters of the string !#$&'()*+,/:;=?@[] could have a special meaning in a URI and are reserved. The ASCII alphanumeric characters and those contained in -._~ are unreserved. Any character outside of these two sets must be percent-encoded before inclusion in a URI. This is what the JavaScript encodeURI() function does, except that it also encodes the square brackets [], which were not yet included in the set of reserved or unreserved characters when the superseded RFC 2396 was written.

The unreserved characters can always be left unencoded, but we may need to encode a subset of the reserved characters.² This subset depends on the URI scheme being used and where in the URI the characters are. The function encodeURIComponent() encodes everything except for the characters of !'()*, which might not need to be encoded as they weren't yet reserved in RFC 2396, and the unreserved characters. This means thirteen of the reserved characters are encoded, but we can encode a smaller subset in the two following cases.

Data URIs

RFC 2397 states that the main content section of a data URI will consist of zero or more urlchar characters, and that these characters are defined in RFC 2396. The definition is restated below as 2396-uric, and allows any percent-encoded or unreserved character, in addition to all but #[] of the reserved characters.

unreserved = '-' | '.' | [0-9] | [A-Z] | '_' | [a-z] | '~'
2396-reserved = '$' | '&' | '+' | ',' | '/' | ':' | ';' | '=' | '?' | '@'
previously-unreserved = '!' | ''' | '(' | ')'| '*'
3986-reserved = '#' | '[' | ']' | 2396-reserved | previously-unreserved
hex = [0-9] | [A-F] | [a-f]
escaped = '%' hex hex
encodeURIComponent = unreserved | previously-unreserved | escaped
2396-uric = 2396-reserved | encodeURIComponent
encodeURI = '#' | 2396-uric

This more formal description of the character sets shows that every character in the output of encodeURI() will either be the number sign U+23 or a urlchar character. This means we can call encodeURI(), then replace all occurrences of # with %23, and the output will be valid within a data URI.

The code below shows how an SVG data URI might be constructed using JavaScript.³

const rectSVG = String.raw`<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 128 128">
	<rect fill="#69E" x="20" y="20" width="15.2" height="87"/>
	<text fill="Gray" x="19" y="19">\o/</text>
</svg>`;
const rectDataURI = 'data:image/svg+xml,' + encodeURI(rectSVG).replaceAll('#', '%23');

The resulting string is data:image/svg+xml,%3Csvg%20xmlns=%22http://www.w3.org/2000/svg%22%20viewBox=%220%200%20128%20128%22%3E%0A%09%3Crect%20fill=%22%2369E%22%20x=%2220%22%20y=%2220%22%20width=%2215.2%22%20height=%2287%22/%3E%0A%09%3Ctext%20fill=%22Gray%22%20x=%2219%22%20y=%2219%22%3E%5Co/%3C/text%3E%0A%3C/svg%3E.

The WHATWG has published a section on data URLs. If you write a valid URL string, it will be converted to a URL record by the URL parser before being processed as a data URL. This processing discards the fragment component of the URL, which begins after U+23 (#), so leaving a number sign unencoded will cut the data short. The rest of the valid URL string may include the following parts: data:, URL-path-segment string, U+2F (/), U+3F (?), URL-query string. All of these parts are formed from URL units, which is a set of characters equal to 2396-uric plus most non-ASCII Unicode characters. The fact that the URL parser splits a data URL into parts has no effect on the data. You can leave forward slash and question mark delimiters unencoded, as the parts will be rejoined by the URL serializer during data URL processing.

In this case, both the RFC 2397 and WHATWG definitions of validity treat ASCII characters in effectively the same way. The difference is that the WHATWG allows unencoded non-ASCII.

Query Strings

From RFC 3986 Section 3.4:

The query component is indicated by the first question mark ("?") character and terminated by a number sign ("#") character or by the end of the URI.

Continuing the description of character sets from the Data URIs section, we can add the definition for URI query components as given in RFC 3986.

sub-delims = '!' | '$' | '&' | ''' | '(' | ')' | '*' | '+' | ',' | ';' | '='
3986-pchar = ':' | '@' | unreserved | escaped | sub-delims
query = *( '/' | '?' | 3986-pchar )

It turns out that the characters allowed within the query are exactly the 2396-uric characters.

A method to store data within the query is not specified in RFC 3986, but usually the characters &+= have special purposes. By reading the query contents with something like the JavaScript URLSearchParams interface, you interpret the query string as a set of key=value pairs. These pairs are separated by U+26 (&) ampersands, and U+2B (+) plus signs are used throughout the string to represent spaces. See the WHATWG's definition of the application/x-www-form-urlencoded parser for an exact description of this process.

The equals sign U+3D (=) needs to be encoded within the key portion of a pair, but not in the value portion, as only the first equals sign separates the two parts. Encode the data as you would for a data URI, then handle these three special characters, and as a final step we can change the encoding of U+20 spaces from %20 to +.

const style = `background-image:url("${rectDataURI}");color-scheme:light dark`;
const vertices = '[[1,0],[0.58,0.58],[0,1],[-0.58,0.58],[-1,0],[-0.58,-0.58],[0,-1],[0.58,-0.58]]';

function encodeQueryValue(val) {
	return encodeURI(val).replace(/[#&+]|%20/g, char => ({
		'#': '%23',
		'&': '%26',
		'+': '%2B',
		'%20': '+'
	}[char]));
}

const query = `?v=${encodeQueryValue(vertices)}&i=${encodeQueryValue(style)}&h=6`;
const mirrorPolygonURI = 'https://home.6t.lt/66c/mirror-polygon.svg' + query;

The above code generates the URI https://home.6t.lt/66c/mirror-polygon.svg?v=%5B%5B1,0%5D,%5B0.58,0.58%5D,%5B0,1%5D,%5B-0.58,0.58%5D,%5B-1,0%5D,%5B-0.58,-0.58%5D,%5B0,-1%5D,%5B0.58,-0.58%5D%5D&i=background-image:url(%22data:image/svg%2Bxml,%253Csvg%2520xmlns=%2522http://www.w3.org/2000/svg%2522%2520viewBox=%25220%25200%2520128%2520128%2522%253E%250A%2509%253Crect%2520fill=%2522%252369E%2522%2520x=%252220%2522%2520y=%252220%2522%2520width=%252215.2%2522%2520height=%252287%2522/%253E%250A%2509%253Ctext%2520fill=%2522Gray%2522%2520x=%252219%2522%2520y=%252219%2522%253E%255Co/%253C/text%253E%250A%253C/svg%253E%22);color-scheme:light+dark&h=6.

If you want to encode the whole query string at once, instead of individual values, then the reserved characters to be encoded are #+[]. You can't encode all ampersands and equals signs as they may be present and acting as delimiters. Any U+26 (&) or U+3D (=) characters intended as data would be manually encoded if necessary, and all of this encoding must be done after using something like encodeURI() to avoid double encoding.

Just as with data URLs, you can leave most non-ASCII characters unencoded in the query component while still complying with the WHATWG's rules for writing URLs.

Endnotes

As of writing, there is an open issue on the WHATWG URL Standard's GitHub page in support of unescaped square brackets. The definition of a valid URL string may change in the future, as some ASCII characters are frequently used in URLs despite being disallowed.

The URL Specification by Alwin Blok is a condensed description of URLs which I found helpful.

I also wrote a webpage comparing encoding methods for data within a query value.