677 - URI Encoding

With the book club this past semester I read Daphne du Maurier's Rebecca, Gerald Durrell's My Family and Other Animals, and most of Tony Mendez's Argo. I also read Cormac McCarthy's All the Pretty Horses.

Percent-Encoding Guide

According to STD 66 RFC 3986, the characters of the string !#$&'()*+,/:;=?@[] could have a special meaning in a URI and are reserved. The ASCII alphanumeric characters and those contained in -._~ are unreserved. Any character outside of these two sets must be percent-encoded before inclusion in a URI. This is what the JavaScript encodeURI() function does, except that it also encodes the square brackets [], which were not yet included in the set of URI characters when the superseded RFC 2396 was written.

The unreserved characters can always be left unencoded, so we just need to encode some subset of the reserved characters. This subset depends on the URI scheme being used and where in the URI the characters are. The function encodeURIComponent() encodes everything except for the characters of !'()*, which often don't need to be encoded as they weren't yet reserved in RFC 2396, and the unreserved characters. This means thirteen of the reserved characters are encoded, but we can encode a smaller subset in the following cases.

Data URIs

RFC 2397 states that the main content section of a data URI will consist of some number of 'uric' characters, and that these characters are defined in RFC 2396. According to that definition, any percent-encoded or unreserved character is allowed, in addition to all but #[] of the reserved characters.

unreserved = '-' | '.' | [0-9] | [A-Z] | '_' | [a-z] | '~'
2396-reserved = '$' | '&' | '+' | ',' | '/' | ':' | ';' | '=' | '?' | '@'
previously-unreserved = '!' | ''' | '(' | ')'| '*'
3986-reserved = '#' | '[' | ']' | 2396-reserved | previously-unreserved
hex = [0-9] | [A-F] | [a-f]
escaped = '%' hex hex
encodeURIComponent = unreserved | previously-unreserved | escaped
2396-uric = 2396-reserved | encodeURIComponent
encodeURI = '#' | 2396-uric

Above is a more formal description of the character sets. It shows that every character in the output of encodeURI() will either be the number sign U+23 or a 'uric' character. This means we can call encodeURI(), then replace all occurrences of # with %23, and the output will be valid within a data URI.

The code below shows how an SVG data URI might be constructed using JavaScript.

const rectSVG = String.raw`
<rect fill="#69E" x="20" y="20" width="15.2" height="87"/>
<text fill="Gray" x="19" y="19">\o/</text>
`;
const rectDataURI = 'data:image/svg+xml,' + encodeURI(rectSVG).replaceAll('#', '%23');

The resulting string is data:image/svg+xml,%3Csvg%20xmlns=%22http://www.w3.org/2000/svg%22%20viewBox=%220%200%20128%20128%22%3E%0A%09%3Crect%20fill=%22%2369E%22%20x=%2220%22%20y=%2220%22%20width=%2215.2%22%20height=%2287%22/%3E%0A%09%3Ctext%20fill=%22Gray%22%20x=%2219%22%20y=%2219%22%3E%5Co/%3C/text%3E%0A%3C/svg%3E.

Note that String.raw() is helpful if the data contains backslashes, but if it contains backticks or the substring ${ then you can no longer simply paste the data into a template literal.

String.raw`_\`_\${_${"`"}_` === "_\\`_\\${_`_"

Query Strings

The query part of a URI begins after a question mark. It is composed of the 'query' characters defined in RFC 3986, and these are exactly the same as the 'uric' characters. However, the characters &+= have special purposes. The query string is a set of 'key=value' pairs, separated by ampersands, and in which plus signs represent spaces. The equals sign needs to be encoded in the 'key' portion, but not in the 'value' portion as only the first equals sign separates the two parts. Encode the data as you would for a data URI, then handle these three special characters, and as a final step we can change the encoding of spaces from %20 to +. This data URI contains two links which compare the encoding of the reserved characters and the space character in a data URI and in a query string.

const style = `background-image:url("${rectDataURI}");color-scheme:light dark`;
const vertices = '[[1,0],[0.58,0.58],[0,1],[-0.58,0.58],[-1,0],[-0.58,-0.58],[0,-1],[0.58,-0.58]]';

function encodeQueryValue(val) {
return encodeURI(val).replace(/[#&'+]|%20/g, char => ({
'#': '%23',
'&': '%26',
"'": '%27',
'+': '%2B',
'%20': '+'
}[char]));
}

const query = `?v=${encodeQueryValue(vertices)}&i=${encodeQueryValue(style)}&h=6`;
const mirrorPolygonURL = 'https://home.6t.lt/66c/mirror-polygon.svg' + query;

The above code also encodes the single quote character, as this GitHub issue suggests doing so in some cases. The code generates the URL https://home.6t.lt/66c/mirror-polygon.svg?v=%5B%5B1,0%5D,%5B0.58,0.58%5D,%5B0,1%5D,%5B-0.58,0.58%5D,%5B-1,0%5D,%5B-0.58,-0.58%5D,%5B0,-1%5D,%5B0.58,-0.58%5D%5D&i=background-image:url(%22data:image/svg%2Bxml,%253Csvg%2520xmlns=%2522http://www.w3.org/2000/svg%2522%2520viewBox=%25220%25200%2520128%2520128%2522%253E%250A%2509%253Crect%2520fill=%2522%252369E%2522%2520x=%252220%2522%2520y=%252220%2522%2520width=%252215.2%2522%2520height=%252287%2522/%253E%250A%2509%253Ctext%2520fill=%2522Gray%2522%2520x=%252219%2522%2520y=%252219%2522%253E%255Co/%253C/text%253E%250A%253C/svg%253E%22);color-scheme:light+dark&h=6.

If you want to encode the whole query string at once, then the reserved characters to be encoded are #'+[], and any ampersands or equals signs could be manually encoded if necessary. Just remember that all extra encoding has to be done after using encodeURI() to avoid double encoding.

Since the square brackets were more recently reserved, I thought they might be allowed in data URIs, but as of now a link like <a href="data:,%23[]">#[]</⁠a> is flagged for an illegal character error by the W3C markup validator. I saw a few GitHub issues like this one in support of unescaped square brackets, so they may be allowed in the future.

End of 2024 Changes

Made a basic HTML Viewer webpage, as an alternative to an HTML data URI. I've set it up at the domain '6t.lt': link.

Started posting bike ride plans for around Amherst on my cycling repository GitHub page.

Uploaded an SVG version of the UMass Recreational Math Club logo.