LZ-based Compression Algorithm

Peter SchmalfeldtPeter Schmalfeldt
13 min read

If you are looking for solutions to work around the platform limits for Session Storage, here is a Helper Script for Salesforce Commerce Cloud (SFCC) to compress and decompress Objects and Strings.

This uses LZ-String to work around SFCC Character Limits by using Compression, as mentioned in this Trailhead Module.

SFCC Session Limits:

  • Strings are limited to 2000 characters

  • Session is limited to 4000 characters in total size

Helper

Create a new file named lzStringHelper.js and place it into your client cartridge scripts helper folder cartridge/scripts/helpers/lzStringHelper.js within your project.

Get the Source Code Below

Usage

Compressing UTF-8 Strings and Objects

Here is an example of how to use the helper file to compress a string.

// Require LZString Helper
var LZString = require('*/cartridge/scripts/helpers/lzStringHelper');

// Fake SFCC Object
var obj = {
  zipcode: '33705',
  storeId: '0083',
  storeName: 'Store Name',
  orderNo: '10000123',
  paymentMethod: 'CREDIT_CARD'
};

// Fake String
var str = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';

// Save Object to Session ( this will automatially be converted to a string )
session.custom.myCompressedObject = LZString.compress(obj);

// Save String to Session
session.custom.myCompressedString = LZString.compress(str);

Compression Results:

session.custom.myCompressedObject

㞂⁞ॠฌ惶İ⦈ׂ̰惬`⬈Ð耳耮瀄鈉ꈮ렁솱晓䁲Șୢ鴀攮䠀ᇴᇚ눪?〈쳙銀䳬䥅큀㭲Ť鎐Ũ醀愀䨁䐀諐ʠ὚쁁宎䀅

  • Decompressed: 112 Bytes

  • Compressed: 56 Bytes

  • Reduction: 50%

session.custom.myCompressedString

ಇ끎੠똀䂖p㎁嵠ጐۜ㉜ű聃⠠‚᠆㄀㬤⚠ӵᎉ櫣ꁳ᠐눒꣝ౌ䋠ꉅЄ限ዣ먔璈ꈥ颀⍰ႅᓃ鈌⣅淪≄䁇ᓄǐ쀊ꑂᴸ놈䥁쀁룹슑医烁킁‑肠䡀x䆂ထ쏐삡揩䋐쏩ᦂ웻጑㢢℉ꔋ?䏉䫐ぃ㫧렀褵ₒ쌦定꧐쉀≀Xﮢ易쇎蝠ꀠ蜈汁∰퇃ᖢ䭣ᤈꃀƘꏲ䔑턗⌑囥뎸ń틔ࠞ숆쏁㔘蓅죑癰琮償᎚ⱀ⢟┏֢⇰酘澌أĀ㰞灞㶍棆䭨곘ᄽ埋Ņऒ㠍ૢ

  • Decompressed: 447 Bytes

  • Compressed: 160 Bytes

  • Reduction: 64%

Decompressing UTF-8 Strings and Objects

Here is an example of how to take the compressed string we created in the example above and decompress it.

// Require LZString Helper
var LZString = require('*/cartridge/scripts/helpers/lzStringHelper');

// Decompress Session Object
var obj = LZString.decompress(session.custom.myCompressedObject);

// Decompress Session String
var str = LZString.decompress(session.custom.myCompressedString);

Working with UTF-16

If you need to work with Base64 characters, use the following ( NOTE: This Base64 Compression is 166% larger than those produced by UTF-8 compress, but can still significantly compress some JSON objects. ):

// Save Object to Session
session.custom.myCompressedObject = LZString.compressToBase64(obj);

// Save String to Session
session.custom.myCompressedString = LZString.compressToBase64(str);

// Decompress Session Object
var obj = LZString.decompressFromBase64(session.custom.myCompressedObject);

// Decompress Session String
var str = LZString.decompressFromBase64(session.custom.myCompressedString);

Working with Base64

If you need to work with Base64 characters, use the following ( NOTE: This Base64 Compression is 166% larger than those produced by UTF-8 compress, but can still significantly compress some JSON objects. ):

// Save Object to Session
session.custom.myCompressedObject = LZString.compressToBase64(obj);

// Save String to Session
session.custom.myCompressedString = LZString.compressToBase64(str);

// Decompress Session Object
var obj = LZString.decompressFromBase64(session.custom.myCompressedObject);

// Decompress Session String
var str = LZString.decompressFromBase64(session.custom.myCompressedString);

Working with Encoded URI Components

If you need to work with Encoded URI characters, use the following ( NOTE: This Encoded URI Compression is similar to Base64 and 166% larger than those produced by UTF-8 compress, but can still significantly compress some JSON objects. ):

// Save Object to Session
session.custom.myCompressedObject = LZString.compressToEncodedURIComponent(obj);

// Save String to Session
session.custom.myCompressedString = LZString.compressToEncodedURIComponent(str);

// Decompress Session Object
var obj = LZString.decompressFromEncodedURIComponent(session.custom.myCompressedObject);

// Decompress Session String
var str = LZString.decompressFromEncodedURIComponent(session.custom.myCompressedString);

Working with Uint8Array

If you need to work with Uint8Array, use the following:

// Save Object to Session
session.custom.myCompressedObject = LZString.compressToUint8Array(obj);

// Save String to Session
session.custom.myCompressedString = LZString.compressToUint8Array(str);

// Decompress Session Object
var obj = LZString.decompressFromUint8Array(session.custom.myCompressedObject);

// Decompress Session String
var str = LZString.decompressFromUint8Array(session.custom.myCompressedString);

lzStringHelper.js

Here is the source code for cartridge/scripts/helpers/lzStringHelper.js

/**
 * LZ-based Compression Algorithm
 * This is a modified version of the following URL for use within SFCC
 * @see https://github.com/pieroxy/lz-string/blob/master/libs/lz-string.js
 */
module.exports = (function() {
    var f = String.fromCharCode;
    var keyStrBase64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=';
    var keyStrUriSafe = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-$';
    var baseReverseDic = {};

    /**
     * Core Compress Function
     *
     * @param {Mixed} uncompressed String or Object to Compress
     * @param {Integer} bitsPerChar Bits per Character
     * @param {Integer} getCharFromInt Get Character from Integer
     * @returns {String}
     */
    function compress(uncompressed, bitsPerChar, getCharFromInt) {
        if (uncompressed == null) return '';
        if (typeof uncompressed !== 'string') {
            try {
                uncompressed = JSON.stringify(uncompressed)
            } catch (err) {
                return err.name + ': ' + err.message;
            }
        }

        var i, value,
            contextDictionary = {},
            contextDictionaryToCreate = {},
            contextC = '',
            contextWC = '',
            contextW = '',
            contextEnlargeIn = 2,
            contextDictSize = 3,
            contextNumBits = 2,
            contextData = [],
            contextDataVal = 0,
            contextDataPosition = 0,
            ii;

        for (ii = 0; ii < uncompressed.length; ii += 1) {
            contextC = uncompressed.charAt(ii);

            if (!Object.prototype.hasOwnProperty.call(contextDictionary, contextC)) {
                contextDictionary[contextC] = contextDictSize++;
                contextDictionaryToCreate[contextC] = true;
            }

            contextWC = contextW + contextC;

            if (Object.prototype.hasOwnProperty.call(contextDictionary, contextWC)) {
                contextW = contextWC;
            } else {
                if (Object.prototype.hasOwnProperty.call(contextDictionaryToCreate, contextW)) {
                    if (contextW.charCodeAt(0) < 256) {
                        for (i = 0; i < contextNumBits; i++) {
                            contextDataVal = (contextDataVal << 1);

                            if (contextDataPosition == bitsPerChar - 1) {
                                contextDataPosition = 0;
                                contextData.push(getCharFromInt(contextDataVal));
                                contextDataVal = 0;
                            } else {
                                contextDataPosition++;
                            }
                        }

                        value = contextW.charCodeAt(0);

                        for (i = 0; i < 8; i++) {
                            contextDataVal = (contextDataVal << 1) | (value & 1);

                            if (contextDataPosition == bitsPerChar - 1) {
                                contextDataPosition = 0;
                                contextData.push(getCharFromInt(contextDataVal));
                                contextDataVal = 0;
                            } else {
                                contextDataPosition++;
                            }

                            value = value >> 1;
                        }
                    } else {
                        value = 1;

                        for (i = 0; i < contextNumBits; i++) {
                            contextDataVal = (contextDataVal << 1) | value;

                            if (contextDataPosition == bitsPerChar - 1) {
                                contextDataPosition = 0;
                                contextData.push(getCharFromInt(contextDataVal));
                                contextDataVal = 0;
                            } else {
                                contextDataPosition++;
                            }

                            value = 0;
                        }

                        value = contextW.charCodeAt(0);

                        for (i = 0; i < 16; i++) {
                            contextDataVal = (contextDataVal << 1) | (value & 1);

                            if (contextDataPosition == bitsPerChar - 1) {
                                contextDataPosition = 0;
                                contextData.push(getCharFromInt(contextDataVal));
                                contextDataVal = 0;
                            } else {
                                contextDataPosition++;
                            }

                            value = value >> 1;
                        }
                    }

                    contextEnlargeIn--;

                    if (contextEnlargeIn == 0) {
                        contextEnlargeIn = Math.pow(2, contextNumBits);
                        contextNumBits++;
                    }

                    delete contextDictionaryToCreate[contextW];
                } else {
                    value = contextDictionary[contextW];

                    for (i = 0; i < contextNumBits; i++) {
                        contextDataVal = (contextDataVal << 1) | (value & 1);

                        if (contextDataPosition == bitsPerChar - 1) {
                            contextDataPosition = 0;
                            contextData.push(getCharFromInt(contextDataVal));
                            contextDataVal = 0;
                        } else {
                            contextDataPosition++;
                        }

                        value = value >> 1;
                    }
                }

                contextEnlargeIn--;

                if (contextEnlargeIn == 0) {
                    contextEnlargeIn = Math.pow(2, contextNumBits);
                    contextNumBits++;
                }

                contextDictionary[contextWC] = contextDictSize++;
                contextW = String(contextC);
            }
        }

        if (contextW !== '') {
            if (Object.prototype.hasOwnProperty.call(contextDictionaryToCreate, contextW)) {
                if (contextW.charCodeAt(0) < 256) {
                    for (i = 0; i < contextNumBits; i++) {
                        contextDataVal = (contextDataVal << 1);

                        if (contextDataPosition == bitsPerChar - 1) {
                            contextDataPosition = 0;
                            contextData.push(getCharFromInt(contextDataVal));
                            contextDataVal = 0;
                        } else {
                            contextDataPosition++;
                        }
                    }

                    value = contextW.charCodeAt(0);

                    for (i = 0; i < 8; i++) {
                        contextDataVal = (contextDataVal << 1) | (value & 1);

                        if (contextDataPosition == bitsPerChar - 1) {
                            contextDataPosition = 0;
                            contextData.push(getCharFromInt(contextDataVal));
                            contextDataVal = 0;
                        } else {
                            contextDataPosition++;
                        }

                        value = value >> 1;
                    }
                } else {
                    value = 1;

                    for (i = 0; i < contextNumBits; i++) {
                        contextDataVal = (contextDataVal << 1) | value;

                        if (contextDataPosition == bitsPerChar - 1) {
                            contextDataPosition = 0;
                            contextData.push(getCharFromInt(contextDataVal));
                            contextDataVal = 0;
                        } else {
                            contextDataPosition++;
                        }

                        value = 0;
                    }

                    value = contextW.charCodeAt(0);

                    for (i = 0; i < 16; i++) {
                        contextDataVal = (contextDataVal << 1) | (value & 1);

                        if (contextDataPosition == bitsPerChar - 1) {
                            contextDataPosition = 0;
                            contextData.push(getCharFromInt(contextDataVal));
                            contextDataVal = 0;
                        } else {
                            contextDataPosition++;
                        }

                        value = value >> 1;
                    }
                }

                contextEnlargeIn--;

                if (contextEnlargeIn == 0) {
                    contextEnlargeIn = Math.pow(2, contextNumBits);
                    contextNumBits++;
                }

                delete contextDictionaryToCreate[contextW];
            } else {
                value = contextDictionary[contextW];

                for (i = 0; i < contextNumBits; i++) {
                    contextDataVal = (contextDataVal << 1) | (value & 1);

                    if (contextDataPosition == bitsPerChar - 1) {
                        contextDataPosition = 0;
                        contextData.push(getCharFromInt(contextDataVal));
                        contextDataVal = 0;
                    } else {
                        contextDataPosition++;
                    }

                    value = value >> 1;
                }
            }

            contextEnlargeIn--;

            if (contextEnlargeIn == 0) {
                contextEnlargeIn = Math.pow(2, contextNumBits);
                contextNumBits++;
            }
        }

        value = 2;

        for (i = 0; i < contextNumBits; i++) {
            contextDataVal = (contextDataVal << 1) | (value & 1);

            if (contextDataPosition == bitsPerChar - 1) {
                contextDataPosition = 0;
                contextData.push(getCharFromInt(contextDataVal));
                contextDataVal = 0;
            } else {
                contextDataPosition++;
            }

            value = value >> 1;
        }

        while (true) {
            contextDataVal = (contextDataVal << 1);

            if (contextDataPosition == bitsPerChar - 1) {
                contextData.push(getCharFromInt(contextDataVal));
                break;
            } else contextDataPosition++;
        }

        return contextData.join('');
    }

    /**
     * Core Decompress Function
     *
     * @param {Integer} length String length
     * @param {Integer} resetValue Reset Value
     * @param {Function} getNextValue Callback Handler
     * @returns {Mixed} Uncompressed String or Object
     */
    function decompress(length, resetValue, getNextValue) {
        var dictionary = [],
            next,
            enlargeIn = 4,
            dictSize = 4,
            numBits = 3,
            entry = '',
            result = [],
            i,
            w,
            bits, resB, maxPower, power,
            c,
            data = {
                val: getNextValue(0),
                position: resetValue,
                index: 1
            };

        for (i = 0; i < 3; i += 1) {
            dictionary[i] = i;
        }

        bits = 0;
        maxPower = Math.pow(2, 2);
        power = 1;

        while (power != maxPower) {
            resB = data.val & data.position;
            data.position >>= 1;

            if (data.position == 0) {
                data.position = resetValue;
                data.val = getNextValue(data.index++);
            }

            bits |= (resB > 0 ? 1 : 0) * power;
            power <<= 1;
        }

        switch (next = bits) {
            case 0:
                bits = 0;
                maxPower = Math.pow(2, 8);
                power = 1;

                while (power != maxPower) {
                    resB = data.val & data.position;
                    data.position >>= 1;

                    if (data.position == 0) {
                        data.position = resetValue;
                        data.val = getNextValue(data.index++);
                    }

                    bits |= (resB > 0 ? 1 : 0) * power;
                    power <<= 1;
                }

                c = f(bits);
                break;
            case 1:
                bits = 0;
                maxPower = Math.pow(2, 16);
                power = 1;

                while (power != maxPower) {
                    resB = data.val & data.position;
                    data.position >>= 1;

                    if (data.position == 0) {
                        data.position = resetValue;
                        data.val = getNextValue(data.index++);
                    }

                    bits |= (resB > 0 ? 1 : 0) * power;
                    power <<= 1;
                }

                c = f(bits);
                break;
            case 2:
                return '';
        }

        dictionary[3] = c;
        w = c;
        result.push(c);

        while (true) {
            if (data.index > length) {
                return '';
            }

            bits = 0;
            maxPower = Math.pow(2, numBits);
            power = 1;

            while (power != maxPower) {
                resB = data.val & data.position;
                data.position >>= 1;

                if (data.position == 0) {
                    data.position = resetValue;
                    data.val = getNextValue(data.index++);
                }

                bits |= (resB > 0 ? 1 : 0) * power;
                power <<= 1;
            }

            switch (c = bits) {
                case 0:
                    bits = 0;
                    maxPower = Math.pow(2, 8);
                    power = 1;

                    while (power != maxPower) {
                        resB = data.val & data.position;
                        data.position >>= 1;

                        if (data.position == 0) {
                            data.position = resetValue;
                            data.val = getNextValue(data.index++);
                        }

                        bits |= (resB > 0 ? 1 : 0) * power;
                        power <<= 1;
                    }

                    dictionary[dictSize++] = f(bits);
                    c = dictSize - 1;
                    enlargeIn--;
                    break;
                case 1:
                    bits = 0;
                    maxPower = Math.pow(2, 16);
                    power = 1;

                    while (power != maxPower) {
                        resB = data.val & data.position;
                        data.position >>= 1;

                        if (data.position == 0) {
                            data.position = resetValue;
                            data.val = getNextValue(data.index++);
                        }

                        bits |= (resB > 0 ? 1 : 0) * power;
                        power <<= 1;
                    }

                    dictionary[dictSize++] = f(bits);
                    c = dictSize - 1;
                    enlargeIn--;
                    break;
                case 2:
                    var output = result.join('');

                    try {
                        return JSON.parse(output)
                    } catch (err) {
                        return output
                    }
            }

            if (enlargeIn == 0) {
                enlargeIn = Math.pow(2, numBits);
                numBits++;
            }

            if (dictionary[c]) {
                entry = dictionary[c];
            } else {
                if (c === dictSize) {
                    entry = w + w.charAt(0);
                } else {
                    return null;
                }
            }

            result.push(entry);

            dictionary[dictSize++] = w + entry.charAt(0);
            enlargeIn--;

            w = entry;

            if (enlargeIn == 0) {
                enlargeIn = Math.pow(2, numBits);
                numBits++;
            }
        }
    }

    /**
     * Get Base Dictionary Value
     *
     * @param {String} alphabet Alphabet to use for Compression
     * @param {String} character Current Character
     * @returns {String} Dictionary Character
     */
    function getBaseValue(alphabet, character) {
        if (!baseReverseDic[alphabet]) {
            baseReverseDic[alphabet] = {};

            for (var i = 0; i < alphabet.length; i++) {
                baseReverseDic[alphabet][alphabet.charAt(i)] = i;
            }
        }

        return baseReverseDic[alphabet][character];
    }

    return {
        /**
         * Compress UTF-8 Strings
         * Can be decompressed with `decompress`
         * @param {Mixed} uncompressed String or Object to Compress
         * @returns {String}
         */
        compress: function(uncompressed) {
            return compress(uncompressed, 16, function(a) {
                return f(a);
            });
        },

        /**
         * Compress to Base64
         *
         * Produces ASCII UTF-16 strings representing the original string encoded in Base64.
         * Can be decompressed with `decompressFromBase64`
         * This works by using only 6bits of storage per character.
         * The strings produced are therefore 166% bigger than those produced by compress.
         * It can still reduce significantly some JSON compressed objects.
         *
         * @param {Mixed} uncompressed String or Object to Compress
         * @returns {String}
         */
        compressToBase64: function(uncompressed) {
            if (uncompressed == null) return '';

            var res = compress(uncompressed, 6, function(a) {
                return keyStrBase64.charAt(a);
            });

            switch (res.length % 4) {
                default:
                case 0:
                    return res;
                case 1:
                    return res + '===';
                case 2:
                    return res + '==';
                case 3:
                    return res + '=';
            }
        },

        /**
         * Compress to Encoded URI Component
         *
         * Produces ASCII strings representing the original string encoded in Base64 with a few tweaks to make these URI safe.
         * Hence, you can send them to the server without thinking about URL encoding them. This saves bandwidth and CPU.
         * These strings can be decompressed with `decompressFromEncodedURIComponent`.
         * See the bullet point above for considerations about size.
         *
         * @param {Mixed} uncompressed String or Object to Compress
         * @returns {String}
         */
        compressToEncodedURIComponent: function(uncompressed) {
            if (uncompressed == null) return '';

            return compress(uncompressed, 6, function(a) {
                return keyStrUriSafe.charAt(a);
            });
        },

        /**
         * Compress to Uint8Array
         *
         * Produces an uint8Array. Can be decompressed with `decompressFromUint8Array`
         * @param {Mixed} uncompressed String or Object to Compress
         * @returns {Uint8Array}
         */
        compressToUint8Array: function(uncompressed) {
            var compressed = compress(uncompressed);
            var buf = new Uint8Array(compressed.length * 2);

            for (var i = 0, TotalLen = compressed.length; i < TotalLen; i++) {
                var currentValue = compressed.charCodeAt(i);

                buf[i * 2] = currentValue >>> 8;
                buf[i * 2 + 1] = currentValue % 256;
            }

            return buf;
        },

        /**
         * Compress to UTF16
         *
         * Produces "valid" UTF-16 strings in the sense that all browsers can store them safely.
         * So they can be stored in localStorage on all browsers tested.
         * Can be decompressed with `decompressFromUTF16`
         * This works by using only 15bits of storage per character.
         * The strings produced are therefore 6.66% bigger than those produced by `compress`
         *
         * @param {*} uncompressed String or Object to Compress
         * @returns {String}
         */
        compressToUTF16: function(uncompressed) {
            if (uncompressed == null) return '';

            return compress(uncompressed, 15, function(a) {
                return f(a + 32);
            }) + ' ';
        },

        /**
         * Decompress output generated from `compress`
         *
         * @param {String} compressed Compress String
         * @returns {Mixed} Decompressed String or Object
         */
        decompress: function(compressed) {
            if (compressed == null) return '';
            if (compressed == '') return null;

            return decompress(compressed.length, 32768, function(index) {
                return compressed.charCodeAt(index);
            });
        },

        /**
         * Decompress output generated from `compressToBase64`
         *
         * @param {String} compressed Compress String
         * @returns {Mixed} Decompressed String or Object
         */
        decompressFromBase64: function(compressed) {
            if (compressed == null) return '';
            if (compressed == '') return null;

            return decompress(compressed.length, 32, function(index) {
                return getBaseValue(keyStrBase64, compressed.charAt(index));
            });
        },

        /**
         * Decompress output generated from `compressToEncodedURIComponent`
         *
         * @param {String} compressed Compress String
         * @returns {Mixed} Decompressed String or Object
         */
        decompressFromEncodedURIComponent: function(compressed) {
            if (compressed == null) return '';
            if (compressed == '') return null;

            compressed = compressed.replace(/ /g, '+');

            return decompress(compressed.length, 32, function(index) {
                return getBaseValue(keyStrUriSafe, compressed.charAt(index));
            });
        },

        /**
         * Decompress output generated from `compressToUint8Array`
         *
         * @param {String} compressed Compress String
         * @returns {Mixed} Decompressed String or Object
         */
        decompressFromUint8Array: function(compressed) {
            if (compressed === null || compressed === undefined) {
                return decompress(compressed);
            } else {
                var buf = new Array(compressed.length / 2);

                for (var i = 0, TotalLen = buf.length; i < TotalLen; i++) {
                    buf[i] = compressed[i * 2] * 256 + compressed[i * 2 + 1];
                }

                var result = [];
                buf.forEach(function(c) {
                    result.push(f(c));
                });

                return decompress(result.join(''));
            }
        },

        /**
         * Decompress output generated from `compressToUTF16`
         *
         * @param {String} compressed Compress String
         * @returns {Mixed} Decompressed String or Object
         */
        decompressFromUTF16: function(compressed) {
            if (compressed == null) return '';
            if (compressed == '') return null;

            return decompress(compressed.length, 16384, function(index) {
                return compressed.charCodeAt(index) - 32;
            });
        }
    };
})();
0
Subscribe to my newsletter

Read articles from Peter Schmalfeldt directly inside your inbox. Subscribe to the newsletter, and don't miss out.

compressioncommerce cloudsession storageSalesforceDevelopment Tools

Written by

Peter Schmalfeldt
Peter Schmalfeldt

Senior Digital Engineer at Patagonia & Founder of SFCC DevOps.