Method for Batch Downloading and Packaging Files from Remote URLs

Background

Recently encountered a problem during code refactoring: needing to batch download and package a set of images from OSS
The old server directly mounted OSS to the filesystem, using Linux commands for copying and packaging (fast but incompatible with the new server architecture)
OSS doesn’t provide direct packaging APIs (Qiniu Cloud supports this)

Solutions Considered

Discarded PHP backend fetching images to memory for packaging (memory overload risk with AI/PSD files)
Discarded PHP backend saving images to disk for packaging (slow fetching and processing for large files)
Implemented Frontend solution: JS fetches file info from backend, asynchronously downloads resources, and packages them client-side

Prerequisites

CORS configuration: Requires adding cross-origin headers on OSS/resource server
CDN bypass: Configure a dedicated domain without CDN acceleration (skip if not using CDN)

Implementation Steps

Use JSZip library for client-side zip operations
Utilize JSZip-utils for handling remote resource fetching
Core implementation logic:

<!-- Include required libraries -->
<script type="text/javascript" src="/xxx/jszip.min.js"></script>
<script type="text/javascript" src="/xxx/jszip-utils.min.js"></script>
<!--[if IE]>
<script type="text/javascript" src="/xxx/jszip-utils-ie.min.js"></script>
<![endif]-->

<script>
let total = 0;
let progress = 0;

// Triggered by button click or other event
$.get('/urls', function(res) {
    // Sample response data structure:
    // {url: 'x.jpg', filename: 'xx.jpg', path: 'xxxx'}
    
    total = res.data.length;
    if (total === 0) {
        console.error('No images available for download in the gallery');
        return;
    }

    let zip = new JSZip();
    let folderMap = {};
    
    res.data.forEach(function(data) {
        JSZipUtils.getBinaryContent(
            data.url,
            function(err, binData) {
                progress++;

                if (err && progress < total) {
                    console.log(err);
                    return;
                }

                if (!folderMap[data.path]) {
                    folderMap[data.path] = zip.folder(data.path);
                }

                let currentFolder = folderMap[data.path];
                currentFolder.file(data.filename, binData, {binary: true});

                if (progress === total) {
                    console.log('Packaging data, please wait...');
                    zip.generateAsync({type: "blob"}).then(function(content) {
                        // Trigger file download
                        const link = document.createElement('a');
                        link.href = URL.createObjectURL(content);
                        link.download = 'archive.zip';
                        document.body.appendChild(link);
                        link.click();
                        document.body.removeChild(link);
                    });
                }
            }
        );
    });
});
</script>

Result

The implemented solution demonstrates better performance than backend-based approaches by leveraging concurrent downloads and client-side processing. Key advantages:

No server-side resource consumption
Parallel downloading improves efficiency
Native browser compression optimizes memory usage