• The operations team reported recurring OOM (Out-Of-Memory) restarts in a container pod. Monitoring data showed periodic memory spikes:

  

• The service in question handles video composition via queue jobs. It uses shell_exec to invoke commands like ffmpeg and spleeter for audio processing and quality adjustments.

• Since external command execution isn’t trackable through PHP’s memory monitoring, we used atop for detailed logging.

Service Setup

## Installation
apt update && apt install -y atop

## Modify the configuration file
cat /etc/default/atop
LOGOPTS=""
LOGINTERVAL=1        # Set to 1s for granular monitoring
LOGGENERATIONS=28    # Retain logs for 28 days
LOGPATH=/var/log/atop

## Start service
systemctl start atop

Diagnosis Process

• Alibaba Cloud monitoring showed a memory spike at 16:20:30:

  

• Investigate historical records using atop:

## View logs sorted by memory usage
atop -m -r
## Press 'b' and enter timestamp 1620
b
1620
## Navigate with 't' (next page) and 'T' (previous page)

• Identified the culprit process:

  

• The memory-intensive operation was spleeter command execution for audio separation.

• Official documentation suggests splitting audio files before processing to reduce memory consumption:
  https://github.com/deezer/spleeter/wiki/5.-FAQ#why-do-i-have-no-output-produced

Solution

## Split audio into 60s segments using ffmpeg
$inputAudio = 'http://xxx.wav'
ffmpeg -i '{$inputAudio}' -f segment -segment_time 60 -c copy {$downloadDir}/%03d.{$extension}

## Process segments separately
foreach ($files as $file) {
  spleeter separate -o {$outpuDir} -p spleeter:2stems {$file}
}

## Merge processed audio
ffmpeg -f concat -safe 0 -i {$listFile} -c copy {$outputAudioFile}