Background
- We previously encountered an issue where the service completely crashed when MySQL went down, with Redis failing to handle any requests.
- The normal business query logic is:
- Get data from Redis first, return if available
- If Redis has no data, query MySQL
- Write the retrieved data to Redis
- Return data
Problem Analysis
- Redis itself wasn’t the issue - the problem occurred at step 2
- When querying MySQL with the database service down, requests get blocked
- Blocking timeout leads to exceptions, preventing step 3 from executing
- Subsequent requests continue trying to connect to MySQL, creating infinite blocking that overloads the application servers
Solution
- Our solution (may not fit all scenarios): Force caching empty data in Redis when MySQL crashes, allowing partial empty responses instead of service failure
Implementation Approach
- Set appropriate MySQL connection timeout
mysqlnd.net_read_timeout = 3
- Throw exceptions on connection timeout
- Create a BaseModel that other models extend, overriding
newEloquentBuilder
<?php
namespace App\Models;
use Illuminate\Database\Eloquent\Model;
class BaseModel extends Model
{
public function newEloquentBuilder($query)
{
return new MysqlCustomBuilder($query);
}
}
- Create a custom query builder
MysqlCustomBuilder
<?php
namespace App\Models\Database;
use Illuminate\Database\Eloquent\Builder;
use Illuminate\Database\Eloquent\Collection;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Str;
class MysqlCustomBuilder extends Builder
{
public function get($columns = ['*'])
{
try {
return parent::get($columns);
} catch (\Exception $e) {
// Handle Laravel's reconnection error codes
$message = $e->getMessage();
if (Str::contains($message, [
'server has gone away',
'no connection to the server',
'Lost connection',
'is dead or not enabled',
'Error while sending',
'decryption failed or bad record mac',
'server closed the connection unexpectedly',
'SSL connection has been closed unexpectedly',
'Error writing data to the connection',
'Resource deadlock avoided',
])) {
// Log error and notify
// Log::error($e);
// Force return empty collection
return Collection::make();
}
throw $e;
}
}
}
- Implement strict monitoring of error logs to distinguish between empty responses caused by configuration issues vs database failures