503 Service Unavailable - maximum request handling time exceeded after reload frankenphp

[kitro]

What happened?

Hi All!

After each deploy we encounter that some requests stagnated and throw 503. return the error maximum request handling time exceeded.

This issue happen only when the deployment process send service frankenphp reload. and the request get in the same window time.

We encountered before the issue of stale symlink in the deployment process in #2215 was resolved by using the force reload for frankenphp to get the new symlink to achieve the zero-downtime ( but I think is not related issue, just as info here because the issue happen when we introduce -force flag)

This a reproducible repo https://github.com/kitro/simlink

The reproducible repo is in docker. But the original issue encounter in deb package. I tested both v1.12.4 and v1.12.3.

Thanks!

Build Type

deb packages

Worker Mode

No

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

phpinfo() output

Configuration

apcu

APCu Support => Disabled
APCu Debugging => Disabled
MMAP Support => Enabled
MMAP File Mask =>
Serialization Support => Disabled

bcmath

BCMath support => enabled

Directive => Local Value => Master Value
bcmath.scale => 0 => 0

bz2

BZip2 Support => Enabled

calendar

Calendar support => enabled

Core

PHP Version => 8.4.18

Directive => Local Value => Master Value
allow_url_fopen => On => On
allow_url_include => Off => Off
arg_separator.input => & => &
arg_separator.output => & => &
auto_append_file => no value => no value
auto_globals_jit => On => On
auto_prepend_file => no value => no value
browscap => no value => no value
default_charset => UTF-8 => UTF-8
default_mimetype => text/html => text/html
disable_classes => no value => no value
disable_functions => no value => no value
display_errors => Off => Off
display_startup_errors => Off => Off
doc_root => no value => no value
docref_ext => no value => no value
docref_root => no value => no value
enable_dl => Off => Off
enable_post_data_reading => On => On
error_append_string => no value => no value
error_log => no value => no value
error_prepend_string => no value => no value
expose_php => On => On
extension_dir => /usr/lib/php-zts/modules => /usr/lib/php-zts/modules
fiber.stack_size => no value => no value
file_uploads => On => On
hard_timeout => 2 => 2
highlight.comment => #FF8000 => #FF8000
highlight.default => #0000BB => #0000BB
highlight.html => #000000 => #000000
highlight.keyword => #007700 => #007700
highlight.string => #DD0000 => #DD0000
html_errors => Off => Off
ignore_repeated_errors => Off => Off
ignore_repeated_source => Off => Off
ignore_user_abort => Off => Off
implicit_flush => On => On
include_path => .: => .:
input_encoding => no value => no value
internal_encoding => no value => no value
log_errors => On => On
mail.add_x_header => Off => Off
mail.force_extra_parameters => no value => no value
mail.log => no value => no value
mail.mixed_lf_and_crlf => Off => Off
max_execution_time => 0 => 0
max_file_uploads => 20 => 20
max_input_nesting_level => 64 => 64
max_input_time => -1 => -1
max_input_vars => 1000 => 1000
max_multipart_body_parts => -1 => -1
memory_limit => 256M => 256M
open_basedir => no value => no value
output_buffering => 0 => 0
output_encoding => no value => no value
output_handler => no value => no value
post_max_size => 8M => 8M
precision => 14 => 14
realpath_cache_size => 4096K => 4096K
realpath_cache_ttl => 120 => 120
register_argc_argv => On => On
report_memleaks => On => On
report_zend_debug => Off => Off
request_order => GP => GP
sendmail_from => no value => no value
serialize_precision => -1 => -1
short_open_tag => Off => Off
static-php-cli.version => 2.8.0 => 2.8.0
sys_temp_dir => no value => no value
syslog.facility => LOG_USER => LOG_USER
syslog.filter => no-ctrl => no-ctrl
syslog.ident => php => php
unserialize_callback_func => no value => no value
upload_max_filesize => 2M => 2M
upload_tmp_dir => no value => no value
user_dir => no value => no value
user_ini.cache_ttl => 300 => 300
user_ini.filename => .user.ini => .user.ini
variables_order => GPCS => GPCS
xmlrpc_error_number => 0 => 0
xmlrpc_errors => Off => Off
zend.assertions => -1 => -1
zend.detect_unicode => On => On
zend.enable_gc => On => On
zend.exception_ignore_args => On => On
zend.exception_string_param_max_len => 0 => 0
zend.max_allowed_stack_size => 0 => 0
zend.multibyte => Off => Off
zend.reserved_stack_size => 0 => 0
zend.script_encoding => no value => no value

ctype

ctype functions => enabled

curl

cURL support => enabled

date

date/time support => enabled

dom

DOM/XML => enabled

exif

EXIF Support => enabled

fileinfo

fileinfo support => enabled

filter

Input Validation and Filtering => enabled

gd

GD Support => enabled

gmp

gmp support => enabled

hash

hash support => enabled

iconv

iconv support => enabled

igbinary

igbinary support => enabled

imagick

imagick module => enabled

intl

Internationalization support => enabled

json

json support => enabled

libxml

libXML support => active

mbstring

Multibyte Support => enabled

msgpack

MessagePack Support => enabled

mysqli

MysqlI Support => enabled

mysqlnd

mysqlnd => enabled

openssl

OpenSSL support => enabled

pcntl

pcntl support => enabled

pcre

PCRE (Perl Compatible Regular Expressions) Support => enabled

PDO

PDO support => enabled

pdo_mysql

PDO Driver for MySQL => enabled

Phar

Phar: PHP Archive support => enabled

posix

POSIX support => enabled

readline

Readline Support => enabled

redis

Redis Support => enabled

Reflection

Reflection => enabled

session

Session Support => enabled

SimpleXML

SimpleXML support => enabled
Schema support => enabled

sockets

Sockets Support => enabled

sodium

sodium support => enabled

SPL

SPL support => enabled

sqlite3

SQLite3 support => enabled

standard

Dynamic Library Support => enabled

tokenizer

Tokenizer Support => enabled

xml

XML Support => active

xmlreader

XMLReader => enabled

xmlwriter

XMLWriter => enabled

Zend OPcache

Opcode Caching => Up and Running
Optimization => Enabled
SHM Cache => Enabled
File Cache => Disabled
JIT => Disabled
Startup => OK
Shared memory model => mmap
Cache hits => 0
Cache misses => 0
Wasted memory => 0
Cached scripts => 0
Cached keys => 0
Max keys => 16229
OOM restarts => 0
Hash keys restarts => 0
Manual restarts => 0
Last restart time => none
Last force restart time => none

Directive => Local Value => Master Value
opcache.blacklist_filename => no value => no value
opcache.dups_fix => Off => Off
opcache.enable => On => On
opcache.enable_cli => On => On
opcache.enable_file_override => Off => Off
opcache.error_log => no value => no value
opcache.file_cache => no value => no value
opcache.file_cache_consistency_checks => On => On
opcache.file_cache_only => Off => Off
opcache.file_update_protection => 2 => 2
opcache.force_restart_timeout => 180 => 180
opcache.huge_code_pages => Off => Off
opcache.interned_strings_buffer => 8 => 8
opcache.jit => disable => disable
opcache.jit_bisect_limit => 0 => 0
opcache.jit_blacklist_root_trace => 16 => 16
opcache.jit_blacklist_side_trace => 8 => 8
opcache.jit_buffer_size => 64M => 64M
opcache.jit_debug => 0 => 0
opcache.jit_hot_func => 127 => 127
opcache.jit_hot_loop => 64 => 64
opcache.jit_hot_return => 8 => 8
opcache.jit_hot_side_exit => 8 => 8
opcache.jit_max_exit_counters => 8192 => 8192
opcache.jit_max_loop_unrolls => 8 => 8
opcache.jit_max_polymorphic_calls => 2 => 2
opcache.jit_max_recursive_calls => 2 => 2
opcache.jit_max_recursive_returns => 2 => 2
opcache.jit_max_root_traces => 1024 => 1024
opcache.jit_max_side_traces => 128 => 128
opcache.jit_max_trace_length => 1024 => 1024
opcache.jit_prof_threshold => 0.005 => 0.005
opcache.lockfile_path => /tmp => /tmp
opcache.log_verbosity_level => 1 => 1
opcache.max_accelerated_files => 10000 => 10000
opcache.max_file_size => 0 => 0
opcache.max_wasted_percentage => 5 => 5
opcache.memory_consumption => 128 => 128
opcache.opt_debug_level => 0 => 0
opcache.optimization_level => 0x7FFEBFFF => 0x7FFEBFFF
opcache.preferred_memory_model => no value => no value
opcache.preload => no value => no value
opcache.preload_user => no value => no value
opcache.protect_memory => Off => Off
opcache.record_warnings => Off => Off
opcache.restrict_api => no value => no value
opcache.revalidate_freq => 2 => 2
opcache.revalidate_path => Off => Off
opcache.save_comments => On => On
opcache.use_cwd => On => On
opcache.validate_permission => Off => Off
opcache.validate_root => Off => Off
opcache.validate_timestamps => On => On

zip

Zip => enabled

zlib

ZLib Support => enabled

Additional Modules

Module Name

Environment

PHP Variables

PHP License
This program is free software; you can redistribute it and/or modify
it under the terms of the PHP License as published by the PHP Group
and included in the distribution in the file: LICENSE

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you did not receive a copy of the PHP license, or have any
questions about PHP licensing, please contact license@php.net.

Relevant log output

Relevant log output

[henderkes]

Can reproduce, we probably introduced a logic error in our reload logic. I'll fix it soon.

[Huaian666]

performance surprised me tbh. expected it to be slower than what we were using but it's noticeably faster in our benchmarks.