Unable to run cattr on Kubernetes

[khalid244]

I have been trying to run cattr on Kubernetes for several days, but I keep encountering the following error message:

/package/admin/s6-overlay/libexec/preinit: notice: /var/run is not a symlink to /run, fixing it
s6-rmrf: fatal: unable to remove /var/run: Read-only file system
s6-overlay-suexec: fatal: child failed with exit code 111

Has anyone else encountered this issue while running cattr on Kubernetes? If so, how did you resolve it? Any help would be greatly appreciated.

[vermorag]

It seems, that you're trying to run your pod in read-only mode. It is not an option with Cattr.
Or you should set up write-allowed folders for storage. At least: /app/storage & /app/bootstrap/cache.
Also if you insist on the read-only mode you should read the notes: https://github.com/just-containers/s6-overlay#read-only-root-filesystem
Basically, Cattr container isn't ready for Kubernetes as it is.

[khalid244]

Hi Vermorag

Thank you for your response. After I dig more in the problem I found that Kubernetes make /run read-only so that's why it is not working here. Also, anything in /app/* have write-allowed. The main problem with s6-overlay. Is there are any way to run Cattr without s6-overlay ?

I have been trying many things to make s6-overlay work but nothing work even when I make read-only mode.

[khalid244]

I also open issue in s6-overlay. But I don't found any good solution
just-containers/s6-overlay#551

[vermorag]

Read the note about s6 I've attached above. It can run with read-only root when ENV variable set + add smth like:

...
<pod description in deployment>
volumeMounts:
  - mountPath: /run
     name: app-run
...
<volumes description in deployment>
      volumes:
        - name: app-run
          emptyDir: {}

Currently Cattr don't run without s6-overlay, but you can ask about it @DanilShapilov

[khalid244]

I already try that using simple deployment, but it does not work

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cattr-app-deployment
  namespace: cattr
  labels:
    app: cattr
spec:
  selector:
    matchLabels:
      app: cattr
  template:
    metadata:
      labels:
        app: cattr
    spec:
      containers:
        - name: app
          image: registry.gyoom.sa/cattr:2023-10-15
          env:
            - name: "S6_READ_ONLY_ROOT"
              value: "1"
          ports:
            - containerPort: 8000
          volumeMounts:
            - mountPath: /run
              name: app-run
          resources:
            limits:
              cpu: 3000m
              hugepages-2Mi: 1Gi
              memory: "8Gi"
            requests:
              cpu: 100m
              memory: "200Mi"
        - name: db
          image: mysql:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_DATABASE
              value: cattr
            - name: MYSQL_ROOT_PASSWORD
              value: password
          resources:
            limits:
              cpu: 2000m
              memory: "2Gi"
            requests:
              cpu: 100m
              memory: "200Mi"
      volumes:
        - name: app-run
          emptyDir: {}

[vermorag]

For the sake of the experiment try:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cattr-app-deployment
  namespace: cattr
  labels:
    app: cattr
spec:
  selector:
    matchLabels:
      app: cattr
  template:
    metadata:
      labels:
        app: cattr
    spec:
      containers:
        - name: app
          image: registry.gyoom.sa/cattr:2023-10-15
          securityContext:
            readOnlyRootFilesystem: true
          env:
            - name: "S6_READ_ONLY_ROOT"
              value: "1"
          ports:
            - containerPort: 8000
          volumeMounts:
            - mountPath: /run
              name: app-run
            - mountPath: /app/bootstrap/cache
              name: bootstrap-cache
            - mountPath: /tmp
              name: app-tmp
            - mountPath: /app/storage
              name: storage-framework
          resources:
            limits:
              cpu: 3000m
              hugepages-2Mi: 1Gi
              memory: "8Gi"
            requests:
              cpu: 100m
              memory: "200Mi"
        - name: db
          image: mysql:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_DATABASE
              value: cattr
            - name: MYSQL_ROOT_PASSWORD
              value: password
          resources:
            limits:
              cpu: 2000m
              memory: "2Gi"
            requests:
              cpu: 100m
              memory: "200Mi"
      volumes:
        - name: bootstrap-cache
          emptyDir: {}
        - name: app-tmp
          emptyDir: {}
        - name: storage-framework
          emptyDir: {}
        - name: app-run
          emptyDir: {}

I've mixed your config with my config from another project, that also uses s6 & laravel, but I can't test this config rnow.

[khalid244]

/package/admin/s6-overlay/libexec/preinit: info: read-only root
/package/admin/s6-overlay/libexec/preinit: info: writable /run. Checking for executability.
/package/admin/s6-overlay/libexec/preinit: notice: /var/run is not a symlink to /run, fixing it
s6-rmrf: fatal: unable to remove /var/run: Read-only file system
s6-overlay-suexec: fatal: child failed with exit code 111

[vermorag]

Okay. Let's try another option.
I see - you use your registry. Can you build and push an image with that Dockerfile and try to use it?

FROM registry.git.amazingcat.net/cattr/core/app:latest
RUN rm -rf /var/run

Just remove /var/run, maybe it will help.
Another option:

FROM registry.git.amazingcat.net/cattr/core/app:latest
RUN rm -rf /var/run && ln -s /var/run /run

Note: use with my deployment config, because, as you see - s6 found writable run. Only thing left - don't try to remove /var/run

[khalid244]

I test both of them, but I have the same error message. I think Kubernetes reconfigure /var/run every time

[vermorag]

Misspelled params for ln...
Can we try:

FROM registry.git.amazingcat.net/cattr/core/app:latest

RUN rm -rf /var/run && ln -s /run /var/run

[khalid244]

Things start working !!!

/package/admin/s6-overlay/libexec/preinit: info: read-only root
/package/admin/s6-overlay/libexec/preinit: info: writable /run. Checking for executability.
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare-filesystem: starting
s6-rc: info: service prepare-cache: starting
s6-rc: info: service supercronic: starting
s6-rc: info: service supercronic successfully started
mkdir: can't create directory '/var/lib/nginx/tmp': Read-only file system
s6-rc: warning: unable to start service prepare-filesystem: command exited 1
time="2023-10-17T15:40:24Z" level=info msg="read crontab: /crontab"
[2023-10-17 15:40:24] production.ERROR: Please provide a valid cache path. {"exception":"[object] (InvalidArgumentException(code: 0): Please provide a valid cache path. at /app/vendor/laravel/framework/src/Illuminate/View/Compilers/Compiler.php:66)
[stacktrace]
#0 /app/vendor/laravel/framework/src/Illuminate/View/ViewServiceProvider.php(96): Illuminate\\View\\Compilers\\Compiler->__construct(Object(Illuminate\\Filesystem\\Filesystem), false, '', true, 'php')
#1 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(910): Illuminate\\View\\ViewServiceProvider->Illuminate\\View\\{closure}(Object(Illuminate\\Foundation\\Application), Array)
#2 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(795): Illuminate\\Container\\Container->build(Object(Closure))
#3 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(933): Illuminate\\Container\\Container->resolve('blade.compiler', Array, true)
#4 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(731): Illuminate\\Foundation\\Application->resolve('blade.compiler', Array)
#5 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(918): Illuminate\\Container\\Container->make('blade.compiler', Array)
#6 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(1456): Illuminate\\Foundation\\Application->make('blade.compiler')
#7 /app/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(236): Illuminate\\Container\\Container->offsetGet('blade.compiler')
#8 /app/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(207): Illuminate\\Support\\Facades\\Facade::resolveFacadeInstance('blade.compiler')
#9 /app/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(347): Illuminate\\Support\\Facades\\Facade::getFacadeRoot()
#10 /app/vendor/elhebert/laravel-sri/src/SriServiceProvider.php(32): Illuminate\\Support\\Facades\\Facade::__callStatic('component', Array)
#11 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Elhebert\\SubresourceIntegrity\\SriServiceProvider->boot()
#12 /app/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#13 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#14 /app/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#15 /app/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#16 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1006): Illuminate\\Container\\Container->call(Array)
#17 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(987): Illuminate\\Foundation\\Application->bootProvider(Object(Elhebert\\SubresourceIntegrity\\SriServiceProvider))
#18 [internal function]: Illuminate\\Foundation\\Application->Illuminate\\Foundation\\{closure}(Object(Elhebert\\SubresourceIntegrity\\SriServiceProvider), 16)
#19 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(986): array_walk(Array, Object(Closure))
#20 /app/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/BootProviders.php(17): Illuminate\\Foundation\\Application->boot()
#21 /app/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(261): Illuminate\\Foundation\\Bootstrap\\BootProviders->bootstrap(Object(Illuminate\\Foundation\\Application))
#22 /app/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(428): Illuminate\\Foundation\\Application->bootstrapWith(Array)
#23 /app/app/Console/Kernel.php(27): Illuminate\\Foundation\\Console\\Kernel->bootstrap()
#24 /app/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(198): App\\Console\\Kernel->bootstrap()
#25 /app/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#26 {main}
"} 

In Compiler.php line 66:
                                      
  Please provide a valid cache path.  
                                      

s6-rc: warning: unable to start service prepare-cache: command exited 1
time="2023-10-17T15:41:00Z" level=info msg=starting iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info msg="In TelescopeServiceProvider.php line 11:" channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info msg="                                                                           " channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info msg="  Class \"Laravel\\Telescope\\TelescopeApplicationServiceProvider\" not found  " channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info msg="                                                                           " channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=info channel=stdout iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:41:00Z" level=error msg="error running command: exit status 1" iteration=0 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info msg=starting iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info msg="In TelescopeServiceProvider.php line 11:" channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info msg="                                                                           " channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info msg="  Class \"Laravel\\Telescope\\TelescopeApplicationServiceProvider\" not found  " channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info msg="                                                                           " channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=info channel=stdout iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"
time="2023-10-17T15:42:00Z" level=error msg="error running command: exit status 1" iteration=1 job.command="php /app/artisan schedule:run" job.position=0 job.schedule="* * * * *"

[vermorag]

Great! This error happens because /app/storage/framework/views does not exist because we've mounted /app/storage as volume.

It seems my access to the repo hasn't been revoked. Can you try to build and run your image from the source, main branch?

[khalid244]

I test many thing but it seems to be not working when there are a mounted paths like /app/storage. I need to debug more to found out why that's happen for now I run it without them and there are this error appear now

#32 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\TrustProxies->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#33 /app/app/Http/Middleware/SentryContext.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#34 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): App\\Http\\Middleware\\SentryContext->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#35 /app/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(62): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#36 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\HandleCors->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#37 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#38 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#39 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#40 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#41 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#42 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#43 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#44 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#45 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#46 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle(Object(Illuminate\\Http\\Request), Object(Closure))
#47 /app/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}(Object(Illuminate\\Http\\Request))
#48 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(175): Illuminate\\Pipeline\\Pipeline->then(Object(Closure))
#49 /app/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(144): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter(Object(Illuminate\\Http\\Request))
#50 /app/vendor/laravel/octane/src/ApplicationGateway.php(37): Illuminate\\Foundation\\Http\\Kernel->handle(Object(Illuminate\\Http\\Request))
#51 /app/vendor/laravel/octane/src/Worker.php(92): Laravel\\Octane\\ApplicationGateway->handle(Object(Illuminate\\Http\\Request))
#52 /app/vendor/laravel/octane/bin/swoole-server(120): Laravel\\Octane\\Worker->handle(Object(Illuminate\\Http\\Request), Object(Laravel\\Octane\\RequestContext))
#53 [internal function]: {closure}(Object(Swoole\\Http\\Request), Object(Swoole\\Http\\Response))
#54 /app/vendor/laravel/octane/bin/swoole-server(170): Swoole\\Server->start()
#55 {main}
"}

[vermorag]

Provide full error message plz.

[khalid244]

This is full log file
app.log

Also, sometimes it fails in migrations process is there are any way to incenses timeout ? See this
app.log