#151 get train log with 404, No such file or directory

How to collect the job logs?

The logs will refer to these services：filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

The logs will refer to these services：filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

Yes, there is no log in logger path.

The logs will refer to these services：filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

As you said logstash receive logs, which one send the log? Does the task container need the log path.

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The log path is exists, but no content. If I create the log file by touch, it can be visited.

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The task container need filebeat?

My task is very simple, the job will be finished very quickly.

The filebeat container's log is

2021-11-05T08:35:51.033Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40919242}}},"cpu":{"system":{"ticks":265700,"time":{"ms":8}},"total":{"ticks":566570,"time":{"ms":39},"value":566570},"user":{"ticks":300870,"time":{"ms":31}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"6d73fe58-e5e3-4e52-99bb-a15963d45dc4","uptime":{"ms":680010088}},"memstats":{"gc_next":27428128,"memory_alloc":14051656,"memory_total":12347316024,"rss":125775872},"runtime":{"goroutines":55}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.2,"5":0.18,"norm":{"1":0.0125,"15":0.025,"5":0.0225}}}}}}

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The task container need filebeat?

No,each node will have a filebeat instance to collect logs of all containers on this node.
How do you output the log?

Can refer to here,the log needs to be output to stdout and stderr in container.

Can refer to here,the log needs to be output to stdout and stderr in container.

ok

Can refer to here,the log needs to be output to stdout and stderr in container.

kubectl logs has data, but no log file in logger path.

kubectl logs -f yc6e593b92ea46fd9ee188afd091cf7d-task0-0 -n d37c68f328b94e8da56bde693916a55e

09:10:15.094 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 05, 2021 9:10:15 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
09:10:15.271 INFO  PrintReads - ------------------------------------------------------------
09:10:15.272 INFO  PrintReads - The Genome Analysis Toolkit (GATK) v4.2.2.0
09:10:15.272 INFO  PrintReads - For support and documentation go to https://software.broadinstitute.org/gatk/
09:10:15.272 INFO  PrintReads - Executing as root@yc6e593b92ea46fd9ee188afd091cf7d-task0-0.yc6e593b92ea46fd9ee188afd091cf7d.d37c68f328b94e8da56bde693916a55e.svc.cluster.local on Linux v3.10.0-1160.el7.x86_64 amd64

The metrics has data.

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

Can you provide this configuration value?

Or you can provide information about the your job, such as image, startup commands, etc.
We can view it in our environment to restore the task

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

Where is filebeat.yml?

It can work now.

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!

The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.