The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?
The logs will refer to these services:`filebeat`,`httpd`,`logstash`.
service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files.
Are these services running normally?
The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?
Yes, there is no log in logger path.
> The logs will refer to these services:`filebeat`,`httpd`,`logstash`.
> service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files.
> Are these services running normally?
Yes, there is no log in logger path.
The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?
As you said logstash receive logs, which one send the log? Does the task container need the log path.
> The logs will refer to these services:`filebeat`,`httpd`,`logstash`.
> service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files.
> Are these services running normally?
As you said `logstash` receive logs, which one send the log? Does the task container need the log path.
filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?
The log path is exists, but no content. If I create the log file by touch, it can be visited.
> `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?`
The log path is exists, but no content. If I create the log file by touch, it can be visited.
My task is very simple, the job will be finished very quickly.
The filebeat container's log is
2021-11-05T08:35:51.033Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40919242}}},"cpu":{"system":{"ticks":265700,"time":{"ms":8}},"total":{"ticks":566570,"time":{"ms":39},"value":566570},"user":{"ticks":300870,"time":{"ms":31}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"6d73fe58-e5e3-4e52-99bb-a15963d45dc4","uptime":{"ms":680010088}},"memstats":{"gc_next":27428128,"memory_alloc":14051656,"memory_total":12347316024,"rss":125775872},"runtime":{"goroutines":55}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.2,"5":0.18,"norm":{"1":0.0125,"15":0.025,"5":0.0225}}}}}}
My task is very simple, the job will be finished very quickly.
The filebeat container's log is
```
2021-11-05T08:35:51.033Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40919242}}},"cpu":{"system":{"ticks":265700,"time":{"ms":8}},"total":{"ticks":566570,"time":{"ms":39},"value":566570},"user":{"ticks":300870,"time":{"ms":31}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"6d73fe58-e5e3-4e52-99bb-a15963d45dc4","uptime":{"ms":680010088}},"memstats":{"gc_next":27428128,"memory_alloc":14051656,"memory_total":12347316024,"rss":125775872},"runtime":{"goroutines":55}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.2,"5":0.18,"norm":{"1":0.0125,"15":0.025,"5":0.0225}}}}}}
```
filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?
The task container need filebeat?
No,each node will have a filebeat instance to collect logs of all containers on this node.
How do you output the log?
> > `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?`
>
> The task container need filebeat?
No,each node will have a filebeat instance to collect logs of all containers on this node.
How do you output the log?
Can refer to here,the log needs to be output to stdout and stderr in container.
Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container.
Can refer to here,the log needs to be output to stdout and stderr in container.
ok
> Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container.
ok
09:10:15.094 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 05, 2021 9:10:15 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
09:10:15.271 INFO PrintReads - ------------------------------------------------------------
09:10:15.272 INFO PrintReads - The Genome Analysis Toolkit (GATK) v4.2.2.0
09:10:15.272 INFO PrintReads - For support and documentation go to https://software.broadinstitute.org/gatk/
09:10:15.272 INFO PrintReads - Executing as root@yc6e593b92ea46fd9ee188afd091cf7d-task0-0.yc6e593b92ea46fd9ee188afd091cf7d.d37c68f328b94e8da56bde693916a55e.svc.cluster.local on Linux v3.10.0-1160.el7.x86_64 amd64
The metrics has data.
> Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container.
kubectl logs has data, but no log file in logger path.
kubectl logs -f yc6e593b92ea46fd9ee188afd091cf7d-task0-0 -n d37c68f328b94e8da56bde693916a55e
```
09:10:15.094 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 05, 2021 9:10:15 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
09:10:15.271 INFO PrintReads - ------------------------------------------------------------
09:10:15.272 INFO PrintReads - The Genome Analysis Toolkit (GATK) v4.2.2.0
09:10:15.272 INFO PrintReads - For support and documentation go to https://software.broadinstitute.org/gatk/
09:10:15.272 INFO PrintReads - Executing as root@yc6e593b92ea46fd9ee188afd091cf7d-task0-0.yc6e593b92ea46fd9ee188afd091cf7d.d37c68f328b94e8da56bde693916a55e.svc.cluster.local on Linux v3.10.0-1160.el7.x86_64 amd64
```
The metrics has data.

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
> I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.
> > I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
>
> Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.
release-v4.0.0
get train log with 404, No such file or directory
How to collect the job logs?
The logs will refer to these services:
filebeat
,httpd
,logstash
.service
logstash
will receive logs and write them to disk, servicehttpd
provides http service to access log files.Are these services running normally?
Yes, there is no log in logger path.
As you said
logstash
receive logs, which one send the log? Does the task container need the log path.filebeat
send the log,What is the value of the configurationpv.logger
in yourvalues.yaml?
The log path is exists, but no content. If I create the log file by touch, it can be visited.
The task container need filebeat?
My task is very simple, the job will be finished very quickly.
The filebeat container's log is
No,each node will have a filebeat instance to collect logs of all containers on this node.
How do you output the log?
Can refer to here,the log needs to be output to stdout and stderr in container.
ok
kubectl logs has data, but no log file in logger path.
kubectl logs -f yc6e593b92ea46fd9ee188afd091cf7d-task0-0 -n d37c68f328b94e8da56bde693916a55e
The metrics has data.
Can you provide this configuration value?
Or you can provide information about the your job, such as image, startup commands, etc.
We can view it in our environment to restore the task
I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
Where is filebeat.yml?
It can work now.
Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.