#151 get train log with 404, No such file or directory

Closed
created 1 year ago by yiluxiangbei · 20 comments

release-v4.0.0

get train log with 404, No such file or directory

10.50.10.19 - - [05/Nov/2021:03:24:46 +0000] "GET /log/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" "10.40.10.37"

2021/11/05 03:24:46 [error] 32#32: *11 open() "/var/www/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log" failed (2: No such file or directory), client: 10.50.10.19, server: , request: "GET /log/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log HTTP/1.1", host: "10.50.10.19"
release-v4.0.0 get train log with 404, No such file or directory ``` 10.50.10.19 - - [05/Nov/2021:03:24:46 +0000] "GET /log/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log HTTP/1.1" 404 555 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36" "10.40.10.37" 2021/11/05 03:24:46 [error] 32#32: *11 open() "/var/www/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log" failed (2: No such file or directory), client: 10.50.10.19, server: , request: "GET /log/user/trainjob/t09aaaaa39b84d4596bafa631c5016df/task0/0/index.log HTTP/1.1", host: "10.50.10.19" ```
yiluxiangbei commented 1 year ago
Poster

How to collect the job logs?

How to collect the job logs?
shamartor commented 1 year ago

The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

The logs will refer to these services:`filebeat`,`httpd`,`logstash`. service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files. Are these services running normally?
shamartor added the
help wanted
label 1 year ago
shamartor added this to the v4.0.5 milestone 1 year ago
shamartor self-assigned this 1 year ago
yiluxiangbei commented 1 year ago
Poster

The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

Yes, there is no log in logger path.

> The logs will refer to these services:`filebeat`,`httpd`,`logstash`. > service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files. > Are these services running normally? Yes, there is no log in logger path.
yiluxiangbei commented 1 year ago
Poster

image

![image](/attachments/4b4e819b-b319-46d6-85ba-0ea979550a58)
yiluxiangbei commented 1 year ago
Poster

The logs will refer to these services:filebeat,httpd,logstash.
service logstash will receive logs and write them to disk, service httpd provides http service to access log files.
Are these services running normally?

As you said logstash receive logs, which one send the log? Does the task container need the log path.

> The logs will refer to these services:`filebeat`,`httpd`,`logstash`. > service `logstash` will receive logs and write them to disk, service `httpd` provides http service to access log files. > Are these services running normally? As you said `logstash` receive logs, which one send the log? Does the task container need the log path.
shamartor commented 1 year ago

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

`filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?`
yiluxiangbei commented 1 year ago
Poster

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The log path is exists, but no content. If I create the log file by touch, it can be visited.

> `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?` The log path is exists, but no content. If I create the log file by touch, it can be visited.
yiluxiangbei commented 1 year ago
Poster

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The task container need filebeat?

> `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?` The task container need filebeat?
yiluxiangbei commented 1 year ago
Poster

My task is very simple, the job will be finished very quickly.

The filebeat container's log is

2021-11-05T08:35:51.033Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40919242}}},"cpu":{"system":{"ticks":265700,"time":{"ms":8}},"total":{"ticks":566570,"time":{"ms":39},"value":566570},"user":{"ticks":300870,"time":{"ms":31}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"6d73fe58-e5e3-4e52-99bb-a15963d45dc4","uptime":{"ms":680010088}},"memstats":{"gc_next":27428128,"memory_alloc":14051656,"memory_total":12347316024,"rss":125775872},"runtime":{"goroutines":55}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.2,"5":0.18,"norm":{"1":0.0125,"15":0.025,"5":0.0225}}}}}}
My task is very simple, the job will be finished very quickly. The filebeat container's log is ``` 2021-11-05T08:35:51.033Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":40919242}}},"cpu":{"system":{"ticks":265700,"time":{"ms":8}},"total":{"ticks":566570,"time":{"ms":39},"value":566570},"user":{"ticks":300870,"time":{"ms":31}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"6d73fe58-e5e3-4e52-99bb-a15963d45dc4","uptime":{"ms":680010088}},"memstats":{"gc_next":27428128,"memory_alloc":14051656,"memory_total":12347316024,"rss":125775872},"runtime":{"goroutines":55}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.2,"5":0.18,"norm":{"1":0.0125,"15":0.025,"5":0.0225}}}}}} ```
shamartor commented 1 year ago

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

The task container need filebeat?

No,each node will have a filebeat instance to collect logs of all containers on this node.
How do you output the log?

> > `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?` > > The task container need filebeat? No,each node will have a filebeat instance to collect logs of all containers on this node. How do you output the log?
shamartor commented 1 year ago

Can refer to here,the log needs to be output to stdout and stderr in container.

Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container.
yiluxiangbei commented 1 year ago
Poster

Can refer to here,the log needs to be output to stdout and stderr in container.

ok

> Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container. ok
yiluxiangbei commented 1 year ago
Poster

Can refer to here,the log needs to be output to stdout and stderr in container.

kubectl logs has data, but no log file in logger path.

kubectl logs -f yc6e593b92ea46fd9ee188afd091cf7d-task0-0 -n d37c68f328b94e8da56bde693916a55e

09:10:15.094 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
Nov 05, 2021 9:10:15 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine
INFO: Failed to detect whether we are running on Google Compute Engine.
09:10:15.271 INFO  PrintReads - ------------------------------------------------------------
09:10:15.272 INFO  PrintReads - The Genome Analysis Toolkit (GATK) v4.2.2.0
09:10:15.272 INFO  PrintReads - For support and documentation go to https://software.broadinstitute.org/gatk/
09:10:15.272 INFO  PrintReads - Executing as root@yc6e593b92ea46fd9ee188afd091cf7d-task0-0.yc6e593b92ea46fd9ee188afd091cf7d.d37c68f328b94e8da56bde693916a55e.svc.cluster.local on Linux v3.10.0-1160.el7.x86_64 amd64

The metrics has data.

image

> Can refer to [here](https://kubernetes.io/docs/concepts/cluster-administration/logging/#logging-at-the-node-level),the log needs to be output to stdout and stderr in container. kubectl logs has data, but no log file in logger path. kubectl logs -f yc6e593b92ea46fd9ee188afd091cf7d-task0-0 -n d37c68f328b94e8da56bde693916a55e ``` 09:10:15.094 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.2.0-local.jar!/com/intel/gkl/native/libgkl_compression.so Nov 05, 2021 9:10:15 AM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine INFO: Failed to detect whether we are running on Google Compute Engine. 09:10:15.271 INFO PrintReads - ------------------------------------------------------------ 09:10:15.272 INFO PrintReads - The Genome Analysis Toolkit (GATK) v4.2.2.0 09:10:15.272 INFO PrintReads - For support and documentation go to https://software.broadinstitute.org/gatk/ 09:10:15.272 INFO PrintReads - Executing as root@yc6e593b92ea46fd9ee188afd091cf7d-task0-0.yc6e593b92ea46fd9ee188afd091cf7d.d37c68f328b94e8da56bde693916a55e.svc.cluster.local on Linux v3.10.0-1160.el7.x86_64 amd64 ``` The metrics has data. ![image](/attachments/82f4438b-4c4b-4664-8728-152feeb06e32)
shamartor commented 1 year ago

filebeat send the log,What is the value of the configuration pv.logger in your values.yaml?

Can you provide this configuration value?

> `filebeat` send the log,What is the value of the configuration `pv.logger` in your `values.yaml?` Can you provide this configuration value?
shamartor commented 1 year ago

Or you can provide information about the your job, such as image, startup commands, etc.
We can view it in our environment to restore the task

Or you can provide information about the your job, such as image, startup commands, etc. We can view it in our environment to restore the task
yiluxiangbei commented 1 year ago
Poster

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker
yiluxiangbei commented 1 year ago
Poster

Where is filebeat.yml?

Where is filebeat.yml?
yiluxiangbei commented 1 year ago
Poster

It can work now.

It can work now.
shamartor commented 1 year ago

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!

> I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!
yiluxiangbei commented 1 year ago
Poster

I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker

Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx!

The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.

> > I think the reason is that the path "/data" is mount from another disk. And the filebeat container has no "/data" path. But the docker path /var/lib/docker links to /data/var/lib/docker > > Was it caused by this problem? How did you solve it in the end? We want to understand clearly so that this project can do better,thx! The container has no path /data, but the container use host path /var/log/containers. I add /data to the container, it can work well.
yangxzh1 closed this issue 1 year ago
yangxzh1 added the
已解决
label 1 year ago
Sign in to join this conversation.
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.