unet2d
into master
@@ -0,0 +1,189 @@ | |||
# U-Net: Semantic segmentation with PyTorch | |||
<a href="#"><img src="https://img.shields.io/github/actions/workflow/status/milesial/PyTorch-UNet/main.yml?logo=github&style=for-the-badge" /></a> | |||
<a href="https://hub.docker.com/r/milesial/unet"><img src="https://img.shields.io/badge/docker%20image-available-blue?logo=Docker&style=for-the-badge" /></a> | |||
<a href="https://pytorch.org/"><img src="https://img.shields.io/badge/PyTorch-v1.13+-red.svg?logo=PyTorch&style=for-the-badge" /></a> | |||
<a href="#"><img src="https://img.shields.io/badge/python-v3.6+-blue.svg?logo=python&style=for-the-badge" /></a> | |||
![input and output for a random image in the test dataset](https://i.imgur.com/GD8FcB7.png) | |||
Customized implementation of the [U-Net](https://arxiv.org/abs/1505.04597) in PyTorch for Kaggle's [Carvana Image Masking Challenge](https://www.kaggle.com/c/carvana-image-masking-challenge) from high definition images. | |||
- [Quick start](#quick-start) | |||
- [Without Docker](#without-docker) | |||
- [With Docker](#with-docker) | |||
- [Description](#description) | |||
- [Usage](#usage) | |||
- [Docker](#docker) | |||
- [Training](#training) | |||
- [Prediction](#prediction) | |||
- [Weights & Biases](#weights--biases) | |||
- [Pretrained model](#pretrained-model) | |||
- [Data](#data) | |||
## Quick start | |||
### Without Docker | |||
1. [Install CUDA](https://developer.nvidia.com/cuda-downloads) | |||
2. [Install PyTorch 1.13 or later](https://pytorch.org/get-started/locally/) | |||
3. Install dependencies | |||
```bash | |||
pip install -r requirements.txt | |||
``` | |||
4. Download the data and run training: | |||
```bash | |||
bash scripts/download_data.sh | |||
python train.py --amp | |||
``` | |||
### With Docker | |||
1. [Install Docker 19.03 or later:](https://docs.docker.com/get-docker/) | |||
```bash | |||
curl https://get.docker.com | sh && sudo systemctl --now enable docker | |||
``` | |||
2. [Install the NVIDIA container toolkit:](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) | |||
```bash | |||
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ | |||
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ | |||
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list | |||
sudo apt-get update | |||
sudo apt-get install -y nvidia-docker2 | |||
sudo systemctl restart docker | |||
``` | |||
3. [Download and run the image:](https://hub.docker.com/repository/docker/milesial/unet) | |||
```bash | |||
sudo docker run --rm --shm-size=8g --ulimit memlock=-1 --gpus all -it milesial/unet | |||
``` | |||
4. Download the data and run training: | |||
```bash | |||
bash scripts/download_data.sh | |||
python train.py --amp | |||
``` | |||
## Description | |||
This model was trained from scratch with 5k images and scored a [Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) of 0.988423 on over 100k test images. | |||
It can be easily used for multiclass segmentation, portrait segmentation, medical segmentation, ... | |||
## Usage | |||
**Note : Use Python 3.6 or newer** | |||
### Docker | |||
A docker image containing the code and the dependencies is available on [DockerHub](https://hub.docker.com/repository/docker/milesial/unet). | |||
You can download and jump in the container with ([docker >=19.03](https://docs.docker.com/get-docker/)): | |||
```console | |||
docker run -it --rm --shm-size=8g --ulimit memlock=-1 --gpus all milesial/unet | |||
``` | |||
### Training | |||
```console | |||
> python train.py -h | |||
usage: train.py [-h] [--epochs E] [--batch-size B] [--learning-rate LR] | |||
[--load LOAD] [--scale SCALE] [--validation VAL] [--amp] | |||
Train the UNet on images and target masks | |||
optional arguments: | |||
-h, --help show this help message and exit | |||
--epochs E, -e E Number of epochs | |||
--batch-size B, -b B Batch size | |||
--learning-rate LR, -l LR | |||
Learning rate | |||
--load LOAD, -f LOAD Load model from a .pth file | |||
--scale SCALE, -s SCALE | |||
Downscaling factor of the images | |||
--validation VAL, -v VAL | |||
Percent of the data that is used as validation (0-100) | |||
--amp Use mixed precision | |||
``` | |||
By default, the `scale` is 0.5, so if you wish to obtain better results (but use more memory), set it to 1. | |||
Automatic mixed precision is also available with the `--amp` flag. [Mixed precision](https://arxiv.org/abs/1710.03740) allows the model to use less memory and to be faster on recent GPUs by using FP16 arithmetic. Enabling AMP is recommended. | |||
### Prediction | |||
After training your model and saving it to `MODEL.pth`, you can easily test the output masks on your images via the CLI. | |||
To predict a single image and save it: | |||
`python predict.py -i image.jpg -o output.jpg` | |||
To predict a multiple images and show them without saving them: | |||
`python predict.py -i image1.jpg image2.jpg --viz --no-save` | |||
```console | |||
> python predict.py -h | |||
usage: predict.py [-h] [--model FILE] --input INPUT [INPUT ...] | |||
[--output INPUT [INPUT ...]] [--viz] [--no-save] | |||
[--mask-threshold MASK_THRESHOLD] [--scale SCALE] | |||
Predict masks from input images | |||
optional arguments: | |||
-h, --help show this help message and exit | |||
--model FILE, -m FILE | |||
Specify the file in which the model is stored | |||
--input INPUT [INPUT ...], -i INPUT [INPUT ...] | |||
Filenames of input images | |||
--output INPUT [INPUT ...], -o INPUT [INPUT ...] | |||
Filenames of output images | |||
--viz, -v Visualize the images as they are processed | |||
--no-save, -n Do not save the output masks | |||
--mask-threshold MASK_THRESHOLD, -t MASK_THRESHOLD | |||
Minimum probability value to consider a mask pixel white | |||
--scale SCALE, -s SCALE | |||
Scale factor for the input images | |||
``` | |||
You can specify which model file to use with `--model MODEL.pth`. | |||
## Weights & Biases | |||
The training progress can be visualized in real-time using [Weights & Biases](https://wandb.ai/). Loss curves, validation curves, weights and gradient histograms, as well as predicted masks are logged to the platform. | |||
When launching a training, a link will be printed in the console. Click on it to go to your dashboard. If you have an existing W&B account, you can link it | |||
by setting the `WANDB_API_KEY` environment variable. If not, it will create an anonymous run which is automatically deleted after 7 days. | |||
## Pretrained model | |||
A [pretrained model](https://github.com/milesial/Pytorch-UNet/releases/tag/v3.0) is available for the Carvana dataset. It can also be loaded from torch.hub: | |||
```python | |||
net = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana', pretrained=True, scale=0.5) | |||
``` | |||
Available scales are 0.5 and 1.0. | |||
## Data | |||
The Carvana data is available on the [Kaggle website](https://www.kaggle.com/c/carvana-image-masking-challenge/data). | |||
You can also download it using the helper script: | |||
``` | |||
bash scripts/download_data.sh | |||
``` | |||
The input images and target masks should be in the `data/imgs` and `data/masks` folders respectively (note that the `imgs` and `masks` folder should not contain any sub-folder or any other files, due to the greedy data-loader). For Carvana, images are RGB and masks are black and white. | |||
You can use your own dataset as long as you make sure it is loaded properly in `utils/data_loading.py`. | |||
--- | |||
Original paper by Olaf Ronneberger, Philipp Fischer, Thomas Brox: | |||
[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597) | |||
![network architecture](https://i.imgur.com/jeDVpqF.png) |
@@ -0,0 +1,5 @@ | |||
matplotlib==3.6.2 | |||
numpy==1.23.5 | |||
Pillow==9.3.0 | |||
tqdm==4.64.1 | |||
wandb==0.13.5 |
@@ -0,0 +1,27 @@ | |||
#!/bin/bash | |||
if [[ ! -f ~/.kaggle/kaggle.json ]]; then | |||
echo -n "Kaggle username: " | |||
read USERNAME | |||
echo | |||
echo -n "Kaggle API key: " | |||
read APIKEY | |||
mkdir -p ~/.kaggle | |||
echo "{\"username\":\"$USERNAME\",\"key\":\"$APIKEY\"}" > ~/.kaggle/kaggle.json | |||
chmod 600 ~/.kaggle/kaggle.json | |||
fi | |||
pip install kaggle --upgrade | |||
kaggle competitions download -c carvana-image-masking-challenge -f train_hq.zip | |||
unzip train_hq.zip | |||
mv train_hq/* data/imgs/ | |||
rm -d train_hq | |||
rm train_hq.zip | |||
kaggle competitions download -c carvana-image-masking-challenge -f train_masks.zip | |||
unzip train_masks.zip | |||
mv train_masks/* data/masks/ | |||
rm -d train_masks | |||
rm train_masks.zip |
@@ -0,0 +1,164 @@ | |||
import argparse | |||
import logging | |||
import os | |||
os.environ['CUDA_VISIBLE_DEVICES'] = '1' | |||
import random | |||
import sys | |||
import ms_adapter.pytorch as torch | |||
import ms_adapter.pytorch.nn as nn | |||
import ms_adapter.pytorch.nn.functional as F | |||
import mindspore as ms | |||
import ms_adapter.torchvision.transforms as transforms | |||
import ms_adapter.torchvision.transforms.functional as TF | |||
from pathlib import Path | |||
from ms_adapter.pytorch import optim | |||
from ms_adapter.pytorch.utils.data import DataLoader, random_split | |||
from tqdm import tqdm | |||
from unet import UNet | |||
from utils.data_loading import BasicDataset, CarvanaDataset | |||
from utils.dice_score import dice_loss | |||
dir_img = Path('/code/UNet-master/2dunet-pytorch/data/imgs/') | |||
dir_mask = Path('/code/UNet-master/2dunet-pytorch/data/masks/') | |||
dir_checkpoint = Path('./checkpoints/') | |||
ms.set_context(device_target='GPU') | |||
def train_model( | |||
model, | |||
epochs: int = 5, | |||
batch_size: int = 1, | |||
learning_rate: float = 1e-5, | |||
save_checkpoint: bool = True, | |||
img_scale: float = 0.5, | |||
weight_decay: float = 1e-8, | |||
momentum: float = 0.999, | |||
): | |||
# 1. Create dataset | |||
try: | |||
dataset = CarvanaDataset(dir_img, dir_mask, img_scale) | |||
print('Load CarvanaDataset') | |||
except (AssertionError, RuntimeError, IndexError): | |||
dataset = BasicDataset(dir_img, dir_mask, img_scale) | |||
print('Load BasicDataset') | |||
n_train = len(dataset) | |||
# 3. Create data loaders | |||
loader_args = dict(batch_size=batch_size, num_workers=os.cpu_count()) | |||
train_loader = DataLoader(dataset, shuffle=True, **loader_args) | |||
#val_loader = DataLoader(val_set, shuffle=False, drop_last=True, **loader_args) | |||
# (Initialize logging) | |||
logging.info(f'''Starting training: | |||
Epochs: {epochs} | |||
Batch size: {batch_size} | |||
Learning rate: {learning_rate} | |||
Training size: {n_train} | |||
Checkpoints: {save_checkpoint} | |||
Images scaling: {img_scale} | |||
''') | |||
# 4. Set up the optimizer, the loss, the learning rate scheduler and the loss scaling for AMP | |||
optimizer = ms.nn.RMSProp(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay) | |||
criterion = nn.CrossEntropyLoss() if model.n_classes > 1 else nn.BCEWithLogitsLoss() | |||
global_step = 0 | |||
def forward_fn(data, label): | |||
masks_pred = model(data) | |||
if model.n_classes == 1: | |||
loss = criterion(masks_pred.squeeze(1), true_masks.float()) | |||
loss += dice_loss(F.sigmoid(masks_pred.squeeze(1)), true_masks.float(), multiclass=False) | |||
else: | |||
loss = criterion(masks_pred, true_masks) | |||
loss += dice_loss(F.softmax(masks_pred, dim=1).float(), | |||
F.one_hot(true_masks, model.n_classes).permute(0, 3, 1, 2).float(), multiclass=True) | |||
return loss, masks_pred | |||
grad_fn = ms.ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True) | |||
def train_step(data, label): | |||
(loss, _), grads = grad_fn(data, label) | |||
loss = ms.ops.depend(loss, optimizer(grads)) | |||
return loss | |||
# 5. Begin training | |||
for epoch in range(1, epochs + 1): | |||
model.train() | |||
epoch_loss = 0 | |||
print('train_loader', len(train_loader), train_loader) | |||
with tqdm(total=n_train, desc=f'Epoch {epoch}/{epochs}', unit='img') as pbar: | |||
for batch in train_loader: | |||
images, true_masks = batch['image'], batch['mask'] | |||
assert images.shape[1] == model.n_channels, \ | |||
f'Network has been defined with {model.n_channels} input channels, ' \ | |||
f'but loaded images have {images.shape[1]} channels. Please check that ' \ | |||
'the images are loaded correctly.' | |||
images = images.to(dtype=torch.float32) | |||
true_masks = true_masks.to(dtype=torch.long) | |||
#masks_pred = model(images) | |||
loss = train_step(images, true_masks) | |||
pbar.update(images.shape[0]) | |||
global_step += 1 | |||
epoch_loss += loss.item() | |||
print( | |||
'train loss: %f' % loss.item(), | |||
'step: %d' % global_step, | |||
'epoch: %d' % epoch | |||
) | |||
pbar.set_postfix(**{'loss (batch)': loss.item()}) | |||
def get_args(): | |||
parser = argparse.ArgumentParser(description='Train the UNet on images and target masks') | |||
parser.add_argument('--epochs', '-e', metavar='E', type=int, default=5, help='Number of epochs') | |||
parser.add_argument('--batch-size', '-b', dest='batch_size', metavar='B', type=int, default=1, help='Batch size') | |||
parser.add_argument('--learning-rate', '-l', metavar='LR', type=float, default=1e-5, | |||
help='Learning rate', dest='lr') | |||
parser.add_argument('--load', '-f', type=str, default=False, help='Load model from a .pth file') | |||
parser.add_argument('--scale', '-s', type=float, default=0.5, help='Downscaling factor of the images') | |||
parser.add_argument('--validation', '-v', dest='val', type=float, default=10.0, | |||
help='Percent of the data that is used as validation (0-100)') | |||
parser.add_argument('--bilinear', action='store_true', default=False, help='Use bilinear upsampling') | |||
parser.add_argument('--classes', '-c', type=int, default=2, help='Number of classes') | |||
return parser.parse_args() | |||
if __name__ == '__main__': | |||
args = get_args() | |||
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s') | |||
# Change here to adapt to your data | |||
# n_channels=3 for RGB images | |||
# n_classes is the number of probabilities you want to get per pixel | |||
model = UNet(n_channels=3, n_classes=args.classes, bilinear=args.bilinear) | |||
logging.info(f'Network:\n' | |||
f'\t{model.n_channels} input channels\n' | |||
f'\t{model.n_classes} output channels (classes)\n' | |||
f'\t{"Bilinear" if model.bilinear else "Transposed conv"} upscaling') | |||
train_model( | |||
model=model, | |||
epochs=args.epochs, | |||
batch_size=args.batch_size, | |||
learning_rate=args.lr, | |||
img_scale=args.scale, | |||
) |
@@ -0,0 +1 @@ | |||
from .unet_model import UNet |
@@ -0,0 +1,48 @@ | |||
""" Full assembly of the parts to form the complete network """ | |||
from .unet_parts import * | |||
class UNet(nn.Module): | |||
def __init__(self, n_channels, n_classes, bilinear=False): | |||
super(UNet, self).__init__() | |||
self.n_channels = n_channels | |||
self.n_classes = n_classes | |||
self.bilinear = bilinear | |||
self.inc = (DoubleConv(n_channels, 64)) | |||
self.down1 = (Down(64, 128)) | |||
self.down2 = (Down(128, 256)) | |||
self.down3 = (Down(256, 512)) | |||
factor = 2 if bilinear else 1 | |||
self.down4 = (Down(512, 1024 // factor)) | |||
self.up1 = (Up(1024, 512 // factor, bilinear)) | |||
self.up2 = (Up(512, 256 // factor, bilinear)) | |||
self.up3 = (Up(256, 128 // factor, bilinear)) | |||
self.up4 = (Up(128, 64, bilinear)) | |||
self.outc = (OutConv(64, n_classes)) | |||
def forward(self, x): | |||
x1 = self.inc(x) | |||
x2 = self.down1(x1) | |||
x3 = self.down2(x2) | |||
x4 = self.down3(x3) | |||
x5 = self.down4(x4) | |||
x = self.up1(x5, x4) | |||
x = self.up2(x, x3) | |||
x = self.up3(x, x2) | |||
x = self.up4(x, x1) | |||
logits = self.outc(x) | |||
return logits | |||
def use_checkpointing(self): | |||
self.inc = torch.utils.checkpoint(self.inc) | |||
self.down1 = torch.utils.checkpoint(self.down1) | |||
self.down2 = torch.utils.checkpoint(self.down2) | |||
self.down3 = torch.utils.checkpoint(self.down3) | |||
self.down4 = torch.utils.checkpoint(self.down4) | |||
self.up1 = torch.utils.checkpoint(self.up1) | |||
self.up2 = torch.utils.checkpoint(self.up2) | |||
self.up3 = torch.utils.checkpoint(self.up3) | |||
self.up4 = torch.utils.checkpoint(self.up4) | |||
self.outc = torch.utils.checkpoint(self.outc) |
@@ -0,0 +1,77 @@ | |||
""" Parts of the U-Net model """ | |||
import ms_adapter.pytorch as torch | |||
import ms_adapter.pytorch.nn as nn | |||
import ms_adapter.pytorch.nn.functional as F | |||
class DoubleConv(nn.Module): | |||
"""(convolution => [BN] => ReLU) * 2""" | |||
def __init__(self, in_channels, out_channels, mid_channels=None): | |||
super().__init__() | |||
if not mid_channels: | |||
mid_channels = out_channels | |||
self.double_conv = nn.Sequential( | |||
nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1, bias=False), | |||
nn.BatchNorm2d(mid_channels), | |||
nn.ReLU(inplace=True), | |||
nn.Conv2d(mid_channels, out_channels, kernel_size=3, padding=1, bias=False), | |||
nn.BatchNorm2d(out_channels), | |||
nn.ReLU(inplace=True) | |||
) | |||
def forward(self, x): | |||
return self.double_conv(x) | |||
class Down(nn.Module): | |||
"""Downscaling with maxpool then double conv""" | |||
def __init__(self, in_channels, out_channels): | |||
super().__init__() | |||
self.maxpool_conv = nn.Sequential( | |||
nn.MaxPool2d(2), | |||
DoubleConv(in_channels, out_channels) | |||
) | |||
def forward(self, x): | |||
return self.maxpool_conv(x) | |||
class Up(nn.Module): | |||
"""Upscaling then double conv""" | |||
def __init__(self, in_channels, out_channels, bilinear=True): | |||
super().__init__() | |||
# if bilinear, use the normal convolutions to reduce the number of channels | |||
if bilinear: | |||
self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True) | |||
self.conv = DoubleConv(in_channels, out_channels, in_channels // 2) | |||
else: | |||
self.up = nn.ConvTranspose2d(in_channels, in_channels // 2, kernel_size=2, stride=2) | |||
self.conv = DoubleConv(in_channels, out_channels) | |||
def forward(self, x1, x2): | |||
x1 = self.up(x1) | |||
# input is CHW | |||
diffY = x2.size()[2] - x1.size()[2] | |||
diffX = x2.size()[3] - x1.size()[3] | |||
x1 = F.pad(x1, [diffX // 2, diffX - diffX // 2, | |||
diffY // 2, diffY - diffY // 2]) | |||
# if you have padding issues, see | |||
# https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a | |||
# https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd | |||
x = torch.cat([x2, x1], dim=1) | |||
return self.conv(x) | |||
class OutConv(nn.Module): | |||
def __init__(self, in_channels, out_channels): | |||
super(OutConv, self).__init__() | |||
self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1) | |||
def forward(self, x): | |||
return self.conv(x) |
@@ -0,0 +1,117 @@ | |||
import logging | |||
import numpy as np | |||
import ms_adapter.pytorch as torch | |||
from PIL import Image | |||
from functools import lru_cache | |||
from functools import partial | |||
from itertools import repeat | |||
from multiprocessing import Pool | |||
from os import listdir | |||
from os.path import splitext, isfile, join | |||
from pathlib import Path | |||
from ms_adapter.pytorch.utils.data import Dataset | |||
from tqdm import tqdm | |||
def load_image(filename): | |||
ext = splitext(filename)[1] | |||
if ext == '.npy': | |||
return Image.fromarray(np.load(filename)) | |||
elif ext in ['.pt', '.pth']: | |||
return Image.fromarray(torch.load(filename).numpy()) | |||
else: | |||
return Image.open(filename) | |||
def unique_mask_values(idx, mask_dir, mask_suffix): | |||
mask_file = list(mask_dir.glob(idx + mask_suffix + '.*'))[0] | |||
mask = np.asarray(load_image(mask_file)) | |||
if mask.ndim == 2: | |||
return np.unique(mask) | |||
elif mask.ndim == 3: | |||
mask = mask.reshape(-1, mask.shape[-1]) | |||
return np.unique(mask, axis=0) | |||
else: | |||
raise ValueError(f'Loaded masks should have 2 or 3 dimensions, found {mask.ndim}') | |||
class BasicDataset(Dataset): | |||
def __init__(self, images_dir: str, mask_dir: str, scale: float = 1.0, mask_suffix: str = ''): | |||
self.images_dir = Path(images_dir) | |||
self.mask_dir = Path(mask_dir) | |||
assert 0 < scale <= 1, 'Scale must be between 0 and 1' | |||
self.scale = scale | |||
self.mask_suffix = mask_suffix | |||
self.ids = [splitext(file)[0] for file in listdir(images_dir) if isfile(join(images_dir, file)) and not file.startswith('.')] | |||
if not self.ids: | |||
raise RuntimeError(f'No input file found in {images_dir}, make sure you put your images there') | |||
logging.info(f'Creating dataset with {len(self.ids)} examples') | |||
logging.info('Scanning mask files to determine unique values') | |||
with Pool() as p: | |||
unique = list(tqdm( | |||
p.imap(partial(unique_mask_values, mask_dir=self.mask_dir, mask_suffix=self.mask_suffix), self.ids), | |||
total=len(self.ids) | |||
)) | |||
self.mask_values = list(sorted(np.unique(np.concatenate(unique), axis=0).tolist())) | |||
logging.info(f'Unique mask values: {self.mask_values}') | |||
def __len__(self): | |||
return len(self.ids) | |||
@staticmethod | |||
def preprocess(mask_values, pil_img, scale, is_mask): | |||
w, h = pil_img.size | |||
newW, newH = int(scale * w), int(scale * h) | |||
assert newW > 0 and newH > 0, 'Scale is too small, resized images would have no pixel' | |||
pil_img = pil_img.resize((newW, newH), resample=Image.NEAREST if is_mask else Image.BICUBIC) | |||
img = np.asarray(pil_img) | |||
if is_mask: | |||
mask = np.zeros((newH, newW), dtype=np.int64) | |||
for i, v in enumerate(mask_values): | |||
if img.ndim == 2: | |||
mask[img == v] = i | |||
else: | |||
mask[(img == v).all(-1)] = i | |||
return mask | |||
else: | |||
if img.ndim == 2: | |||
img = img[np.newaxis, ...] | |||
else: | |||
img = img.transpose((2, 0, 1)) | |||
if (img > 1).any(): | |||
img = img / 255.0 | |||
return img | |||
def __getitem__(self, idx): | |||
name = self.ids[idx] | |||
mask_file = list(self.mask_dir.glob(name + self.mask_suffix + '.*')) | |||
img_file = list(self.images_dir.glob(name + '.*')) | |||
assert len(img_file) == 1, f'Either no image or multiple images found for the ID {name}: {img_file}' | |||
assert len(mask_file) == 1, f'Either no mask or multiple masks found for the ID {name}: {mask_file}' | |||
mask = load_image(mask_file[0]) | |||
img = load_image(img_file[0]) | |||
assert img.size == mask.size, \ | |||
f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}' | |||
img = self.preprocess(self.mask_values, img, self.scale, is_mask=False) | |||
mask = self.preprocess(self.mask_values, mask, self.scale, is_mask=True) | |||
return { | |||
'image': torch.as_tensor(img.copy()).float().contiguous(), | |||
'mask': torch.as_tensor(mask.copy()).long().contiguous() | |||
} | |||
class CarvanaDataset(BasicDataset): | |||
def __init__(self, images_dir, mask_dir, scale=1): | |||
super().__init__(images_dir, mask_dir, scale, mask_suffix='_mask') |
@@ -0,0 +1,28 @@ | |||
import ms_adapter.pytorch as torch | |||
from ms_adapter.pytorch import Tensor | |||
def dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): | |||
# Average of Dice coefficient for all batches, or for a single mask | |||
assert input.size() == target.size() | |||
assert input.dim() == 3 or not reduce_batch_first | |||
sum_dim = (-1, -2) if input.dim() == 2 or not reduce_batch_first else (-1, -2, -3) | |||
inter = 2 * (input * target).sum(dim=sum_dim) | |||
sets_sum = input.sum(dim=sum_dim) + target.sum(dim=sum_dim) | |||
sets_sum = torch.where(sets_sum == 0, inter, sets_sum) | |||
dice = (inter + epsilon) / (sets_sum + epsilon) | |||
return dice.mean() | |||
def multiclass_dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): | |||
# Average of Dice coefficient for all classes | |||
return dice_coeff(input.flatten(0, 1), target.flatten(0, 1), reduce_batch_first, epsilon) | |||
def dice_loss(input: Tensor, target: Tensor, multiclass: bool = False): | |||
# Dice loss (objective to minimize) between 0 and 1 | |||
fn = multiclass_dice_coeff if multiclass else dice_coeff | |||
return 1 - fn(input, target, reduce_batch_first=True) |
@@ -0,0 +1,13 @@ | |||
import matplotlib.pyplot as plt | |||
def plot_img_and_mask(img, mask): | |||
classes = mask.max() + 1 | |||
fig, ax = plt.subplots(1, classes + 1) | |||
ax[0].set_title('Input image') | |||
ax[0].imshow(img) | |||
for i in range(classes): | |||
ax[i + 1].set_title(f'Mask (class {i + 1})') | |||
ax[i + 1].imshow(mask == i) | |||
plt.xticks([]), plt.yticks([]) | |||
plt.show() |
@@ -0,0 +1,312 @@ | |||
2023-03-10 00:10:05,030 INFO StreamThr :17139 [internal.py:wandb_internal():90] W&B internal server running at pid: 17139, started at: 2023-03-10 00:10:05.028238 | |||
2023-03-10 00:10:05,031 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status | |||
2023-03-10 00:10:05,033 INFO WriterThread:17139 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb | |||
2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: header | |||
2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: run | |||
2023-03-10 00:10:05,174 INFO SenderThread:17139 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/215re05m | |||
2023-03-10 00:10:06,363 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-10 00:10:06,364 INFO SenderThread:17139 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files | |||
2023-03-10 00:10:06,364 INFO SenderThread:17139 [sender.py:_start_run_threads():1081] run started: 215re05m with start time 1678378205.024339 | |||
2023-03-10 00:10:06,364 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-10 00:10:06,422 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-10 00:10:06,422 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: check_version | |||
2023-03-10 00:10:07,368 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json | |||
2023-03-10 00:10:11,365 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:11,423 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:11,444 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():31] System info init | |||
2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():46] System info init done | |||
2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-10 00:10:11,449 INFO SystemMonitor:17139 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-10 00:10:11,450 INFO SystemMonitor:17139 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started network monitoring | |||
2023-03-10 00:10:11,489 DEBUG HandlerThread:17139 [system_info.py:probe():195] Probing system | |||
2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [git.py:repo():40] git repository is invalid | |||
2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_info.py:probe():240] Probing system done | |||
2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T16:10:11.489682', 'startedAt': '2023-03-09T16:10:04.972918', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.301262500001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.047, 'min': 0.0, 'max': 0.0}, {'current': 2505.411, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2497.435, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.275, 'min': 0.0, 'max': 0.0}, {'current': 2497.673, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2501.929, 'min': 0.0, 'max': 0.0}, {'current': 2500.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.824, 'min': 0.0, 'max': 0.0}, {'current': 2498.743, 'min': 0.0, 'max': 0.0}, {'current': 2498.804, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2501.118, 'min': 0.0, 'max': 0.0}, {'current': 2502.038, 'min': 0.0, 'max': 0.0}, {'current': 2499.891, 'min': 0.0, 'max': 0.0}, {'current': 2500.327, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2499.951, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.433, 'min': 0.0, 'max': 0.0}, {'current': 2501.418, 'min': 0.0, 'max': 0.0}, {'current': 2501.066, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.969, 'min': 0.0, 'max': 0.0}, {'current': 2499.963, 'min': 0.0, 'max': 0.0}, {'current': 2499.319, 'min': 0.0, 'max': 0.0}, {'current': 2499.62, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2502.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2504.876, 'min': 0.0, 'max': 0.0}, {'current': 2499.546, 'min': 0.0, 'max': 0.0}, {'current': 2504.283, 'min': 0.0, 'max': 0.0}, {'current': 2506.421, 'min': 0.0, 'max': 0.0}, {'current': 2499.741, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2498.673, 'min': 0.0, 'max': 0.0}, {'current': 2505.326, 'min': 0.0, 'max': 0.0}, {'current': 2504.107, 'min': 0.0, 'max': 0.0}, {'current': 2505.212, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2509.28, 'min': 0.0, 'max': 0.0}, {'current': 2500.431, 'min': 0.0, 'max': 0.0}, {'current': 2499.538, 'min': 0.0, 'max': 0.0}, {'current': 2500.876, 'min': 0.0, 'max': 0.0}, {'current': 2503.961, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2500.3, 'min': 0.0, 'max': 0.0}, {'current': 2500.458, 'min': 0.0, 'max': 0.0}, {'current': 2499.894, 'min': 0.0, 'max': 0.0}, {'current': 2502.906, 'min': 0.0, 'max': 0.0}, {'current': 2499.885, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2504.513, 'min': 0.0, 'max': 0.0}, {'current': 2504.644, 'min': 0.0, 'max': 0.0}, {'current': 2503.685, 'min': 0.0, 'max': 0.0}, {'current': 2500.655, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.238, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.672, 'min': 0.0, 'max': 0.0}, {'current': 2504.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.59, 'min': 0.0, 'max': 0.0}, {'current': 2500.478, 'min': 0.0, 'max': 0.0}, {'current': 2500.85, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6511535644531}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-10 00:10:11,517 DEBUG HandlerThread:17139 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-10 00:10:12,268 DEBUG HandlerThread:17139 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-10 00:10:12,302 INFO HandlerThread:17139 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-10 00:10:12,305 DEBUG SenderThread:17139 [sender.py:send():336] send: files | |||
2023-03-10 00:10:12,306 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-10 00:10:12,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-10 00:10:12,333 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml | |||
2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt | |||
2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json | |||
2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: telemetry | |||
2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: config | |||
2023-03-10 00:10:13,506 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /tmp/tmpf905ug1_wandb/11iue9v6-wandb-metadata.json | |||
2023-03-10 00:10:13,547 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log | |||
2023-03-10 00:10:15,549 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log | |||
2023-03-10 00:10:16,804 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:21,805 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:26,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:31,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:32,316 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:36,812 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:37,317 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:37,569 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml | |||
2023-03-10 00:10:42,303 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:42,319 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:47,304 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:47,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:52,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:52,322 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:10:57,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:10:57,323 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:02,306 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:02,324 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:07,307 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:07,325 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:11,452 DEBUG SystemMonitor:17139 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-10 00:11:11,458 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:11:12,326 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:12,459 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:17,327 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:17,460 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:22,329 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:22,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:27,330 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:27,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:32,331 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:32,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:37,332 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:37,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:41,460 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:11:42,333 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:43,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:47,334 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:48,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:52,336 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:53,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:11:57,337 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:11:58,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:02,338 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:03,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:07,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:08,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:11,463 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:12:12,340 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:14,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:17,342 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:19,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:22,343 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:24,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:27,344 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:29,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:32,345 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:34,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:37,346 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:39,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:41,466 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:12:42,348 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:45,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:47,349 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:50,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:52,350 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:12:55,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:12:57,351 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:00,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:02,352 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:05,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:07,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:10,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:11,469 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:13:12,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:16,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:17,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:21,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:22,357 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:26,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:27,358 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:31,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:32,359 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:36,473 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:37,360 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:41,548 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:41,550 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:13:42,362 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:46,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:47,373 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:51,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:52,375 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:13:56,553 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:13:57,376 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:01,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:02,377 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:06,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:07,378 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:11,553 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:14:12,379 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:12,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:17,381 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:17,555 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:22,382 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:22,556 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:27,383 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:27,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:32,384 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:32,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:37,386 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:37,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:41,555 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:14:42,387 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:43,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:47,388 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:48,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:52,389 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:53,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:14:57,390 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:14:58,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:02,391 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:15:03,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:07,393 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:15:08,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:11,558 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:15:12,394 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:15:14,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:17,409 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:15:18,419 DEBUG SenderThread:17139 [sender.py:send():336] send: exit | |||
2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():559] handling exit code: 1 | |||
2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():561] handling runtime: 306 | |||
2023-03-10 00:15:18,723 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-10 00:15:18,724 INFO SenderThread:17139 [sender.py:send_exit():567] send defer | |||
2023-03-10 00:15:18,724 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:18,725 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 0 | |||
2023-03-10 00:15:18,725 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 0 | |||
2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 1 | |||
2023-03-10 00:15:18,726 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:18,726 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 1 | |||
2023-03-10 00:15:18,726 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 1 | |||
2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 2 | |||
2023-03-10 00:15:18,808 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:18,808 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 2 | |||
2023-03-10 00:15:18,808 INFO HandlerThread:17139 [system_monitor.py:finish():193] Stopping system monitor | |||
2023-03-10 00:15:19,020 DEBUG SystemMonitor:17139 [system_monitor.py:_start():168] Finished system metrics aggregation loop | |||
2023-03-10 00:15:19,021 DEBUG SystemMonitor:17139 [system_monitor.py:_start():172] Publishing last batch of metrics | |||
2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined cpu monitor | |||
2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined disk monitor | |||
2023-03-10 00:15:19,110 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json | |||
2023-03-10 00:15:20,110 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined gpu monitor | |||
2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined memory monitor | |||
2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined network monitor | |||
2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:20,113 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 2 | |||
2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 3 | |||
2023-03-10 00:15:20,114 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:20,114 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 3 | |||
2023-03-10 00:15:20,208 DEBUG SenderThread:17139 [sender.py:send():336] send: stats | |||
2023-03-10 00:15:20,209 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-10 00:15:20,210 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 3 | |||
2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 4 | |||
2023-03-10 00:15:20,211 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:20,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log | |||
2023-03-10 00:15:20,211 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 4 | |||
2023-03-10 00:15:20,211 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,211 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 4 | |||
2023-03-10 00:15:20,212 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 5 | |||
2023-03-10 00:15:20,212 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:20,212 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 5 | |||
2023-03-10 00:15:20,212 DEBUG SenderThread:17139 [sender.py:send():336] send: summary | |||
2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-10 00:15:20,233 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 5 | |||
2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 6 | |||
2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 6 | |||
2023-03-10 00:15:20,234 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 6 | |||
2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 7 | |||
2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 7 | |||
2023-03-10 00:15:20,235 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:20,235 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 7 | |||
2023-03-10 00:15:20,511 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-10 00:15:21,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json | |||
2023-03-10 00:15:23,339 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 8 | |||
2023-03-10 00:15:23,339 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-10 00:15:23,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:23,340 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 8 | |||
2023-03-10 00:15:23,340 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:23,340 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 8 | |||
2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 9 | |||
2023-03-10 00:15:23,341 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:23,341 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 9 | |||
2023-03-10 00:15:23,341 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 9 | |||
2023-03-10 00:15:23,341 INFO SenderThread:17139 [dir_watcher.py:finish():365] shutting down directory watcher | |||
2023-03-10 00:15:24,213 INFO SenderThread:17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log | |||
2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files | |||
2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml conda-environment.yaml | |||
2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log output.log | |||
2023-03-10 00:15:24,215 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt requirements.txt | |||
2023-03-10 00:15:24,220 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml config.yaml | |||
2023-03-10 00:15:24,241 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json wandb-summary.json | |||
2023-03-10 00:15:24,256 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json wandb-metadata.json | |||
2023-03-10 00:15:24,261 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 10 | |||
2023-03-10 00:15:24,277 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:24,287 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 10 | |||
2023-03-10 00:15:24,419 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:24,424 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 10 | |||
2023-03-10 00:15:24,424 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-10 00:15:25,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-10 00:15:25,536 INFO wandb-upload_4:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json | |||
2023-03-10 00:15:25,612 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-10 00:15:25,889 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log | |||
2023-03-10 00:15:25,909 INFO wandb-upload_2:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt | |||
2023-03-10 00:15:26,005 INFO wandb-upload_3:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml | |||
2023-03-10 00:15:26,205 INFO Thread-12 :17139 [sender.py:transition_state():587] send defer: 11 | |||
2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 11 | |||
2023-03-10 00:15:26,206 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 11 | |||
2023-03-10 00:15:26,206 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher | |||
2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 12 | |||
2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 12 | |||
2023-03-10 00:15:26,207 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:26,207 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 12 | |||
2023-03-10 00:15:28,352 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 13 | |||
2023-03-10 00:15:28,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:28,353 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 13 | |||
2023-03-10 00:15:28,353 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 13 | |||
2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 14 | |||
2023-03-10 00:15:28,354 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-10 00:15:28,354 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 14 | |||
2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: final | |||
2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: footer | |||
2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer | |||
2023-03-10 00:15:28,354 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 14 | |||
2023-03-10 00:15:28,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-10 00:15:28,355 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-10 00:15:28,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: server_info | |||
2023-03-10 00:15:28,356 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: server_info | |||
2023-03-10 00:15:28,392 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: get_summary | |||
2023-03-10 00:15:28,438 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: sampled_history | |||
2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3429] rendering history | |||
2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3461] rendering summary | |||
2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_sync_info():3387] logging synced files | |||
2023-03-10 00:15:29,089 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: shutdown | |||
2023-03-10 00:15:29,090 INFO HandlerThread:17139 [handler.py:finish():842] shutting down handler | |||
2023-03-10 00:15:29,356 INFO WriterThread:17139 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb | |||
2023-03-10 00:15:30,089 INFO SenderThread:17139 [sender.py:finish():1504] shutting down sender | |||
2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher |
@@ -0,0 +1,28 @@ | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Configure stats pid to 15417 | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():546] calling init triggers | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():602] starting backend | |||
2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():606] setting up manager | |||
2023-03-10 00:10:05,022 INFO MainThread:15417 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-10 00:10:05,024 INFO MainThread:15417 [wandb_init.py:init():613] backend started and connected | |||
2023-03-10 00:10:05,028 INFO MainThread:15417 [wandb_init.py:init():701] updated telemetry | |||
2023-03-10 00:10:05,029 INFO MainThread:15417 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-10 00:10:06,363 INFO MainThread:15417 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-10 00:10:12,314 INFO MainThread:15417 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-10 00:10:12,315 INFO MainThread:15417 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} | |||
2023-03-10 00:15:30,109 WARNING MsgRouterThr:15417 [router.py:message_loop():77] message_loop has been closed |
@@ -0,0 +1,40 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678374242.245928 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,6 @@ | |||
Traceback (most recent call last): | |||
File "train.py", line 170, in <module> | |||
img_scale=args.scale, | |||
File "train.py", line 68, in train_model | |||
''') | |||
NameError: name 'n_val' is not defined |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:04:08.672479", | |||
"startedAt": "2023-03-09T15:04:02.165764", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.5035375, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2501.774, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.632, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.844, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.227, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.25, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.66, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.269, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.584, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.412, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.957, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.437, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.395, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.827, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.381, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.589, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.571, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.823, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.653, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2494.369, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.905, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.341, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.224, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.123, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.567, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.922, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.487, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.288, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.937, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.938, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.099, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.616, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.366, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.679, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.537, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.027, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.211, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.78, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.362, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.002, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.673, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.016, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.354, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.742, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.466, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.896, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.131, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.496, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.341, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.252, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.01, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.34, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.367, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.053, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.115, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.91, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.027, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.852, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.801, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.094, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.488, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.156, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2497.755, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.669, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.714, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.401, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.973, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.132, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.65414810180664 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{"_wandb": {"runtime": 1}} |
@@ -0,0 +1,209 @@ | |||
2023-03-09 23:04:02,252 INFO StreamThr :5402 [internal.py:wandb_internal():90] W&B internal server running at pid: 5402, started at: 2023-03-09 23:04:02.250393 | |||
2023-03-09 23:04:02,254 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:04:02,255 INFO WriterThread:5402 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb | |||
2023-03-09 23:04:02,261 DEBUG SenderThread:5402 [sender.py:send():336] send: header | |||
2023-03-09 23:04:02,261 DEBUG SenderThread:5402 [sender.py:send():336] send: run | |||
2023-03-09 23:04:02,318 INFO SenderThread:5402 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/tfw61kps | |||
2023-03-09 23:04:03,555 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:04:03,556 INFO SenderThread:5402 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files | |||
2023-03-09 23:04:03,556 INFO SenderThread:5402 [sender.py:_start_run_threads():1081] run started: tfw61kps with start time 1678374242.245928 | |||
2023-03-09 23:04:03,556 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:04:03,606 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:04:03,606 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:04:04,560 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json | |||
2023-03-09 23:04:08,556 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:04:08,608 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:04:08,626 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:04:08,633 DEBUG HandlerThread:5402 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:04:08,633 DEBUG HandlerThread:5402 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:04:08,633 INFO HandlerThread:5402 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:04:08,634 INFO SystemMonitor:5402 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:04:08,634 INFO HandlerThread:5402 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:04:08,634 INFO SystemMonitor:5402 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:04:08,635 INFO SystemMonitor:5402 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:04:08,635 INFO SystemMonitor:5402 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:04:08,636 INFO SystemMonitor:5402 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:04:08,636 INFO SystemMonitor:5402 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:04:08,672 DEBUG HandlerThread:5402 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:04:08.672479', 'startedAt': '2023-03-09T15:04:02.165764', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5035375, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2501.774, 'min': 0.0, 'max': 0.0}, {'current': 2499.632, 'min': 0.0, 'max': 0.0}, {'current': 2500.844, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.227, 'min': 0.0, 'max': 0.0}, {'current': 2498.25, 'min': 0.0, 'max': 0.0}, {'current': 2498.66, 'min': 0.0, 'max': 0.0}, {'current': 2501.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.584, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.412, 'min': 0.0, 'max': 0.0}, {'current': 2500.957, 'min': 0.0, 'max': 0.0}, {'current': 2500.437, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.381, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2499.571, 'min': 0.0, 'max': 0.0}, {'current': 2500.823, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2494.369, 'min': 0.0, 'max': 0.0}, {'current': 2499.905, 'min': 0.0, 'max': 0.0}, {'current': 2500.341, 'min': 0.0, 'max': 0.0}, {'current': 2500.224, 'min': 0.0, 'max': 0.0}, {'current': 2500.123, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2499.922, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.487, 'min': 0.0, 'max': 0.0}, {'current': 2500.288, 'min': 0.0, 'max': 0.0}, {'current': 2499.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.938, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2501.366, 'min': 0.0, 'max': 0.0}, {'current': 2499.679, 'min': 0.0, 'max': 0.0}, {'current': 2505.537, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2505.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.78, 'min': 0.0, 'max': 0.0}, {'current': 2504.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2505.673, 'min': 0.0, 'max': 0.0}, {'current': 2499.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.354, 'min': 0.0, 'max': 0.0}, {'current': 2503.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.466, 'min': 0.0, 'max': 0.0}, {'current': 2500.896, 'min': 0.0, 'max': 0.0}, {'current': 2500.131, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.496, 'min': 0.0, 'max': 0.0}, {'current': 2505.341, 'min': 0.0, 'max': 0.0}, {'current': 2505.252, 'min': 0.0, 'max': 0.0}, {'current': 2505.01, 'min': 0.0, 'max': 0.0}, {'current': 2506.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2503.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.115, 'min': 0.0, 'max': 0.0}, {'current': 2505.91, 'min': 0.0, 'max': 0.0}, {'current': 2504.027, 'min': 0.0, 'max': 0.0}, {'current': 2501.852, 'min': 0.0, 'max': 0.0}, {'current': 2504.801, 'min': 0.0, 'max': 0.0}, {'current': 2499.094, 'min': 0.0, 'max': 0.0}, {'current': 2504.488, 'min': 0.0, 'max': 0.0}, {'current': 2504.156, 'min': 0.0, 'max': 0.0}, {'current': 2497.755, 'min': 0.0, 'max': 0.0}, {'current': 2504.669, 'min': 0.0, 'max': 0.0}, {'current': 2504.714, 'min': 0.0, 'max': 0.0}, {'current': 2504.401, 'min': 0.0, 'max': 0.0}, {'current': 2504.973, 'min': 0.0, 'max': 0.0}, {'current': 2501.132, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.65414810180664}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:04:08,697 INFO HandlerThread:5402 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:04:08,697 INFO HandlerThread:5402 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:04:08,730 DEBUG HandlerThread:5402 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:04:08,731 DEBUG HandlerThread:5402 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:04:09,530 DEBUG HandlerThread:5402 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:04:09,551 INFO HandlerThread:5402 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:04:09,555 DEBUG SenderThread:5402 [sender.py:send():336] send: files | |||
2023-03-09 23:04:09,556 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:04:09,574 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:04:09,607 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:04:09,801 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml | |||
2023-03-09 23:04:09,802 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt | |||
2023-03-09 23:04:09,802 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json | |||
2023-03-09 23:04:10,151 DEBUG SenderThread:5402 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:04:10,156 DEBUG SenderThread:5402 [sender.py:send():336] send: config | |||
2023-03-09 23:04:10,168 DEBUG SenderThread:5402 [sender.py:send():336] send: exit | |||
2023-03-09 23:04:10,168 INFO SenderThread:5402 [sender.py:send_exit():559] handling exit code: 1 | |||
2023-03-09 23:04:10,168 INFO SenderThread:5402 [sender.py:send_exit():561] handling runtime: 1 | |||
2023-03-09 23:04:10,227 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:04:10,237 INFO SenderThread:5402 [sender.py:send_exit():567] send defer | |||
2023-03-09 23:04:10,253 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,258 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 0 | |||
2023-03-09 23:04:10,278 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,283 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 0 | |||
2023-03-09 23:04:10,283 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 1 | |||
2023-03-09 23:04:10,283 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,283 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 1 | |||
2023-03-09 23:04:10,284 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,284 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 1 | |||
2023-03-09 23:04:10,284 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 2 | |||
2023-03-09 23:04:10,284 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,284 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 2 | |||
2023-03-09 23:04:10,284 INFO HandlerThread:5402 [system_monitor.py:finish():193] Stopping system monitor | |||
2023-03-09 23:04:10,285 DEBUG SystemMonitor:5402 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:04:10,286 DEBUG SystemMonitor:5402 [system_monitor.py:_start():168] Finished system metrics aggregation loop | |||
2023-03-09 23:04:10,286 DEBUG SystemMonitor:5402 [system_monitor.py:_start():172] Publishing last batch of metrics | |||
2023-03-09 23:04:10,288 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined cpu monitor | |||
2023-03-09 23:04:10,288 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined disk monitor | |||
2023-03-09 23:04:10,302 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined gpu monitor | |||
2023-03-09 23:04:10,303 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined memory monitor | |||
2023-03-09 23:04:10,303 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined network monitor | |||
2023-03-09 23:04:10,304 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,304 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 2 | |||
2023-03-09 23:04:10,304 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 3 | |||
2023-03-09 23:04:10,305 DEBUG SenderThread:5402 [sender.py:send():336] send: stats | |||
2023-03-09 23:04:10,305 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,305 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 3 | |||
2023-03-09 23:04:10,306 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 3 | |||
2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 4 | |||
2023-03-09 23:04:10,306 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,306 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 4 | |||
2023-03-09 23:04:10,306 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 4 | |||
2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 5 | |||
2023-03-09 23:04:10,307 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,307 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 5 | |||
2023-03-09 23:04:10,307 DEBUG SenderThread:5402 [sender.py:send():336] send: summary | |||
2023-03-09 23:04:10,334 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:04:10,335 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 5 | |||
2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 6 | |||
2023-03-09 23:04:10,335 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,335 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 6 | |||
2023-03-09 23:04:10,335 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 6 | |||
2023-03-09 23:04:10,341 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:04:10,671 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:10,767 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 7 | |||
2023-03-09 23:04:10,767 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:10,767 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:10,767 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 7 | |||
2023-03-09 23:04:10,768 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:10,768 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 7 | |||
2023-03-09 23:04:10,784 INFO Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml | |||
2023-03-09 23:04:10,784 INFO Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json | |||
2023-03-09 23:04:10,863 INFO wandb-upload_0:5402 [upload_job.py:push():138] Uploaded file /tmp/tmpbm6fg7powandb/0dy5vwa2-wandb-metadata.json | |||
2023-03-09 23:04:11,672 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:12,197 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 8 | |||
2023-03-09 23:04:12,202 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:12,203 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:12,208 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 8 | |||
2023-03-09 23:04:12,229 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:12,229 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 8 | |||
2023-03-09 23:04:12,265 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 9 | |||
2023-03-09 23:04:12,280 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:12,280 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 9 | |||
2023-03-09 23:04:12,286 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:12,286 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 9 | |||
2023-03-09 23:04:12,286 INFO SenderThread:5402 [dir_watcher.py:finish():365] shutting down directory watcher | |||
2023-03-09 23:04:12,684 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:12,786 INFO SenderThread:5402 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files | |||
2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml conda-environment.yaml | |||
2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt requirements.txt | |||
2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml config.yaml | |||
2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json wandb-summary.json | |||
2023-03-09 23:04:12,792 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json wandb-metadata.json | |||
2023-03-09 23:04:12,793 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 10 | |||
2023-03-09 23:04:12,793 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:12,818 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:12,818 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 10 | |||
2023-03-09 23:04:12,940 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:12,946 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 10 | |||
2023-03-09 23:04:12,946 INFO SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:04:13,726 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:13,747 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:13,947 INFO wandb-upload_3:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json | |||
2023-03-09 23:04:14,378 INFO wandb-upload_2:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml | |||
2023-03-09 23:04:14,732 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:14,733 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:15,734 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:15,734 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:16,734 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:16,735 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:17,736 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:17,736 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:18,736 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:18,737 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:19,244 ERROR wandb-upload_1:5402 [internal_api.py:execute():245] 502 response executing GraphQL. | |||
2023-03-09 23:04:19,245 ERROR wandb-upload_1:5402 [internal_api.py:execute():246] | |||
<html><head> | |||
<meta http-equiv="content-type" content="text/html;charset=utf-8"> | |||
<title>502 Server Error</title> | |||
</head> | |||
<body text=#000000 bgcolor=#ffffff> | |||
<h1>Error: Server Error</h1> | |||
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2> | |||
<h2></h2> | |||
</body></html> | |||
2023-03-09 23:04:19,738 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:04:19,738 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:19,739 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:20,739 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:20,740 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:21,423 INFO wandb-upload_1:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt | |||
2023-03-09 23:04:21,623 INFO Thread-12 :5402 [sender.py:transition_state():587] send defer: 11 | |||
2023-03-09 23:04:21,623 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:21,623 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 11 | |||
2023-03-09 23:04:21,624 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 11 | |||
2023-03-09 23:04:21,624 INFO SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher | |||
2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 12 | |||
2023-03-09 23:04:21,624 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:21,624 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 12 | |||
2023-03-09 23:04:21,624 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 12 | |||
2023-03-09 23:04:21,809 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:22,109 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 13 | |||
2023-03-09 23:04:22,109 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:22,109 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:22,110 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 13 | |||
2023-03-09 23:04:22,110 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:22,110 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 13 | |||
2023-03-09 23:04:22,110 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 14 | |||
2023-03-09 23:04:22,110 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:04:22,111 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 14 | |||
2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send():336] send: final | |||
2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send():336] send: footer | |||
2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:04:22,111 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 14 | |||
2023-03-09 23:04:22,112 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:22,112 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:22,113 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:04:22,113 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:04:22,113 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: server_info | |||
2023-03-09 23:04:22,114 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: server_info | |||
2023-03-09 23:04:22,150 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: get_summary | |||
2023-03-09 23:04:22,165 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: sampled_history | |||
2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_history_summary_info():3429] rendering history | |||
2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_history_summary_info():3461] rendering summary | |||
2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_sync_info():3387] logging synced files | |||
2023-03-09 23:04:23,211 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: shutdown | |||
2023-03-09 23:04:23,212 INFO HandlerThread:5402 [handler.py:finish():842] shutting down handler | |||
2023-03-09 23:04:24,208 INFO WriterThread:5402 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb | |||
2023-03-09 23:04:24,209 INFO SenderThread:5402 [sender.py:finish():1504] shutting down sender | |||
2023-03-09 23:04:24,210 INFO SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:04:24,210 INFO SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher |
@@ -0,0 +1,29 @@ | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Configure stats pid to 3604 | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:04:02,242 INFO MainThread:3604 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:04:02,245 INFO MainThread:3604 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:04:02,249 INFO MainThread:3604 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:04:02,251 INFO MainThread:3604 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:04:03,555 INFO MainThread:3604 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:04:08,608 INFO MainThread:3604 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:04:08,608 INFO MainThread:3604 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:04:09,563 INFO MainThread:3604 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:04:09,564 INFO MainThread:3604 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} | |||
2023-03-09 23:04:24,411 WARNING MsgRouterThr:3604 [router.py:message_loop():77] message_loop has been closed |
@@ -0,0 +1,40 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678374396.698895 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,14 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Device: cuda | |||
Images scaling: 0.5 | |||
Traceback (most recent call last): | |||
File "train.py", line 169, in <module> | |||
img_scale=args.scale, | |||
File "train.py", line 70, in train_model | |||
optimizer = ms.nn.RMSprop(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay) | |||
AttributeError: module 'mindspore.nn' has no attribute 'RMSprop' |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:06:43.530245", | |||
"startedAt": "2023-03-09T15:06:36.660035", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2499.997949999998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.992, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.979, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.992, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.988, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.985, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.987, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.992, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.981, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.985, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.992, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.984, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.012, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.992, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.006, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.008, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.006, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.996, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.981, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.987, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.049, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.006, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.984, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.063, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.004, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.014, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.002, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6544303894043 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{"_wandb": {"runtime": 1}} |
@@ -0,0 +1,177 @@ | |||
2023-03-09 23:06:36,705 INFO StreamThr :7419 [internal.py:wandb_internal():90] W&B internal server running at pid: 7419, started at: 2023-03-09 23:06:36.703095 | |||
2023-03-09 23:06:36,706 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:06:36,708 INFO WriterThread:7419 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb | |||
2023-03-09 23:06:36,712 DEBUG SenderThread:7419 [sender.py:send():336] send: header | |||
2023-03-09 23:06:36,713 DEBUG SenderThread:7419 [sender.py:send():336] send: run | |||
2023-03-09 23:06:36,804 INFO SenderThread:7419 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/3p0zeu2z | |||
2023-03-09 23:06:38,422 INFO SenderThread:7419 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files | |||
2023-03-09 23:06:38,422 INFO SenderThread:7419 [sender.py:_start_run_threads():1081] run started: 3p0zeu2z with start time 1678374396.698895 | |||
2023-03-09 23:06:38,423 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:06:38,426 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:06:38,444 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:06:38,444 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:06:39,426 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json | |||
2023-03-09 23:06:43,427 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:06:43,446 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:06:43,504 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:06:43,509 DEBUG HandlerThread:7419 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:06:43,509 DEBUG HandlerThread:7419 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:06:43,509 INFO HandlerThread:7419 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:06:43,510 INFO SystemMonitor:7419 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:06:43,510 INFO HandlerThread:7419 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:06:43,510 INFO SystemMonitor:7419 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:06:43,511 INFO SystemMonitor:7419 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:06:43,511 INFO SystemMonitor:7419 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:06:43,512 INFO SystemMonitor:7419 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:06:43,512 INFO SystemMonitor:7419 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:06:43,530 DEBUG HandlerThread:7419 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:06:43.530245', 'startedAt': '2023-03-09T15:06:36.660035', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2499.997949999998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.979, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.049, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.014, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6544303894043}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:06:43,549 INFO HandlerThread:7419 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:06:43,549 INFO HandlerThread:7419 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:06:43,576 DEBUG HandlerThread:7419 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:06:43,577 DEBUG HandlerThread:7419 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:06:44,293 DEBUG HandlerThread:7419 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:06:44,346 INFO HandlerThread:7419 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:06:44,349 DEBUG SenderThread:7419 [sender.py:send():336] send: files | |||
2023-03-09 23:06:44,349 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:06:44,368 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:06:44,384 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml | |||
2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json | |||
2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt | |||
2023-03-09 23:06:44,866 DEBUG SenderThread:7419 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:06:44,871 DEBUG SenderThread:7419 [sender.py:send():336] send: config | |||
2023-03-09 23:06:44,914 DEBUG SenderThread:7419 [sender.py:send():336] send: exit | |||
2023-03-09 23:06:44,920 INFO SenderThread:7419 [sender.py:send_exit():559] handling exit code: 1 | |||
2023-03-09 23:06:44,920 INFO SenderThread:7419 [sender.py:send_exit():561] handling runtime: 1 | |||
2023-03-09 23:06:45,056 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:06:45,067 INFO SenderThread:7419 [sender.py:send_exit():567] send defer | |||
2023-03-09 23:06:45,068 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,069 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 0 | |||
2023-03-09 23:06:45,069 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,069 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 0 | |||
2023-03-09 23:06:45,069 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 1 | |||
2023-03-09 23:06:45,069 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,069 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 1 | |||
2023-03-09 23:06:45,070 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,070 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 1 | |||
2023-03-09 23:06:45,070 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 2 | |||
2023-03-09 23:06:45,070 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,070 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 2 | |||
2023-03-09 23:06:45,070 INFO HandlerThread:7419 [system_monitor.py:finish():193] Stopping system monitor | |||
2023-03-09 23:06:45,070 DEBUG SystemMonitor:7419 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:06:45,071 DEBUG SystemMonitor:7419 [system_monitor.py:_start():168] Finished system metrics aggregation loop | |||
2023-03-09 23:06:45,071 DEBUG SystemMonitor:7419 [system_monitor.py:_start():172] Publishing last batch of metrics | |||
2023-03-09 23:06:45,072 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined cpu monitor | |||
2023-03-09 23:06:45,072 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined disk monitor | |||
2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined gpu monitor | |||
2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined memory monitor | |||
2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined network monitor | |||
2023-03-09 23:06:45,087 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,087 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 2 | |||
2023-03-09 23:06:45,087 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 3 | |||
2023-03-09 23:06:45,087 DEBUG SenderThread:7419 [sender.py:send():336] send: stats | |||
2023-03-09 23:06:45,087 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,087 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 3 | |||
2023-03-09 23:06:45,088 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,088 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 3 | |||
2023-03-09 23:06:45,088 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 4 | |||
2023-03-09 23:06:45,088 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,088 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 4 | |||
2023-03-09 23:06:45,089 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,089 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 4 | |||
2023-03-09 23:06:45,089 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 5 | |||
2023-03-09 23:06:45,089 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,089 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 5 | |||
2023-03-09 23:06:45,089 DEBUG SenderThread:7419 [sender.py:send():336] send: summary | |||
2023-03-09 23:06:45,107 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:06:45,108 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 5 | |||
2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 6 | |||
2023-03-09 23:06:45,108 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,108 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 6 | |||
2023-03-09 23:06:45,108 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 6 | |||
2023-03-09 23:06:45,114 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:06:45,573 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 7 | |||
2023-03-09 23:06:45,573 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:45,573 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 7 | |||
2023-03-09 23:06:45,574 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:45,574 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 7 | |||
2023-03-09 23:06:45,591 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json | |||
2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml | |||
2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log | |||
2023-03-09 23:06:45,608 INFO wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /tmp/tmph6vgygn9wandb/2ibjar1l-wandb-metadata.json | |||
2023-03-09 23:06:46,926 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 8 | |||
2023-03-09 23:06:46,927 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:06:46,927 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:46,927 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 8 | |||
2023-03-09 23:06:46,928 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:46,928 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 8 | |||
2023-03-09 23:06:46,928 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 9 | |||
2023-03-09 23:06:46,928 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:46,928 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 9 | |||
2023-03-09 23:06:46,928 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:46,929 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 9 | |||
2023-03-09 23:06:46,929 INFO SenderThread:7419 [dir_watcher.py:finish():365] shutting down directory watcher | |||
2023-03-09 23:06:47,597 INFO SenderThread:7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log | |||
2023-03-09 23:06:47,597 INFO SenderThread:7419 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files | |||
2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml conda-environment.yaml | |||
2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log output.log | |||
2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt requirements.txt | |||
2023-03-09 23:06:47,603 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml config.yaml | |||
2023-03-09 23:06:47,629 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json wandb-summary.json | |||
2023-03-09 23:06:47,720 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json wandb-metadata.json | |||
2023-03-09 23:06:47,720 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 10 | |||
2023-03-09 23:06:47,857 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:47,858 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 10 | |||
2023-03-09 23:06:47,891 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:47,896 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 10 | |||
2023-03-09 23:06:47,896 INFO SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:06:48,926 INFO wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log | |||
2023-03-09 23:06:49,240 INFO wandb-upload_3:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml | |||
2023-03-09 23:06:49,315 INFO wandb-upload_4:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json | |||
2023-03-09 23:06:50,115 INFO wandb-upload_2:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt | |||
2023-03-09 23:06:50,316 INFO Thread-12 :7419 [sender.py:transition_state():587] send defer: 11 | |||
2023-03-09 23:06:50,316 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:50,316 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 11 | |||
2023-03-09 23:06:50,317 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:50,317 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 11 | |||
2023-03-09 23:06:50,317 INFO SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher | |||
2023-03-09 23:06:50,317 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 12 | |||
2023-03-09 23:06:50,317 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:50,317 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 12 | |||
2023-03-09 23:06:50,317 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:50,318 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 12 | |||
2023-03-09 23:06:50,626 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:06:50,662 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 13 | |||
2023-03-09 23:06:50,662 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:50,662 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 13 | |||
2023-03-09 23:06:50,662 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:50,663 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 13 | |||
2023-03-09 23:06:50,663 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 14 | |||
2023-03-09 23:06:50,663 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:06:50,663 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 14 | |||
2023-03-09 23:06:50,663 DEBUG SenderThread:7419 [sender.py:send():336] send: final | |||
2023-03-09 23:06:50,664 DEBUG SenderThread:7419 [sender.py:send():336] send: footer | |||
2023-03-09 23:06:50,664 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:06:50,664 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 14 | |||
2023-03-09 23:06:50,664 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:06:50,665 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:06:50,665 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: server_info | |||
2023-03-09 23:06:50,666 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: get_summary | |||
2023-03-09 23:06:50,666 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: server_info | |||
2023-03-09 23:06:50,717 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: sampled_history | |||
2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_history_summary_info():3429] rendering history | |||
2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_history_summary_info():3461] rendering summary | |||
2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_sync_info():3387] logging synced files | |||
2023-03-09 23:06:51,504 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: shutdown | |||
2023-03-09 23:06:51,504 INFO HandlerThread:7419 [handler.py:finish():842] shutting down handler | |||
2023-03-09 23:06:51,666 INFO WriterThread:7419 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb | |||
2023-03-09 23:06:52,503 INFO SenderThread:7419 [sender.py:finish():1504] shutting down sender | |||
2023-03-09 23:06:52,503 INFO SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:06:52,503 INFO SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher |
@@ -0,0 +1,28 @@ | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Configure stats pid to 5620 | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log | |||
2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log | |||
2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:06:36,697 INFO MainThread:5620 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:06:36,698 INFO MainThread:5620 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:06:36,703 INFO MainThread:5620 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:06:36,704 INFO MainThread:5620 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:06:38,425 INFO MainThread:5620 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:06:43,446 INFO MainThread:5620 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:06:43,447 INFO MainThread:5620 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:06:44,356 INFO MainThread:5620 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:06:44,356 INFO MainThread:5620 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:06:44,357 INFO MainThread:5620 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:06:44,357 INFO MainThread:5620 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:06:44,358 INFO MainThread:5620 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:06:44,358 INFO MainThread:5620 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} | |||
2023-03-09 23:06:52,524 WARNING MsgRouterThr:5620 [router.py:message_loop():77] message_loop has been closed |
@@ -0,0 +1,40 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678374643.540464 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,14 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Device: cuda | |||
Images scaling: 0.5 | |||
Traceback (most recent call last): | |||
File "train.py", line 169, in <module> | |||
img_scale=args.scale, | |||
File "train.py", line 88, in train_model | |||
grad_fn = ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True) | |||
NameError: name 'ops' is not defined |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:10:50.243518", | |||
"startedAt": "2023-03-09T15:10:43.500810", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.419575, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.322, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.664, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.917, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.586, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.207, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.024, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.054, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.294, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2497.793, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.641, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.912, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.087, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.927, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.004, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.109, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.008, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.666, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.422, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.34, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.446, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.667, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.071, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.954, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.346, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.336, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.318, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.379, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.717, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.668, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.274, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.083, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.294, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.426, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.706, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.578, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.8, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.034, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.781, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.461, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.789, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.345, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.362, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.448, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.053, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.154, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.213, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.574, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.754, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.345, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.409, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.953, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.981, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.045, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.47, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.703, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.13, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.197, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.917, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.623, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.272, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.349, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.464, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.376, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.544, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.74, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.053, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6549377441406 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{"_wandb": {"runtime": 1}} |
@@ -0,0 +1,183 @@ | |||
2023-03-09 23:10:43,546 INFO StreamThr :9187 [internal.py:wandb_internal():90] W&B internal server running at pid: 9187, started at: 2023-03-09 23:10:43.544652 | |||
2023-03-09 23:10:43,548 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:10:43,549 INFO WriterThread:9187 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb | |||
2023-03-09 23:10:43,554 DEBUG SenderThread:9187 [sender.py:send():336] send: header | |||
2023-03-09 23:10:43,554 DEBUG SenderThread:9187 [sender.py:send():336] send: run | |||
2023-03-09 23:10:43,617 INFO SenderThread:9187 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/osho06z5 | |||
2023-03-09 23:10:45,099 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:10:45,100 INFO SenderThread:9187 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files | |||
2023-03-09 23:10:45,100 INFO SenderThread:9187 [sender.py:_start_run_threads():1081] run started: osho06z5 with start time 1678374643.540464 | |||
2023-03-09 23:10:45,100 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:10:45,131 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:10:45,132 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:10:46,104 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json | |||
2023-03-09 23:10:50,100 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:10:50,133 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:10:50,192 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:10:50,205 DEBUG HandlerThread:9187 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:10:50,205 DEBUG HandlerThread:9187 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:10:50,205 INFO HandlerThread:9187 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:10:50,206 INFO SystemMonitor:9187 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:10:50,206 INFO HandlerThread:9187 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:10:50,206 INFO SystemMonitor:9187 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:10:50,207 INFO SystemMonitor:9187 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:10:50,207 INFO SystemMonitor:9187 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:10:50,208 INFO SystemMonitor:9187 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:10:50,208 INFO SystemMonitor:9187 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:10:50,243 DEBUG HandlerThread:9187 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:10:50.243518', 'startedAt': '2023-03-09T15:10:43.500810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.419575, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.664, 'min': 0.0, 'max': 0.0}, {'current': 2499.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.586, 'min': 0.0, 'max': 0.0}, {'current': 2500.207, 'min': 0.0, 'max': 0.0}, {'current': 2500.024, 'min': 0.0, 'max': 0.0}, {'current': 2500.054, 'min': 0.0, 'max': 0.0}, {'current': 2501.294, 'min': 0.0, 'max': 0.0}, {'current': 2497.793, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.641, 'min': 0.0, 'max': 0.0}, {'current': 2503.912, 'min': 0.0, 'max': 0.0}, {'current': 2502.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2502.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2503.109, 'min': 0.0, 'max': 0.0}, {'current': 2501.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.666, 'min': 0.0, 'max': 0.0}, {'current': 2501.422, 'min': 0.0, 'max': 0.0}, {'current': 2500.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.446, 'min': 0.0, 'max': 0.0}, {'current': 2499.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.071, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.954, 'min': 0.0, 'max': 0.0}, {'current': 2500.346, 'min': 0.0, 'max': 0.0}, {'current': 2500.336, 'min': 0.0, 'max': 0.0}, {'current': 2501.318, 'min': 0.0, 'max': 0.0}, {'current': 2500.379, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.668, 'min': 0.0, 'max': 0.0}, {'current': 2499.274, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.083, 'min': 0.0, 'max': 0.0}, {'current': 2499.294, 'min': 0.0, 'max': 0.0}, {'current': 2505.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.706, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.578, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.8, 'min': 0.0, 'max': 0.0}, {'current': 2499.034, 'min': 0.0, 'max': 0.0}, {'current': 2504.781, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2500.789, 'min': 0.0, 'max': 0.0}, {'current': 2501.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.448, 'min': 0.0, 'max': 0.0}, {'current': 2499.053, 'min': 0.0, 'max': 0.0}, {'current': 2503.154, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.213, 'min': 0.0, 'max': 0.0}, {'current': 2504.574, 'min': 0.0, 'max': 0.0}, {'current': 2506.754, 'min': 0.0, 'max': 0.0}, {'current': 2500.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.409, 'min': 0.0, 'max': 0.0}, {'current': 2504.953, 'min': 0.0, 'max': 0.0}, {'current': 2501.981, 'min': 0.0, 'max': 0.0}, {'current': 2500.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.47, 'min': 0.0, 'max': 0.0}, {'current': 2500.703, 'min': 0.0, 'max': 0.0}, {'current': 2503.13, 'min': 0.0, 'max': 0.0}, {'current': 2500.197, 'min': 0.0, 'max': 0.0}, {'current': 2505.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.99, 'min': 0.0, 'max': 0.0}, {'current': 2503.272, 'min': 0.0, 'max': 0.0}, {'current': 2502.349, 'min': 0.0, 'max': 0.0}, {'current': 2504.464, 'min': 0.0, 'max': 0.0}, {'current': 2503.376, 'min': 0.0, 'max': 0.0}, {'current': 2503.997, 'min': 0.0, 'max': 0.0}, {'current': 2505.544, 'min': 0.0, 'max': 0.0}, {'current': 2505.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6549377441406}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:10:50,267 INFO HandlerThread:9187 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:10:50,267 INFO HandlerThread:9187 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:10:50,267 DEBUG HandlerThread:9187 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:10:50,309 DEBUG HandlerThread:9187 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:10:50,309 DEBUG HandlerThread:9187 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:10:51,019 DEBUG HandlerThread:9187 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:10:51,065 INFO HandlerThread:9187 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:10:51,068 DEBUG SenderThread:9187 [sender.py:send():336] send: files | |||
2023-03-09 23:10:51,068 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:10:51,084 DEBUG SenderThread:9187 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:10:51,084 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:10:51,094 DEBUG SenderThread:9187 [sender.py:send():336] send: config | |||
2023-03-09 23:10:51,130 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:10:51,233 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json | |||
2023-03-09 23:10:51,262 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml | |||
2023-03-09 23:10:51,268 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt | |||
2023-03-09 23:10:51,721 DEBUG SenderThread:9187 [sender.py:send():336] send: exit | |||
2023-03-09 23:10:51,721 INFO SenderThread:9187 [sender.py:send_exit():559] handling exit code: 1 | |||
2023-03-09 23:10:51,721 INFO SenderThread:9187 [sender.py:send_exit():561] handling runtime: 1 | |||
2023-03-09 23:10:51,797 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:10:51,808 INFO SenderThread:9187 [sender.py:send_exit():567] send defer | |||
2023-03-09 23:10:51,828 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,832 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 0 | |||
2023-03-09 23:10:51,833 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,833 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 0 | |||
2023-03-09 23:10:51,833 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 1 | |||
2023-03-09 23:10:51,833 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,833 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 1 | |||
2023-03-09 23:10:51,834 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,834 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 1 | |||
2023-03-09 23:10:51,834 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 2 | |||
2023-03-09 23:10:51,834 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,834 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 2 | |||
2023-03-09 23:10:51,834 INFO HandlerThread:9187 [system_monitor.py:finish():193] Stopping system monitor | |||
2023-03-09 23:10:51,834 DEBUG SystemMonitor:9187 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:10:51,835 DEBUG SystemMonitor:9187 [system_monitor.py:_start():168] Finished system metrics aggregation loop | |||
2023-03-09 23:10:51,835 DEBUG SystemMonitor:9187 [system_monitor.py:_start():172] Publishing last batch of metrics | |||
2023-03-09 23:10:51,837 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined cpu monitor | |||
2023-03-09 23:10:51,838 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined disk monitor | |||
2023-03-09 23:10:51,850 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined gpu monitor | |||
2023-03-09 23:10:51,851 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined memory monitor | |||
2023-03-09 23:10:51,851 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined network monitor | |||
2023-03-09 23:10:51,852 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,852 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 2 | |||
2023-03-09 23:10:51,852 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 3 | |||
2023-03-09 23:10:51,852 DEBUG SenderThread:9187 [sender.py:send():336] send: stats | |||
2023-03-09 23:10:51,853 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,853 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 3 | |||
2023-03-09 23:10:51,853 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,853 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 3 | |||
2023-03-09 23:10:51,853 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 4 | |||
2023-03-09 23:10:51,854 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,854 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 4 | |||
2023-03-09 23:10:51,854 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,854 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 4 | |||
2023-03-09 23:10:51,854 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 5 | |||
2023-03-09 23:10:51,854 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,854 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 5 | |||
2023-03-09 23:10:51,854 DEBUG SenderThread:9187 [sender.py:send():336] send: summary | |||
2023-03-09 23:10:51,873 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:10:51,874 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 5 | |||
2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 6 | |||
2023-03-09 23:10:51,874 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:51,874 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 6 | |||
2023-03-09 23:10:51,874 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 6 | |||
2023-03-09 23:10:51,879 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:10:52,221 INFO Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json | |||
2023-03-09 23:10:52,221 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log | |||
2023-03-09 23:10:52,323 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 7 | |||
2023-03-09 23:10:52,323 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:52,323 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 7 | |||
2023-03-09 23:10:52,324 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:52,324 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 7 | |||
2023-03-09 23:10:52,324 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:10:52,402 INFO wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /tmp/tmpfner7v0awandb/mcayg890-wandb-metadata.json | |||
2023-03-09 23:10:53,222 INFO Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml | |||
2023-03-09 23:10:53,747 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 8 | |||
2023-03-09 23:10:53,753 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:10:53,753 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:53,758 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 8 | |||
2023-03-09 23:10:53,789 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:53,801 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 8 | |||
2023-03-09 23:10:53,827 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 9 | |||
2023-03-09 23:10:53,840 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:53,840 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 9 | |||
2023-03-09 23:10:53,840 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:53,840 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 9 | |||
2023-03-09 23:10:53,840 INFO SenderThread:9187 [dir_watcher.py:finish():365] shutting down directory watcher | |||
2023-03-09 23:10:54,228 INFO SenderThread:9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log | |||
2023-03-09 23:10:54,228 INFO SenderThread:9187 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files | |||
2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml conda-environment.yaml | |||
2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log output.log | |||
2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt requirements.txt | |||
2023-03-09 23:10:54,235 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml config.yaml | |||
2023-03-09 23:10:54,255 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json wandb-summary.json | |||
2023-03-09 23:10:54,304 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json wandb-metadata.json | |||
2023-03-09 23:10:54,304 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 10 | |||
2023-03-09 23:10:54,523 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:10:54,529 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 10 | |||
2023-03-09 23:10:54,542 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:10:54,542 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 10 | |||
2023-03-09 23:10:54,542 INFO SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:10:55,533 INFO wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt | |||
2023-03-09 23:10:55,794 INFO wandb-upload_3:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml | |||
2023-03-09 23:10:55,904 INFO wandb-upload_1:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log | |||
2023-03-09 23:10:57,354 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():1997] upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb81c30ea10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) | |||
2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():1999] upload_file request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '26'} | |||
2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():2001] upload_file response body: | |||
2023-03-09 23:11:00,543 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:11:02,356 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:11:03,317 INFO wandb-upload_4:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json | |||
2023-03-09 23:11:03,517 INFO Thread-12 :9187 [sender.py:transition_state():587] send defer: 11 | |||
2023-03-09 23:11:03,518 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:11:03,518 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 11 | |||
2023-03-09 23:11:03,518 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:11:03,518 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 11 | |||
2023-03-09 23:11:03,518 INFO SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher | |||
2023-03-09 23:11:03,518 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 12 | |||
2023-03-09 23:11:03,519 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:11:03,519 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 12 | |||
2023-03-09 23:11:03,519 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:11:03,519 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 12 | |||
2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 13 | |||
2023-03-09 23:11:04,731 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:11:04,731 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 13 | |||
2023-03-09 23:11:04,731 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 13 | |||
2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 14 | |||
2023-03-09 23:11:04,732 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer | |||
2023-03-09 23:11:04,732 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 14 | |||
2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send():336] send: final | |||
2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send():336] send: footer | |||
2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer | |||
2023-03-09 23:11:04,732 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 14 | |||
2023-03-09 23:11:04,733 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit | |||
2023-03-09 23:11:04,733 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit | |||
2023-03-09 23:11:04,734 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: server_info | |||
2023-03-09 23:11:04,734 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: server_info | |||
2023-03-09 23:11:04,770 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: get_summary | |||
2023-03-09 23:11:04,800 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: sampled_history | |||
2023-03-09 23:11:05,697 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:11:05,697 INFO MainThread:9187 [wandb_run.py:_footer_history_summary_info():3429] rendering history | |||
2023-03-09 23:11:05,697 INFO MainThread:9187 [wandb_run.py:_footer_history_summary_info():3461] rendering summary | |||
2023-03-09 23:11:05,698 INFO MainThread:9187 [wandb_run.py:_footer_sync_info():3387] logging synced files | |||
2023-03-09 23:11:05,698 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: shutdown | |||
2023-03-09 23:11:05,698 INFO HandlerThread:9187 [handler.py:finish():842] shutting down handler | |||
2023-03-09 23:11:06,697 INFO SenderThread:9187 [sender.py:finish():1504] shutting down sender | |||
2023-03-09 23:11:06,697 INFO SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher | |||
2023-03-09 23:11:06,697 INFO SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher | |||
2023-03-09 23:11:06,697 INFO WriterThread:9187 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb |
@@ -0,0 +1,28 @@ | |||
2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Configure stats pid to 7468 | |||
2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:10:43,538 INFO MainThread:7468 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:10:43,540 INFO MainThread:7468 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:10:43,544 INFO MainThread:7468 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:10:43,545 INFO MainThread:7468 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:10:45,099 INFO MainThread:7468 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:10:50,134 INFO MainThread:7468 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:10:50,134 INFO MainThread:7468 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:10:51,076 INFO MainThread:7468 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:10:51,076 INFO MainThread:7468 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} | |||
2023-03-09 23:11:06,747 WARNING MsgRouterThr:7468 [router.py:message_loop():77] message_loop has been closed |
@@ -0,0 +1,40 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678374750.437312 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,8 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Device: cuda | |||
Images scaling: 0.5 |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:12:37.409031", | |||
"startedAt": "2023-03-09T15:12:30.397388", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.8709499999995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2503.176, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.665, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.337, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.575, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.932, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.175, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.232, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.76, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.386, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.645, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.742, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.602, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.547, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.785, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.892, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.913, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.64, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.636, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.065, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.447, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.937, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.397, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.317, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.778, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.691, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.442, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.403, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.367, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.909, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.332, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.151, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.712, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.725, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.245, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.616, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.382, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.774, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2509.295, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.964, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.445, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.114, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.765, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.667, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.401, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.018, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.418, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.384, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.864, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.165, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.535, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.818, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.705, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.344, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.589, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.395, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.818, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.757, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.398, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.645, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.011, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.031, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.391, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.45, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.348, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.345, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.644, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.027, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.544, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.622, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.011, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.421, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6551322937012 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,155 @@ | |||
2023-03-09 23:12:30,443 INFO StreamThr :11017 [internal.py:wandb_internal():90] W&B internal server running at pid: 11017, started at: 2023-03-09 23:12:30.441643 | |||
2023-03-09 23:12:30,445 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:12:30,446 INFO WriterThread:11017 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/run-qbg0lcfa.wandb | |||
2023-03-09 23:12:30,452 DEBUG SenderThread:11017 [sender.py:send():336] send: header | |||
2023-03-09 23:12:30,452 DEBUG SenderThread:11017 [sender.py:send():336] send: run | |||
2023-03-09 23:12:30,506 INFO SenderThread:11017 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/qbg0lcfa | |||
2023-03-09 23:12:32,285 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:12:32,285 INFO SenderThread:11017 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files | |||
2023-03-09 23:12:32,285 INFO SenderThread:11017 [sender.py:_start_run_threads():1081] run started: qbg0lcfa with start time 1678374750.437312 | |||
2023-03-09 23:12:32,285 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:12:32,305 INFO SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:12:32,305 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:12:33,289 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json | |||
2023-03-09 23:12:37,286 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:12:37,306 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:12:37,368 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:12:37,373 DEBUG HandlerThread:11017 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:12:37,373 DEBUG HandlerThread:11017 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:12:37,373 INFO HandlerThread:11017 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:12:37,374 INFO SystemMonitor:11017 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:12:37,374 INFO HandlerThread:11017 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:12:37,374 INFO SystemMonitor:11017 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:12:37,375 INFO SystemMonitor:11017 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:12:37,375 INFO SystemMonitor:11017 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:12:37,376 INFO SystemMonitor:11017 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:12:37,376 INFO SystemMonitor:11017 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:12:37,408 DEBUG HandlerThread:11017 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:12:37,420 DEBUG HandlerThread:11017 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:12:37,420 DEBUG HandlerThread:11017 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:12:37,421 DEBUG HandlerThread:11017 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:12:37.409031', 'startedAt': '2023-03-09T15:12:30.397388', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.8709499999995, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2503.176, 'min': 0.0, 'max': 0.0}, {'current': 2499.665, 'min': 0.0, 'max': 0.0}, {'current': 2500.337, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2502.932, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.175, 'min': 0.0, 'max': 0.0}, {'current': 2499.232, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.76, 'min': 0.0, 'max': 0.0}, {'current': 2500.386, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.602, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.547, 'min': 0.0, 'max': 0.0}, {'current': 2498.785, 'min': 0.0, 'max': 0.0}, {'current': 2503.892, 'min': 0.0, 'max': 0.0}, {'current': 2499.913, 'min': 0.0, 'max': 0.0}, {'current': 2499.64, 'min': 0.0, 'max': 0.0}, {'current': 2500.636, 'min': 0.0, 'max': 0.0}, {'current': 2505.065, 'min': 0.0, 'max': 0.0}, {'current': 2498.447, 'min': 0.0, 'max': 0.0}, {'current': 2500.937, 'min': 0.0, 'max': 0.0}, {'current': 2505.397, 'min': 0.0, 'max': 0.0}, {'current': 2499.317, 'min': 0.0, 'max': 0.0}, {'current': 2500.778, 'min': 0.0, 'max': 0.0}, {'current': 2500.691, 'min': 0.0, 'max': 0.0}, {'current': 2501.442, 'min': 0.0, 'max': 0.0}, {'current': 2501.403, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.909, 'min': 0.0, 'max': 0.0}, {'current': 2502.332, 'min': 0.0, 'max': 0.0}, {'current': 2501.151, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.712, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2503.245, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2503.382, 'min': 0.0, 'max': 0.0}, {'current': 2504.774, 'min': 0.0, 'max': 0.0}, {'current': 2504.997, 'min': 0.0, 'max': 0.0}, {'current': 2509.295, 'min': 0.0, 'max': 0.0}, {'current': 2503.964, 'min': 0.0, 'max': 0.0}, {'current': 2500.445, 'min': 0.0, 'max': 0.0}, {'current': 2501.114, 'min': 0.0, 'max': 0.0}, {'current': 2499.765, 'min': 0.0, 'max': 0.0}, {'current': 2500.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2506.018, 'min': 0.0, 'max': 0.0}, {'current': 2500.418, 'min': 0.0, 'max': 0.0}, {'current': 2500.384, 'min': 0.0, 'max': 0.0}, {'current': 2505.864, 'min': 0.0, 'max': 0.0}, {'current': 2505.165, 'min': 0.0, 'max': 0.0}, {'current': 2507.535, 'min': 0.0, 'max': 0.0}, {'current': 2506.818, 'min': 0.0, 'max': 0.0}, {'current': 2504.705, 'min': 0.0, 'max': 0.0}, {'current': 2500.344, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.818, 'min': 0.0, 'max': 0.0}, {'current': 2503.757, 'min': 0.0, 'max': 0.0}, {'current': 2504.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.031, 'min': 0.0, 'max': 0.0}, {'current': 2500.391, 'min': 0.0, 'max': 0.0}, {'current': 2500.45, 'min': 0.0, 'max': 0.0}, {'current': 2505.348, 'min': 0.0, 'max': 0.0}, {'current': 2505.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.644, 'min': 0.0, 'max': 0.0}, {'current': 2507.027, 'min': 0.0, 'max': 0.0}, {'current': 2499.544, 'min': 0.0, 'max': 0.0}, {'current': 2500.622, 'min': 0.0, 'max': 0.0}, {'current': 2501.011, 'min': 0.0, 'max': 0.0}, {'current': 2501.421, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6551322937012}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:12:37,421 INFO HandlerThread:11017 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:12:37,421 INFO HandlerThread:11017 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:12:37,421 DEBUG HandlerThread:11017 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:12:37,486 DEBUG HandlerThread:11017 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:12:37,486 DEBUG HandlerThread:11017 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:12:38,197 DEBUG HandlerThread:11017 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:12:38,229 INFO HandlerThread:11017 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:12:38,232 DEBUG SenderThread:11017 [sender.py:send():336] send: files | |||
2023-03-09 23:12:38,232 INFO SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:12:38,248 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:12:38,268 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/conda-environment.yaml | |||
2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json | |||
2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt | |||
2023-03-09 23:12:38,720 DEBUG SenderThread:11017 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:12:38,721 DEBUG SenderThread:11017 [sender.py:send():336] send: config | |||
2023-03-09 23:12:39,475 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log | |||
2023-03-09 23:12:39,490 INFO wandb-upload_0:11017 [upload_job.py:push():138] Uploaded file /tmp/tmp03hehnd5wandb/0g7l9wyo-wandb-metadata.json | |||
2023-03-09 23:12:41,477 INFO Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log | |||
2023-03-09 23:12:42,727 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:12:47,727 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:12:52,728 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:12:53,241 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:12:53,242 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:12:57,729 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:12:58,243 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:02,735 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:03,257 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:03,497 INFO Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml | |||
2023-03-09 23:13:08,259 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:08,259 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:13,260 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:13,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:18,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:18,262 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:23,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:23,263 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:28,262 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:28,265 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:33,263 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:33,266 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:37,377 DEBUG SystemMonitor:11017 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:13:37,381 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:13:38,267 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:38,382 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:43,269 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:43,383 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:48,270 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:48,384 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:53,272 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:53,384 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:13:58,273 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:13:58,385 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:03,274 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:03,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:07,384 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:14:08,275 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:09,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:13,277 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:14,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:18,278 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:19,387 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:23,294 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:24,409 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:28,309 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:29,410 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:33,509 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:34,411 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:37,414 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:14:38,609 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:39,426 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:43,622 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:44,608 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:48,623 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:49,609 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:53,709 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:54,610 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:14:58,711 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:14:59,611 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:03,712 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:04,611 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:07,416 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:15:08,713 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:10,418 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:13,715 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:15,419 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:18,716 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:20,420 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:23,717 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:25,420 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:28,718 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:30,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:33,720 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:35,422 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:37,419 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:15:38,722 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:41,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:41,969 WARNING FileStreamThread:11017 [file_stream.py:request_with_retry():675] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream. func: <bound method Session.post of <requests.sessions.Session object at 0x7f3cd90d0c50>>, args: ('https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 4, 'content': ['{"system.disk": 34.0, "system.network.sent": 648043.93, "system.network.recv": 292258.87, "system.proc.memory.availableMB": 329995.35, "system.memory": 14.43, "system.proc.memory.rssMB": 22483.33, "system.proc.memory.percent": 5.83, "system.cpu": 0.01, "system.cpu.0.cpu_percent": 13.63, "system.cpu.1.cpu_percent": 14.26, "system.cpu.2.cpu_percent": 13.02, "system.cpu.3.cpu_percent": 11.75, "system.cpu.4.cpu_percent": 11.36, "system.cpu.5.cpu_percent": 13.88, "system.cpu.6.cpu_percent": 11.97, "system.cpu.7.cpu_percent": 11.55, "system.cpu.8.cpu_percent": 10.73, "system.cpu.9.cpu_percent": 11.03, "system.cpu.10.cpu_percent": 12.05, "system.cpu.11.cpu_percent": 12.08, "system.cpu.12.cpu_percent": 12.5, "system.cpu.13.cpu_percent": 10.96, "system.cpu.14.cpu_percent": 11.1, "system.cpu.15.cpu_percent": 11.49, "system.cpu.16.cpu_percent": 11.22, "system.cpu.17.cpu_percent": 11.7, "system.cpu.18.cpu_percent": 12.28, "system.cpu.19.cpu_percent": 10.81, "system.cpu.20.cpu_percent": 13.6, "system.cpu.21.cpu_percent": 13.91, "system.cpu.22.cpu_percent": 13.94, "system.cpu.23.cpu_percent": 11.47, "system.cpu.24.cpu_percent": 10.25, "system.cpu.25.cpu_percent": 12.91, "system.cpu.26.cpu_percent": 10.18, "system.cpu.27.cpu_percent": 10.71, "system.cpu.28.cpu_percent": 10.51, "system.cpu.29.cpu_percent": 9.33, "system.cpu.30.cpu_percent": 10.98, "system.cpu.31.cpu_percent": 10.93, "system.cpu.32.cpu_percent": 9.93, "system.cpu.33.cpu_percent": 10.65, "system.cpu.34.cpu_percent": 11.46, "system.cpu.35.cpu_percent": 9.93, "system.cpu.36.cpu_percent": 10.47, "system.cpu.37.cpu_percent": 10.19, "system.cpu.38.cpu_percent": 9.21, "system.cpu.39.cpu_percent": 10.19, "system.cpu.40.cpu_percent": 12.19, "system.cpu.41.cpu_percent": 12.61, "system.cpu.42.cpu_percent": 18.98, "system.cpu.43.cpu_percent": 11.38, "system.cpu.44.cpu_percent": 11.84, "system.cpu.45.cpu_percent": 11.58, "system.cpu.46.cpu_percent": 10.05, "system.cpu.47.cpu_percent": 11.54, "system.cpu.48.cpu_percent": 11.37, "system.cpu.49.cpu_percent": 12.76, "system.cpu.50.cpu_percent": 11.16, "system.cpu.51.cpu_percent": 11.69, "system.cpu.52.cpu_percent": 10.77, "system.cpu.53.cpu_percent": 10.6, "system.cpu.54.cpu_percent": 10.82, "system.cpu.55.cpu_percent": 11.67, "system.cpu.56.cpu_percent": 17.88, "system.cpu.57.cpu_percent": 14.37, "system.cpu.58.cpu_percent": 10.75, "system.cpu.59.cpu_percent": 11.79, "system.cpu.60.cpu_percent": 11.43, "system.cpu.61.cpu_percent": 10.05, "system.cpu.62.cpu_percent": 9.11, "system.cpu.63.cpu_percent": 9.93, "system.cpu.64.cpu_percent": 11.24, "system.cpu.65.cpu_percent": 16.09, "system.cpu.66.cpu_percent": 12.53, "system.cpu.67.cpu_percent": 17.75, "system.cpu.68.cpu_percent": 12.15, "system.cpu.69.cpu_percent": 10.11, "system.cpu.70.cpu_percent": 9.68, "system.cpu.71.cpu_percent": 10.84, "system.cpu.72.cpu_percent": 16.69, "system.cpu.73.cpu_percent": 10.49, "system.cpu.74.cpu_percent": 10.3, "system.cpu.75.cpu_percent": 12.14, "system.cpu.76.cpu_percent": 12.36, "system.cpu.77.cpu_percent": 9.46, "system.cpu.78.cpu_percent": 10.52, "system.cpu.79.cpu_percent": 11.16, "system.proc.cpu.threads": 88, "system.gpu.0.memoryAllocated": 32.36, "system.gpu.1.memoryAllocated": 0.02, "system.gpu.0.memory": 0.0, "system.gpu.1.memory": 0.0, "system.gpu.0.gpu": 0.0, "system.gpu.1.gpu": 0.0, "system.gpu.0.temp": 56.0, "system.gpu.1.temp": 39.0, "system.gpu.0.powerWatts": 29.95, "system.gpu.1.powerWatts": 9.91, "system.gpu.0.powerPercent": 42.78, "system.gpu.1.powerPercent": 14.18, "_wandb": true, "_timestamp": 1678374937.415186, "_runtime": 186.977874}']}}, 'dropped': 0}} | |||
2023-03-09 23:15:43,723 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:46,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:48,724 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:51,422 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:53,725 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:15:56,423 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:15:58,726 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:01,423 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:03,728 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:06,424 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:07,490 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:16:08,729 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:11,492 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:13,730 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:16,492 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:18,732 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:21,493 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:23,733 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:26,494 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:28,734 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:31,494 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:33,736 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:36,495 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:37,493 DEBUG SenderThread:11017 [sender.py:send():336] send: stats | |||
2023-03-09 23:16:38,737 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:16:42,508 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:16:45,728 INFO MainThread:11017 [internal.py:handle_exit():77] Internal process exited |
@@ -0,0 +1,27 @@ | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Configure stats pid to 9232 | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:12:30,435 INFO MainThread:9232 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:12:30,437 INFO MainThread:9232 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:12:30,440 INFO MainThread:9232 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:12:30,441 INFO MainThread:9232 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:12:32,284 INFO MainThread:9232 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:12:37,307 INFO MainThread:9232 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:12:37,307 INFO MainThread:9232 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:12:38,240 INFO MainThread:9232 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:12:38,242 INFO MainThread:9232 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:12:38,243 INFO MainThread:9232 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,40 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678375959.443294 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,8 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Device: cuda | |||
Images scaling: 0.5 |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:32:46.785082", | |||
"startedAt": "2023-03-09T15:32:39.401025", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.3775624999994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.614, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.081, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.009, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.73, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.934, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.158, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.351, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.211, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.412, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.243, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.648, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.74, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.191, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.465, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.182, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.405, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.933, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.646, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.576, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.25, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.654, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.366, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.578, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.947, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.823, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.483, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.788, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.937, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.939, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.089, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.405, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.625, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.759, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.618, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.609, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.887, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.558, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.769, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.432, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.978, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.278, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.876, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.435, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.423, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.059, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.103, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.258, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.683, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.349, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.848, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.132, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.036, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.927, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.827, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.608, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.265, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.142, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.334, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.609, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.275, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.014, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.764, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.409, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.183, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.052, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.383, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.035, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.466, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.834, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.654, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.935, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6485137939453 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,94 @@ | |||
2023-03-09 23:32:39,448 INFO StreamThr :21832 [internal.py:wandb_internal():90] W&B internal server running at pid: 21832, started at: 2023-03-09 23:32:39.446798 | |||
2023-03-09 23:32:39,450 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:32:39,451 INFO WriterThread:21832 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/run-58pb6znt.wandb | |||
2023-03-09 23:32:39,456 DEBUG SenderThread:21832 [sender.py:send():336] send: header | |||
2023-03-09 23:32:39,456 DEBUG SenderThread:21832 [sender.py:send():336] send: run | |||
2023-03-09 23:32:39,536 INFO SenderThread:21832 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/58pb6znt | |||
2023-03-09 23:32:41,675 INFO SenderThread:21832 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files | |||
2023-03-09 23:32:41,675 INFO SenderThread:21832 [sender.py:_start_run_threads():1081] run started: 58pb6znt with start time 1678375959.443294 | |||
2023-03-09 23:32:41,676 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:32:41,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:32:41,715 INFO SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:32:41,716 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:32:42,679 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json | |||
2023-03-09 23:32:46,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:32:46,717 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:32:46,737 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:32:46,742 DEBUG HandlerThread:21832 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:32:46,742 DEBUG HandlerThread:21832 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:32:46,742 INFO HandlerThread:21832 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:32:46,742 INFO SystemMonitor:21832 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:32:46,742 INFO HandlerThread:21832 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:32:46,743 INFO SystemMonitor:21832 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:32:46,744 INFO SystemMonitor:21832 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:32:46,744 INFO SystemMonitor:21832 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:32:46,745 INFO SystemMonitor:21832 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:32:46,746 INFO SystemMonitor:21832 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:32:46,785 DEBUG HandlerThread:21832 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:32:46.785082', 'startedAt': '2023-03-09T15:32:39.401025', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.3775624999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.614, 'min': 0.0, 'max': 0.0}, {'current': 2501.081, 'min': 0.0, 'max': 0.0}, {'current': 2500.009, 'min': 0.0, 'max': 0.0}, {'current': 2500.73, 'min': 0.0, 'max': 0.0}, {'current': 2502.934, 'min': 0.0, 'max': 0.0}, {'current': 2500.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.351, 'min': 0.0, 'max': 0.0}, {'current': 2500.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.412, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2499.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.191, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.465, 'min': 0.0, 'max': 0.0}, {'current': 2500.182, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2501.933, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.646, 'min': 0.0, 'max': 0.0}, {'current': 2499.576, 'min': 0.0, 'max': 0.0}, {'current': 2499.25, 'min': 0.0, 'max': 0.0}, {'current': 2499.654, 'min': 0.0, 'max': 0.0}, {'current': 2500.366, 'min': 0.0, 'max': 0.0}, {'current': 2501.578, 'min': 0.0, 'max': 0.0}, {'current': 2499.947, 'min': 0.0, 'max': 0.0}, {'current': 2503.823, 'min': 0.0, 'max': 0.0}, {'current': 2501.483, 'min': 0.0, 'max': 0.0}, {'current': 2499.788, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2500.089, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2499.625, 'min': 0.0, 'max': 0.0}, {'current': 2504.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2500.887, 'min': 0.0, 'max': 0.0}, {'current': 2504.558, 'min': 0.0, 'max': 0.0}, {'current': 2500.769, 'min': 0.0, 'max': 0.0}, {'current': 2500.432, 'min': 0.0, 'max': 0.0}, {'current': 2504.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.278, 'min': 0.0, 'max': 0.0}, {'current': 2503.876, 'min': 0.0, 'max': 0.0}, {'current': 2500.435, 'min': 0.0, 'max': 0.0}, {'current': 2504.423, 'min': 0.0, 'max': 0.0}, {'current': 2500.059, 'min': 0.0, 'max': 0.0}, {'current': 2505.103, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2500.258, 'min': 0.0, 'max': 0.0}, {'current': 2503.683, 'min': 0.0, 'max': 0.0}, {'current': 2504.349, 'min': 0.0, 'max': 0.0}, {'current': 2499.848, 'min': 0.0, 'max': 0.0}, {'current': 2505.132, 'min': 0.0, 'max': 0.0}, {'current': 2505.036, 'min': 0.0, 'max': 0.0}, {'current': 2504.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2505.608, 'min': 0.0, 'max': 0.0}, {'current': 2501.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.142, 'min': 0.0, 'max': 0.0}, {'current': 2506.334, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2504.275, 'min': 0.0, 'max': 0.0}, {'current': 2501.014, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.409, 'min': 0.0, 'max': 0.0}, {'current': 2505.183, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2500.383, 'min': 0.0, 'max': 0.0}, {'current': 2503.035, 'min': 0.0, 'max': 0.0}, {'current': 2504.466, 'min': 0.0, 'max': 0.0}, {'current': 2503.834, 'min': 0.0, 'max': 0.0}, {'current': 2500.654, 'min': 0.0, 'max': 0.0}, {'current': 2503.935, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6485137939453}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:32:46,807 INFO HandlerThread:21832 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:32:46,807 INFO HandlerThread:21832 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:32:46,807 DEBUG HandlerThread:21832 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:32:46,861 DEBUG HandlerThread:21832 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:32:46,861 DEBUG HandlerThread:21832 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:32:47,586 DEBUG HandlerThread:21832 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:32:47,646 INFO HandlerThread:21832 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:32:47,649 DEBUG SenderThread:21832 [sender.py:send():336] send: files | |||
2023-03-09 23:32:47,649 INFO SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:32:47,666 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:32:47,666 DEBUG SenderThread:21832 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:32:47,682 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:32:47,868 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/conda-environment.yaml | |||
2023-03-09 23:32:47,878 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt | |||
2023-03-09 23:32:47,888 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json | |||
2023-03-09 23:32:48,228 DEBUG SenderThread:21832 [sender.py:send():336] send: config | |||
2023-03-09 23:32:48,916 INFO wandb-upload_0:21832 [upload_job.py:push():138] Uploaded file /tmp/tmpckj69pbswandb/vyd13lcr-wandb-metadata.json | |||
2023-03-09 23:32:52,258 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:32:57,259 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:02,260 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:02,657 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:02,657 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:07,260 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:07,659 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:12,266 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:12,660 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:13,878 INFO Thread-13 :21832 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml | |||
2023-03-09 23:33:17,661 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:18,617 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:22,662 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:23,618 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:27,663 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:28,619 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:32,665 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:33,619 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:37,666 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:38,620 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:42,667 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:43,621 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:46,746 DEBUG SystemMonitor:21832 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:33:46,751 DEBUG SenderThread:21832 [sender.py:send():336] send: stats | |||
2023-03-09 23:33:47,668 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:48,752 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:52,669 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:53,753 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:33:57,671 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:33:58,754 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:02,672 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:34:03,754 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:07,673 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:34:08,755 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:12,674 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:34:13,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:16,754 DEBUG SenderThread:21832 [sender.py:send():336] send: stats | |||
2023-03-09 23:34:17,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:34:19,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:22,677 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:34:24,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:28,873 ERROR memory :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) | |||
2023-03-09 23:34:28,873 ERROR memory :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) | |||
2023-03-09 23:34:28,915 ERROR cpu :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) | |||
2023-03-09 23:34:28,917 ERROR cpu :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) | |||
2023-03-09 23:34:29,757 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:34:29,823 INFO MainThread:21832 [internal.py:handle_exit():77] Internal process exited |
@@ -0,0 +1,27 @@ | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Configure stats pid to 20104 | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log | |||
2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log | |||
2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:32:39,440 INFO MainThread:20104 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:32:39,442 INFO MainThread:20104 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:32:39,447 INFO MainThread:20104 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:32:39,448 INFO MainThread:20104 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:32:41,675 INFO MainThread:20104 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:32:46,718 INFO MainThread:20104 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:32:46,718 INFO MainThread:20104 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:32:47,657 INFO MainThread:20104 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:32:47,658 INFO MainThread:20104 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,22 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: true | |||
python_version: 3.7.11 | |||
start_time: 1678376153.802776 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 2 | |||
- 5 |
@@ -0,0 +1,7 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Images scaling: 0.5 |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:36:00.197376", | |||
"startedAt": "2023-03-09T15:35:53.764072", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.5297500000006, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2500.096, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.615, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.893, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.591, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.521, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.243, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.638, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.144, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.868, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.649, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.109, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.04, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.822, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.83, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.476, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.859, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.572, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.283, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.012, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.618, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.178, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.045, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.773, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.834, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.236, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.061, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.96, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.082, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.189, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.323, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.69, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.152, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.988, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.041, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.322, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.397, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.099, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.921, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.55, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.546, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.646, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.091, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.889, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.029, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.008, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.061, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.807, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.396, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.189, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.804, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.96, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.432, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.638, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.016, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.045, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.726, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.212, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.968, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.027, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.15, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.153, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.227, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.381, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.1, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.225, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.803, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.722, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.718, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.523, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.426, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.206, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.864, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.673, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.64935302734375 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,51 @@ | |||
2023-03-09 23:35:53,808 INFO StreamThr :24054 [internal.py:wandb_internal():90] W&B internal server running at pid: 24054, started at: 2023-03-09 23:35:53.806812 | |||
2023-03-09 23:35:53,810 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:35:53,811 INFO WriterThread:24054 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/run-yc1qtvil.wandb | |||
2023-03-09 23:35:53,828 DEBUG SenderThread:24054 [sender.py:send():336] send: header | |||
2023-03-09 23:35:53,828 DEBUG SenderThread:24054 [sender.py:send():336] send: run | |||
2023-03-09 23:35:53,864 INFO SenderThread:24054 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/yc1qtvil | |||
2023-03-09 23:35:55,068 INFO SenderThread:24054 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files | |||
2023-03-09 23:35:55,069 INFO SenderThread:24054 [sender.py:_start_run_threads():1081] run started: yc1qtvil with start time 1678376153.802776 | |||
2023-03-09 23:35:55,069 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:35:55,069 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:35:55,122 INFO SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:35:55,123 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:35:56,073 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json | |||
2023-03-09 23:36:00,070 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:36:00,124 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:36:00,144 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:36:00,154 DEBUG HandlerThread:24054 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:36:00,154 DEBUG HandlerThread:24054 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:36:00,155 INFO HandlerThread:24054 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:36:00,155 INFO SystemMonitor:24054 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:36:00,155 INFO HandlerThread:24054 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:36:00,156 INFO SystemMonitor:24054 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:36:00,156 INFO SystemMonitor:24054 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:36:00,157 INFO SystemMonitor:24054 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:36:00,157 INFO SystemMonitor:24054 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:36:00,158 INFO SystemMonitor:24054 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:36:00,197 DEBUG HandlerThread:24054 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:36:00.197376', 'startedAt': '2023-03-09T15:35:53.764072', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5297500000006, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.096, 'min': 0.0, 'max': 0.0}, {'current': 2502.615, 'min': 0.0, 'max': 0.0}, {'current': 2500.893, 'min': 0.0, 'max': 0.0}, {'current': 2499.591, 'min': 0.0, 'max': 0.0}, {'current': 2499.521, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2499.638, 'min': 0.0, 'max': 0.0}, {'current': 2499.144, 'min': 0.0, 'max': 0.0}, {'current': 2501.868, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2501.109, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.822, 'min': 0.0, 'max': 0.0}, {'current': 2498.83, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.572, 'min': 0.0, 'max': 0.0}, {'current': 2499.283, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2501.178, 'min': 0.0, 'max': 0.0}, {'current': 2501.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.773, 'min': 0.0, 'max': 0.0}, {'current': 2501.834, 'min': 0.0, 'max': 0.0}, {'current': 2506.236, 'min': 0.0, 'max': 0.0}, {'current': 2501.061, 'min': 0.0, 'max': 0.0}, {'current': 2502.96, 'min': 0.0, 'max': 0.0}, {'current': 2501.082, 'min': 0.0, 'max': 0.0}, {'current': 2499.189, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.323, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2502.152, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.397, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2503.921, 'min': 0.0, 'max': 0.0}, {'current': 2500.55, 'min': 0.0, 'max': 0.0}, {'current': 2500.546, 'min': 0.0, 'max': 0.0}, {'current': 2505.646, 'min': 0.0, 'max': 0.0}, {'current': 2501.091, 'min': 0.0, 'max': 0.0}, {'current': 2499.889, 'min': 0.0, 'max': 0.0}, {'current': 2500.029, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2504.061, 'min': 0.0, 'max': 0.0}, {'current': 2505.807, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2500.189, 'min': 0.0, 'max': 0.0}, {'current': 2506.804, 'min': 0.0, 'max': 0.0}, {'current': 2500.96, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.432, 'min': 0.0, 'max': 0.0}, {'current': 2506.638, 'min': 0.0, 'max': 0.0}, {'current': 2507.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.726, 'min': 0.0, 'max': 0.0}, {'current': 2504.212, 'min': 0.0, 'max': 0.0}, {'current': 2499.968, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2504.15, 'min': 0.0, 'max': 0.0}, {'current': 2505.153, 'min': 0.0, 'max': 0.0}, {'current': 2499.227, 'min': 0.0, 'max': 0.0}, {'current': 2500.381, 'min': 0.0, 'max': 0.0}, {'current': 2504.1, 'min': 0.0, 'max': 0.0}, {'current': 2504.225, 'min': 0.0, 'max': 0.0}, {'current': 2500.803, 'min': 0.0, 'max': 0.0}, {'current': 2504.722, 'min': 0.0, 'max': 0.0}, {'current': 2504.718, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2500.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.206, 'min': 0.0, 'max': 0.0}, {'current': 2501.864, 'min': 0.0, 'max': 0.0}, {'current': 2504.673, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64935302734375}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:36:00,235 INFO HandlerThread:24054 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:36:00,235 INFO HandlerThread:24054 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:36:00,235 DEBUG HandlerThread:24054 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:36:00,274 DEBUG HandlerThread:24054 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:36:00,274 DEBUG HandlerThread:24054 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:36:00,993 DEBUG HandlerThread:24054 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:36:01,023 INFO HandlerThread:24054 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:36:01,027 DEBUG SenderThread:24054 [sender.py:send():336] send: files | |||
2023-03-09 23:36:01,027 INFO SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:36:01,036 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:36:01,049 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:36:01,270 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/conda-environment.yaml | |||
2023-03-09 23:36:01,271 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json | |||
2023-03-09 23:36:01,271 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt | |||
2023-03-09 23:36:01,524 DEBUG SenderThread:24054 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:36:01,524 DEBUG SenderThread:24054 [sender.py:send():336] send: config | |||
2023-03-09 23:36:02,264 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log | |||
2023-03-09 23:36:02,406 INFO wandb-upload_0:24054 [upload_job.py:push():138] Uploaded file /tmp/tmpzwt8zs5awandb/qutsputw-wandb-metadata.json | |||
2023-03-09 23:36:04,279 INFO Thread-13 :24054 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log | |||
2023-03-09 23:36:05,528 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:36:10,529 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report |
@@ -0,0 +1,27 @@ | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Configure stats pid to 22330 | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:35:53,800 INFO MainThread:22330 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:35:53,802 INFO MainThread:22330 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:35:53,806 INFO MainThread:22330 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:35:53,807 INFO MainThread:22330 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:35:55,067 INFO MainThread:22330 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:36:00,125 INFO MainThread:22330 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:36:00,125 INFO MainThread:22330 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:36:01,035 INFO MainThread:22330 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:36:01,036 INFO MainThread:22330 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,39 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: false | |||
python_version: 3.7.11 | |||
start_time: 1678376631.533235 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,7 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Images scaling: 0.5 |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:44:01.780995", | |||
"startedAt": "2023-03-09T15:43:51.492578", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "n1d6fa00775e48a18f80acc1355070be-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2501.7918, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2500.007, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.205, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.476, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.178, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.693, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.211, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.485, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.073, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.962, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.188, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.333, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.473, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.557, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.576, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.927, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.451, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.543, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.573, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.677, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.653, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.368, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.888, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.582, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.338, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.318, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.259, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.245, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.213, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.943, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.789, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.016, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.944, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.978, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.631, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.601, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.717, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.733, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.492, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.656, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.187, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.269, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.918, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.545, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.833, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.211, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.345, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.134, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.032, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.317, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.961, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.426, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.555, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.063, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.827, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.429, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.572, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.604, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.86, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.01, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.523, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.704, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.206, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.531, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.875, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.19, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.872, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.063, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.075, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.623, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.85, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.509, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.271, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6454200744629 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,114 @@ | |||
2023-03-09 23:43:51,539 INFO StreamThr :1879 [internal.py:wandb_internal():90] W&B internal server running at pid: 1879, started at: 2023-03-09 23:43:51.537061 | |||
2023-03-09 23:43:51,540 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:43:51,541 INFO WriterThread:1879 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/run-w9y25r7k.wandb | |||
2023-03-09 23:43:51,546 DEBUG SenderThread:1879 [sender.py:send():336] send: header | |||
2023-03-09 23:43:51,546 DEBUG SenderThread:1879 [sender.py:send():336] send: run | |||
2023-03-09 23:43:51,616 INFO SenderThread:1879 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/w9y25r7k | |||
2023-03-09 23:43:56,543 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:43:56,651 INFO SenderThread:1879 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files | |||
2023-03-09 23:43:56,651 INFO SenderThread:1879 [sender.py:_start_run_threads():1081] run started: w9y25r7k with start time 1678376631.533235 | |||
2023-03-09 23:43:56,652 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:43:56,652 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:43:56,654 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:43:56,675 INFO SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:43:56,676 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:43:57,655 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json | |||
2023-03-09 23:44:01,654 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:01,677 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:01,736 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:44:01,741 DEBUG HandlerThread:1879 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:44:01,741 DEBUG HandlerThread:1879 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:44:01,741 INFO HandlerThread:1879 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:44:01,741 INFO SystemMonitor:1879 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:44:01,742 INFO HandlerThread:1879 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:44:01,742 INFO SystemMonitor:1879 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:44:01,743 INFO SystemMonitor:1879 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:44:01,743 INFO SystemMonitor:1879 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:44:01,744 INFO SystemMonitor:1879 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:44:01,744 INFO SystemMonitor:1879 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:44:01,780 DEBUG HandlerThread:1879 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:44:01.780995', 'startedAt': '2023-03-09T15:43:51.492578', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.7918, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2503.205, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.178, 'min': 0.0, 'max': 0.0}, {'current': 2498.693, 'min': 0.0, 'max': 0.0}, {'current': 2499.211, 'min': 0.0, 'max': 0.0}, {'current': 2502.485, 'min': 0.0, 'max': 0.0}, {'current': 2500.073, 'min': 0.0, 'max': 0.0}, {'current': 2503.962, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2502.188, 'min': 0.0, 'max': 0.0}, {'current': 2503.333, 'min': 0.0, 'max': 0.0}, {'current': 2500.473, 'min': 0.0, 'max': 0.0}, {'current': 2499.557, 'min': 0.0, 'max': 0.0}, {'current': 2501.576, 'min': 0.0, 'max': 0.0}, {'current': 2501.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.451, 'min': 0.0, 'max': 0.0}, {'current': 2499.543, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2499.677, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2502.888, 'min': 0.0, 'max': 0.0}, {'current': 2503.582, 'min': 0.0, 'max': 0.0}, {'current': 2499.338, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.318, 'min': 0.0, 'max': 0.0}, {'current': 2501.259, 'min': 0.0, 'max': 0.0}, {'current': 2501.245, 'min': 0.0, 'max': 0.0}, {'current': 2501.213, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2499.789, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2499.944, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.631, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.601, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2505.733, 'min': 0.0, 'max': 0.0}, {'current': 2504.492, 'min': 0.0, 'max': 0.0}, {'current': 2505.656, 'min': 0.0, 'max': 0.0}, {'current': 2498.187, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2499.918, 'min': 0.0, 'max': 0.0}, {'current': 2504.545, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.833, 'min': 0.0, 'max': 0.0}, {'current': 2506.211, 'min': 0.0, 'max': 0.0}, {'current': 2506.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.032, 'min': 0.0, 'max': 0.0}, {'current': 2507.317, 'min': 0.0, 'max': 0.0}, {'current': 2505.961, 'min': 0.0, 'max': 0.0}, {'current': 2506.426, 'min': 0.0, 'max': 0.0}, {'current': 2498.555, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2503.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.429, 'min': 0.0, 'max': 0.0}, {'current': 2502.572, 'min': 0.0, 'max': 0.0}, {'current': 2503.604, 'min': 0.0, 'max': 0.0}, {'current': 2504.86, 'min': 0.0, 'max': 0.0}, {'current': 2500.01, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2503.704, 'min': 0.0, 'max': 0.0}, {'current': 2503.206, 'min': 0.0, 'max': 0.0}, {'current': 2500.531, 'min': 0.0, 'max': 0.0}, {'current': 2502.875, 'min': 0.0, 'max': 0.0}, {'current': 2500.19, 'min': 0.0, 'max': 0.0}, {'current': 2500.872, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2505.075, 'min': 0.0, 'max': 0.0}, {'current': 2503.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.85, 'min': 0.0, 'max': 0.0}, {'current': 2504.509, 'min': 0.0, 'max': 0.0}, {'current': 2504.271, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6454200744629}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:44:01,791 INFO HandlerThread:1879 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:44:01,791 INFO HandlerThread:1879 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:44:01,791 DEBUG HandlerThread:1879 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:44:01,819 DEBUG HandlerThread:1879 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:44:01,819 DEBUG HandlerThread:1879 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:44:02,534 DEBUG HandlerThread:1879 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:44:02,581 INFO HandlerThread:1879 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:44:02,585 DEBUG SenderThread:1879 [sender.py:send():336] send: files | |||
2023-03-09 23:44:02,585 INFO SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:44:02,600 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:44:02,616 DEBUG SenderThread:1879 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:44:02,626 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:44:02,790 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/conda-environment.yaml | |||
2023-03-09 23:44:02,803 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json | |||
2023-03-09 23:44:02,819 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt | |||
2023-03-09 23:44:03,640 DEBUG SenderThread:1879 [sender.py:send():336] send: config | |||
2023-03-09 23:44:03,732 INFO wandb-upload_0:1879 [upload_job.py:push():138] Uploaded file /tmp/tmp01qe7zccwandb/qvio2pk7-wandb-metadata.json | |||
2023-03-09 23:44:03,793 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log | |||
2023-03-09 23:44:05,810 INFO Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log | |||
2023-03-09 23:44:07,644 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:12,645 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:17,593 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:17,593 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:17,646 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:22,595 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:22,652 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:23,826 INFO Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml | |||
2023-03-09 23:44:27,596 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:28,126 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:32,598 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:33,126 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:37,599 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:38,127 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:42,601 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:43,128 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:47,602 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:48,128 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:52,603 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:53,129 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:44:57,604 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:44:58,130 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:01,745 DEBUG SystemMonitor:1879 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:45:01,750 DEBUG SenderThread:1879 [sender.py:send():336] send: stats | |||
2023-03-09 23:45:02,606 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:03,751 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:07,607 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:08,752 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:12,608 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:13,752 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:17,609 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:18,753 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:22,610 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:23,754 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:27,612 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:28,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:31,752 DEBUG SenderThread:1879 [sender.py:send():336] send: stats | |||
2023-03-09 23:45:32,613 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:34,754 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:37,614 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:39,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:42,617 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:44,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:47,618 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:49,756 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:52,619 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:54,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:45:57,620 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:45:59,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:46:01,755 DEBUG SenderThread:1879 [sender.py:send():336] send: stats | |||
2023-03-09 23:46:02,621 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:46:05,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:46:07,623 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:46:10,758 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:46:12,624 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:46:15,758 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:46:15,881 ERROR memory :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) | |||
2023-03-09 23:46:15,881 ERROR memory :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) | |||
2023-03-09 23:46:15,955 ERROR cpu :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) | |||
2023-03-09 23:46:15,957 ERROR cpu :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) | |||
2023-03-09 23:46:16,690 WARNING StreamThr :1879 [internal.py:is_dead():416] Internal process exiting, parent pid 77 disappeared | |||
2023-03-09 23:46:16,690 ERROR StreamThr :1879 [internal.py:wandb_internal():153] Internal process shutdown. | |||
2023-03-09 23:46:16,744 INFO MainThread:1879 [internal.py:handle_exit():77] Internal process exited |
@@ -0,0 +1,28 @@ | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Configure stats pid to 77 | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:43:51,531 INFO MainThread:77 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:43:51,532 INFO MainThread:77 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:43:51,540 INFO MainThread:77 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:43:51,541 INFO MainThread:77 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:43:56,652 INFO MainThread:77 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:44:01,678 INFO MainThread:77 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:44:01,678 INFO MainThread:77 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:44:02,591 INFO MainThread:77 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:44:02,593 INFO MainThread:77 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:44:02,594 INFO MainThread:77 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,39 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: false | |||
python_version: 3.7.11 | |||
start_time: 1678377055.792902 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,8 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Images scaling: 0.5 | |||
Epoch 1/5: 0%| | 0/5088 [00:00<?, ?img/s] |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:51:02.055192", | |||
"startedAt": "2023-03-09T15:50:55.747682", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "n1d6fa00775e48a18f80acc1355070be-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2499.8833999999993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.977, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.997, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.988, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.917, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.97, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.943, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.54, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2495.346, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.962, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.004, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.959, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.301, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.965, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.368, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.009, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.994, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.888, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.939, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.988, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.823, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.983, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.913, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.978, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.926, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.036, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.982, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.974, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.973, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.013, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2496.799, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.978, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.89, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.002, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.023, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.004, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.98, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.012, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.983, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.039, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.007, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.962, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.03, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.971, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.002, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.03, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.067, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.04, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.001, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.011, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.025, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.999, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.003, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.024, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.914, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.006, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.99, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.056, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.979, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.11, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.137, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.01, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.998, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.6466979980469 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,71 @@ | |||
2023-03-09 23:50:55,798 INFO StreamThr :4276 [internal.py:wandb_internal():90] W&B internal server running at pid: 4276, started at: 2023-03-09 23:50:55.796150 | |||
2023-03-09 23:50:55,800 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:50:55,801 INFO WriterThread:4276 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/run-ay46c760.wandb | |||
2023-03-09 23:50:55,806 DEBUG SenderThread:4276 [sender.py:send():336] send: header | |||
2023-03-09 23:50:55,806 DEBUG SenderThread:4276 [sender.py:send():336] send: run | |||
2023-03-09 23:50:55,827 INFO SenderThread:4276 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/ay46c760 | |||
2023-03-09 23:50:56,963 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:50:56,964 INFO SenderThread:4276 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files | |||
2023-03-09 23:50:56,964 INFO SenderThread:4276 [sender.py:_start_run_threads():1081] run started: ay46c760 with start time 1678377055.792902 | |||
2023-03-09 23:50:56,964 DEBUG SenderThread:4276 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:50:57,005 INFO SenderThread:4276 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:50:57,006 DEBUG SenderThread:4276 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:50:57,967 INFO Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-summary.json | |||
2023-03-09 23:51:01,964 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:02,007 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:02,027 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:51:02,034 DEBUG HandlerThread:4276 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:51:02,034 DEBUG HandlerThread:4276 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:51:02,034 INFO HandlerThread:4276 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:51:02,034 INFO SystemMonitor:4276 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:51:02,034 INFO HandlerThread:4276 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:51:02,035 INFO SystemMonitor:4276 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:51:02,036 INFO SystemMonitor:4276 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:51:02,036 INFO SystemMonitor:4276 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:51:02,037 INFO SystemMonitor:4276 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:51:02,037 INFO SystemMonitor:4276 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:51:02,055 DEBUG HandlerThread:4276 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:51:02,081 DEBUG HandlerThread:4276 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:51:02,081 DEBUG HandlerThread:4276 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:51:02,081 DEBUG HandlerThread:4276 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:51:02.055192', 'startedAt': '2023-03-09T15:50:55.747682', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2499.8833999999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.977, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.97, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2499.54, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2495.346, 'min': 0.0, 'max': 0.0}, {'current': 2499.962, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.959, 'min': 0.0, 'max': 0.0}, {'current': 2499.301, 'min': 0.0, 'max': 0.0}, {'current': 2499.965, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2500.009, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.888, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.823, 'min': 0.0, 'max': 0.0}, {'current': 2499.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.913, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.926, 'min': 0.0, 'max': 0.0}, {'current': 2500.036, 'min': 0.0, 'max': 0.0}, {'current': 2499.982, 'min': 0.0, 'max': 0.0}, {'current': 2499.974, 'min': 0.0, 'max': 0.0}, {'current': 2499.973, 'min': 0.0, 'max': 0.0}, {'current': 2500.013, 'min': 0.0, 'max': 0.0}, {'current': 2496.799, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.89, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2500.023, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.98, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.039, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2499.962, 'min': 0.0, 'max': 0.0}, {'current': 2500.03, 'min': 0.0, 'max': 0.0}, {'current': 2499.971, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2500.03, 'min': 0.0, 'max': 0.0}, {'current': 2500.067, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.025, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.024, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.914, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2500.056, 'min': 0.0, 'max': 0.0}, {'current': 2499.979, 'min': 0.0, 'max': 0.0}, {'current': 2500.11, 'min': 0.0, 'max': 0.0}, {'current': 2500.137, 'min': 0.0, 'max': 0.0}, {'current': 2500.01, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6466979980469}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:51:02,081 INFO HandlerThread:4276 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:51:02,081 INFO HandlerThread:4276 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:51:02,081 DEBUG HandlerThread:4276 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:51:02,151 DEBUG HandlerThread:4276 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:51:02,151 DEBUG HandlerThread:4276 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:51:02,871 DEBUG HandlerThread:4276 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:51:02,931 INFO HandlerThread:4276 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:51:02,934 DEBUG SenderThread:4276 [sender.py:send():336] send: files | |||
2023-03-09 23:51:02,934 INFO SenderThread:4276 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:51:02,943 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:51:02,959 DEBUG SenderThread:4276 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:51:02,969 DEBUG SenderThread:4276 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:51:03,108 INFO Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/conda-environment.yaml | |||
2023-03-09 23:51:03,131 INFO Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/requirements.txt | |||
2023-03-09 23:51:03,144 INFO Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-metadata.json | |||
2023-03-09 23:51:04,016 DEBUG SenderThread:4276 [sender.py:send():336] send: config | |||
2023-03-09 23:51:04,093 INFO wandb-upload_0:4276 [upload_job.py:push():138] Uploaded file /tmp/tmpnt2sknetwandb/70huozag-wandb-metadata.json | |||
2023-03-09 23:51:04,100 INFO Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log | |||
2023-03-09 23:51:06,147 INFO Thread-13 :4276 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log | |||
2023-03-09 23:51:07,026 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:12,027 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:17,028 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:17,942 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:17,942 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:22,029 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:22,943 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:27,035 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:27,945 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:28,167 INFO Thread-13 :4276 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml | |||
2023-03-09 23:51:32,560 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:51:32,946 DEBUG HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:51:36,120 ERROR memory :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543) | |||
2023-03-09 23:51:36,121 ERROR memory :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543) | |||
2023-03-09 23:51:36,159 ERROR cpu :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543) | |||
2023-03-09 23:51:36,162 ERROR cpu :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543) | |||
2023-03-09 23:51:36,843 WARNING StreamThr :4276 [internal.py:is_dead():416] Internal process exiting, parent pid 2543 disappeared | |||
2023-03-09 23:51:36,843 ERROR StreamThr :4276 [internal.py:wandb_internal():153] Internal process shutdown. | |||
2023-03-09 23:51:36,946 INFO HandlerThread:4276 [handler.py:finish():842] shutting down handler | |||
2023-03-09 23:51:37,565 INFO SenderThread:4276 [sender.py:finish():1504] shutting down sender | |||
2023-03-09 23:51:37,565 INFO WriterThread:4276 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/run-ay46c760.wandb | |||
2023-03-09 23:51:38,072 INFO MainThread:4276 [internal.py:handle_exit():77] Internal process exited |
@@ -0,0 +1,27 @@ | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Configure stats pid to 2543 | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:50:55,775 INFO MainThread:2543 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug.log | |||
2023-03-09 23:50:55,776 INFO MainThread:2543 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug-internal.log | |||
2023-03-09 23:50:55,776 INFO MainThread:2543 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:50:55,776 INFO MainThread:2543 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:50:55,776 INFO MainThread:2543 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:50:55,776 INFO MainThread:2543 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:50:55,790 INFO MainThread:2543 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:50:55,792 INFO MainThread:2543 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:50:55,796 INFO MainThread:2543 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:50:55,797 INFO MainThread:2543 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:50:56,962 INFO MainThread:2543 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:51:02,007 INFO MainThread:2543 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:51:02,007 INFO MainThread:2543 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:51:02,940 INFO MainThread:2543 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:51:02,941 INFO MainThread:2543 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:51:02,941 INFO MainThread:2543 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:51:02,941 INFO MainThread:2543 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:51:02,942 INFO MainThread:2543 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:51:02,943 INFO MainThread:2543 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,39 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: false | |||
python_version: 3.7.11 | |||
start_time: 1678377318.650455 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,7 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Images scaling: 0.5 |
@@ -0,0 +1,136 @@ | |||
appdirs==1.4.4 | |||
argon2-cffi-bindings==21.2.0 | |||
argon2-cffi==21.3.0 | |||
asttokens==2.0.8 | |||
astunparse==1.6.3 | |||
attrs==21.4.0 | |||
backcall==0.2.0 | |||
beautifulsoup4==4.10.0 | |||
bleach==5.0.0 | |||
brotlipy==0.7.0 | |||
certifi==2021.10.8 | |||
cffi==1.14.6 | |||
chardet==4.0.0 | |||
click==8.1.3 | |||
conda-build==3.21.5 | |||
conda-package-handling==1.7.3 | |||
conda==4.10.3 | |||
cryptography==35.0.0 | |||
cycler==0.11.0 | |||
debugpy==1.6.0 | |||
decorator==5.1.0 | |||
defusedxml==0.7.1 | |||
dnspython==2.1.0 | |||
docker-pycreds==0.4.0 | |||
easydict==1.10 | |||
entrypoints==0.4 | |||
fastjsonschema==2.15.3 | |||
filelock==3.3.1 | |||
fonttools==4.38.0 | |||
gitdb==4.0.10 | |||
gitpython==3.1.31 | |||
glob2==0.7 | |||
idna==2.10 | |||
imageio==2.25.0 | |||
importlib-metadata==4.11.3 | |||
importlib-resources==5.7.1 | |||
ipykernel==6.13.0 | |||
ipython-genutils==0.2.0 | |||
ipython==7.29.0 | |||
ipywidgets==8.0.4 | |||
jedi==0.18.0 | |||
jinja2==3.1.1 | |||
json5==0.9.6 | |||
jsonschema==4.4.0 | |||
jupyter-client==7.3.0 | |||
jupyter-core==4.10.0 | |||
jupyterlab-pygments==0.2.2 | |||
jupyterlab-server==1.2.0 | |||
jupyterlab-widgets==3.0.5 | |||
jupyterlab==2.2.5 | |||
kiwisolver==1.4.4 | |||
libarchive-c==2.9 | |||
markupsafe==2.0.1 | |||
matplotlib-inline==0.1.2 | |||
matplotlib==3.5.3 | |||
mindspore-cuda11-dev==2.0.0.dev20221108 | |||
mindspore-dev==2.0.0.dev20230109 | |||
minkowskiengine==0.5.4 | |||
mistune==0.8.4 | |||
mkl-fft==1.3.1 | |||
mkl-random==1.2.2 | |||
mkl-service==2.4.0 | |||
ms-adapter==0.1.0 | |||
msadapter==0.0.1a0 | |||
nbclient==0.6.0 | |||
nbconvert==6.5.0 | |||
nbformat==5.3.0 | |||
nest-asyncio==1.5.5 | |||
networkx==2.6.3 | |||
ninja==1.11.1 | |||
notebook==6.4.11 | |||
numpy==1.21.2 | |||
olefile==0.46 | |||
open3d-python==0.7.0.0 | |||
opencv-python==4.6.0.66 | |||
packaging==21.3 | |||
pandas==1.3.5 | |||
pandocfilters==1.5.0 | |||
parso==0.8.2 | |||
pathtools==0.1.2 | |||
pexpect==4.8.0 | |||
pickleshare==0.7.5 | |||
pillow==8.4.0 | |||
pip==21.0.1 | |||
pkginfo==1.7.1 | |||
prometheus-client==0.14.1 | |||
prompt-toolkit==3.0.20 | |||
protobuf==3.20.3 | |||
psutil==5.8.0 | |||
ptyprocess==0.7.0 | |||
pycosat==0.6.3 | |||
pycparser==2.20 | |||
pygments==2.10.0 | |||
pyopenssl==20.0.1 | |||
pyparsing==3.0.8 | |||
pypng==0.20220715.0 | |||
pyrsistent==0.18.1 | |||
pysocks==1.7.1 | |||
python-dateutil==2.8.2 | |||
python-etcd==0.4.5 | |||
pytz==2021.3 | |||
pywavelets==1.3.0 | |||
pyyaml==6.0 | |||
pyzmq==22.3.0 | |||
requests==2.25.1 | |||
ruamel-yaml-conda==0.15.100 | |||
scikit-image==0.19.3 | |||
scipy==1.7.3 | |||
send2trash==1.8.0 | |||
sentry-sdk==1.16.0 | |||
setproctitle==1.3.2 | |||
setuptools==58.0.4 | |||
six==1.16.0 | |||
smmap==5.0.0 | |||
soupsieve==2.2.1 | |||
tensorboardx==2.6 | |||
terminado==0.13.3 | |||
tifffile==2021.11.2 | |||
tinycss2==1.1.1 | |||
torch==1.10.0 | |||
torchac==0.9.3 | |||
torchelastic==0.2.0 | |||
torchtext==0.11.0 | |||
torchvision==0.11.1 | |||
tornado==6.1 | |||
tqdm==4.61.2 | |||
traitlets==5.1.0 | |||
typing-extensions==3.10.0.2 | |||
urllib3==1.26.14 | |||
wandb==0.13.11 | |||
wcwidth==0.2.5 | |||
webencodings==0.5.1 | |||
wheel==0.36.2 | |||
widgetsnbextension==4.0.5 | |||
xlrd==1.2.0 | |||
zipp==3.8.0 |
@@ -0,0 +1,443 @@ | |||
{ | |||
"os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", | |||
"python": "3.7.11", | |||
"heartbeatAt": "2023-03-09T15:55:25.128694", | |||
"startedAt": "2023-03-09T15:55:18.607606", | |||
"docker": null, | |||
"cuda": null, | |||
"args": [], | |||
"state": "running", | |||
"program": "train.py", | |||
"codePath": "train.py", | |||
"host": "n1d6fa00775e48a18f80acc1355070be-task0-0", | |||
"username": "root", | |||
"executable": "/opt/conda/bin/python", | |||
"cpu_count": 40, | |||
"cpu_count_logical": 80, | |||
"cpu_freq": { | |||
"current": 2502.0496374999993, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
"cpu_freq_per_core": [ | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.767, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.475, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.398, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.156, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.385, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.678, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.643, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.163, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.395, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.117, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.927, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2497.953, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.177, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.15, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.575, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.649, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.6, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.068, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.553, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.434, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.396, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.163, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.428, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.283, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.769, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.975, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.573, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.008, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.441, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.995, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.077, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.185, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.401, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.465, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.575, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2499.949, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.373, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.085, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.672, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2498.784, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.976, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.28, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.92, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.817, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.072, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.767, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.946, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.257, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.292, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2502.734, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.093, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.904, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.615, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.676, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2503.732, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.387, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.376, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.485, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.054, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.539, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.0, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2507.286, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.724, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2508.643, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2501.17, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.69, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.158, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.016, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.234, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.759, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.361, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.269, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2500.461, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2505.742, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2504.565, | |||
"min": 0.0, | |||
"max": 0.0 | |||
}, | |||
{ | |||
"current": 2506.111, | |||
"min": 0.0, | |||
"max": 0.0 | |||
} | |||
], | |||
"disk": { | |||
"total": 878.6640281677246, | |||
"used": 283.64724349975586 | |||
}, | |||
"gpu": "Tesla T4", | |||
"gpu_count": 2, | |||
"gpu_devices": [ | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
}, | |||
{ | |||
"name": "Tesla T4", | |||
"memory_total": 15843721216 | |||
} | |||
], | |||
"memory": { | |||
"total": 376.5794219970703 | |||
} | |||
} |
@@ -0,0 +1 @@ | |||
{} |
@@ -0,0 +1,133 @@ | |||
2023-03-09 23:55:18,660 INFO StreamThr :6152 [internal.py:wandb_internal():90] W&B internal server running at pid: 6152, started at: 2023-03-09 23:55:18.657868 | |||
2023-03-09 23:55:18,661 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status | |||
2023-03-09 23:55:18,663 INFO WriterThread:6152 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/run-wqdqlwjv.wandb | |||
2023-03-09 23:55:18,671 DEBUG SenderThread:6152 [sender.py:send():336] send: header | |||
2023-03-09 23:55:18,671 DEBUG SenderThread:6152 [sender.py:send():336] send: run | |||
2023-03-09 23:55:18,690 INFO SenderThread:6152 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/wqdqlwjv | |||
2023-03-09 23:55:19,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: check_version | |||
2023-03-09 23:55:19,991 INFO SenderThread:6152 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files | |||
2023-03-09 23:55:19,991 INFO SenderThread:6152 [sender.py:_start_run_threads():1081] run started: wqdqlwjv with start time 1678377318.650455 | |||
2023-03-09 23:55:19,991 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: summary_record | |||
2023-03-09 23:55:20,015 INFO SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end | |||
2023-03-09 23:55:20,016 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: check_version | |||
2023-03-09 23:55:20,998 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json | |||
2023-03-09 23:55:24,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:25,017 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:25,076 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: run_start | |||
2023-03-09 23:55:25,081 DEBUG HandlerThread:6152 [system_info.py:__init__():31] System info init | |||
2023-03-09 23:55:25,081 DEBUG HandlerThread:6152 [system_info.py:__init__():46] System info init done | |||
2023-03-09 23:55:25,081 INFO HandlerThread:6152 [system_monitor.py:start():183] Starting system monitor | |||
2023-03-09 23:55:25,082 INFO SystemMonitor:6152 [system_monitor.py:_start():147] Starting system asset monitoring threads | |||
2023-03-09 23:55:25,082 INFO HandlerThread:6152 [system_monitor.py:probe():204] Collecting system info | |||
2023-03-09 23:55:25,082 INFO SystemMonitor:6152 [interfaces.py:start():187] Started cpu monitoring | |||
2023-03-09 23:55:25,083 INFO SystemMonitor:6152 [interfaces.py:start():187] Started disk monitoring | |||
2023-03-09 23:55:25,083 INFO SystemMonitor:6152 [interfaces.py:start():187] Started gpu monitoring | |||
2023-03-09 23:55:25,084 INFO SystemMonitor:6152 [interfaces.py:start():187] Started memory monitoring | |||
2023-03-09 23:55:25,085 INFO SystemMonitor:6152 [interfaces.py:start():187] Started network monitoring | |||
2023-03-09 23:55:25,128 DEBUG HandlerThread:6152 [system_info.py:probe():195] Probing system | |||
2023-03-09 23:55:25,168 DEBUG HandlerThread:6152 [git.py:repo():40] git repository is invalid | |||
2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_info.py:probe():240] Probing system done | |||
2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:55:25.128694', 'startedAt': '2023-03-09T15:55:18.607606', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2502.0496374999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.475, 'min': 0.0, 'max': 0.0}, {'current': 2500.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.156, 'min': 0.0, 'max': 0.0}, {'current': 2500.385, 'min': 0.0, 'max': 0.0}, {'current': 2501.678, 'min': 0.0, 'max': 0.0}, {'current': 2499.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.163, 'min': 0.0, 'max': 0.0}, {'current': 2500.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.117, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2497.953, 'min': 0.0, 'max': 0.0}, {'current': 2499.177, 'min': 0.0, 'max': 0.0}, {'current': 2501.15, 'min': 0.0, 'max': 0.0}, {'current': 2503.575, 'min': 0.0, 'max': 0.0}, {'current': 2501.649, 'min': 0.0, 'max': 0.0}, {'current': 2500.6, 'min': 0.0, 'max': 0.0}, {'current': 2501.068, 'min': 0.0, 'max': 0.0}, {'current': 2501.553, 'min': 0.0, 'max': 0.0}, {'current': 2500.434, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2499.163, 'min': 0.0, 'max': 0.0}, {'current': 2501.428, 'min': 0.0, 'max': 0.0}, {'current': 2501.283, 'min': 0.0, 'max': 0.0}, {'current': 2501.769, 'min': 0.0, 'max': 0.0}, {'current': 2499.975, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.441, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.077, 'min': 0.0, 'max': 0.0}, {'current': 2499.185, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2500.465, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2499.949, 'min': 0.0, 'max': 0.0}, {'current': 2501.373, 'min': 0.0, 'max': 0.0}, {'current': 2501.085, 'min': 0.0, 'max': 0.0}, {'current': 2500.672, 'min': 0.0, 'max': 0.0}, {'current': 2498.784, 'min': 0.0, 'max': 0.0}, {'current': 2504.976, 'min': 0.0, 'max': 0.0}, {'current': 2505.28, 'min': 0.0, 'max': 0.0}, {'current': 2503.92, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.817, 'min': 0.0, 'max': 0.0}, {'current': 2507.072, 'min': 0.0, 'max': 0.0}, {'current': 2504.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2503.257, 'min': 0.0, 'max': 0.0}, {'current': 2500.292, 'min': 0.0, 'max': 0.0}, {'current': 2502.734, 'min': 0.0, 'max': 0.0}, {'current': 2507.093, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.904, 'min': 0.0, 'max': 0.0}, {'current': 2505.615, 'min': 0.0, 'max': 0.0}, {'current': 2503.676, 'min': 0.0, 'max': 0.0}, {'current': 2503.732, 'min': 0.0, 'max': 0.0}, {'current': 2505.387, 'min': 0.0, 'max': 0.0}, {'current': 2500.376, 'min': 0.0, 'max': 0.0}, {'current': 2501.485, 'min': 0.0, 'max': 0.0}, {'current': 2506.054, 'min': 0.0, 'max': 0.0}, {'current': 2506.539, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2507.286, 'min': 0.0, 'max': 0.0}, {'current': 2500.724, 'min': 0.0, 'max': 0.0}, {'current': 2508.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.17, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2505.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2500.234, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2504.361, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2505.742, 'min': 0.0, 'max': 0.0}, {'current': 2504.565, 'min': 0.0, 'max': 0.0}, {'current': 2506.111, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64724349975586}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} | |||
2023-03-09 23:55:25,169 INFO HandlerThread:6152 [system_monitor.py:probe():214] Finished collecting system info | |||
2023-03-09 23:55:25,169 INFO HandlerThread:6152 [system_monitor.py:probe():217] Publishing system info | |||
2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |||
2023-03-09 23:55:25,187 DEBUG HandlerThread:6152 [system_info.py:_save_pip():67] Saving pip packages done | |||
2023-03-09 23:55:25,187 DEBUG HandlerThread:6152 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |||
2023-03-09 23:55:25,907 DEBUG HandlerThread:6152 [system_info.py:_save_conda():86] Saving conda packages done | |||
2023-03-09 23:55:25,960 INFO HandlerThread:6152 [system_monitor.py:probe():219] Finished publishing system info | |||
2023-03-09 23:55:25,963 DEBUG SenderThread:6152 [sender.py:send():336] send: files | |||
2023-03-09 23:55:25,964 INFO SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now | |||
2023-03-09 23:55:25,999 DEBUG SenderThread:6152 [sender.py:send():336] send: telemetry | |||
2023-03-09 23:55:25,999 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: stop_status | |||
2023-03-09 23:55:26,008 DEBUG SenderThread:6152 [sender.py:send():336] send: config | |||
2023-03-09 23:55:26,055 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: stop_status | |||
2023-03-09 23:55:26,143 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt | |||
2023-03-09 23:55:26,165 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json | |||
2023-03-09 23:55:26,171 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/conda-environment.yaml | |||
2023-03-09 23:55:27,136 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log | |||
2023-03-09 23:55:27,210 INFO wandb-upload_0:6152 [upload_job.py:push():138] Uploaded file /tmp/tmpcpr9x05xwandb/qhhlrewz-wandb-metadata.json | |||
2023-03-09 23:55:29,138 INFO Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log | |||
2023-03-09 23:55:30,921 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:35,922 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:40,923 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:40,972 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:40,973 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:45,923 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:45,974 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:50,929 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:55:50,991 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:52,159 INFO Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml | |||
2023-03-09 23:55:55,977 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:55:56,728 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:00,979 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:01,728 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:05,980 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:06,729 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:10,982 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:11,730 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:15,983 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:16,731 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:20,984 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:21,731 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:25,085 DEBUG SystemMonitor:6152 [system_monitor.py:_start():161] Starting system metrics aggregation loop | |||
2023-03-09 23:56:25,090 DEBUG SenderThread:6152 [sender.py:send():336] send: stats | |||
2023-03-09 23:56:25,986 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:27,092 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:30,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:32,092 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:35,988 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:37,093 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:40,990 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:42,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:45,991 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:47,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:50,992 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:52,095 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:56:55,093 DEBUG SenderThread:6152 [sender.py:send():336] send: stats | |||
2023-03-09 23:56:55,994 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:56:58,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:00,995 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:03,095 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:05,996 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:08,096 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:10,997 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:13,096 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:15,998 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:18,097 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:21,000 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:23,098 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:25,095 DEBUG SenderThread:6152 [sender.py:send():336] send: stats | |||
2023-03-09 23:57:26,001 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:29,097 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:31,003 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:34,098 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:36,004 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:39,099 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:41,005 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:44,099 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:46,007 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:49,100 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:51,008 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:57:54,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:57:55,098 DEBUG SenderThread:6152 [sender.py:send():336] send: stats | |||
2023-03-09 23:57:56,009 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:00,100 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:01,010 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:05,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:06,012 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:10,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:11,013 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:15,102 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:16,014 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:20,103 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:21,015 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:25,101 DEBUG SenderThread:6152 [sender.py:send():336] send: stats | |||
2023-03-09 23:58:26,018 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:26,103 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:31,018 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:31,104 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:36,019 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:36,105 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:41,021 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive | |||
2023-03-09 23:58:41,108 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report | |||
2023-03-09 23:58:45,110 INFO MainThread:6152 [internal.py:handle_exit():77] Internal process exited |
@@ -0,0 +1,27 @@ | |||
2023-03-09 23:55:18,636 INFO MainThread:4410 [wandb_setup.py:_flush():76] Configure stats pid to 4410 | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():546] calling init triggers | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():553] wandb.init called with sweep_config: {} | |||
config: {} | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():602] starting backend | |||
2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():606] setting up manager | |||
2023-03-09 23:55:18,648 INFO MainThread:4410 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |||
2023-03-09 23:55:18,650 INFO MainThread:4410 [wandb_init.py:init():613] backend started and connected | |||
2023-03-09 23:55:18,654 INFO MainThread:4410 [wandb_init.py:init():701] updated telemetry | |||
2023-03-09 23:55:18,655 INFO MainThread:4410 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout | |||
2023-03-09 23:55:19,986 INFO MainThread:4410 [wandb_run.py:_on_init():2131] communicating current version | |||
2023-03-09 23:55:25,018 INFO MainThread:4410 [wandb_run.py:_on_init():2140] got version response | |||
2023-03-09 23:55:25,018 INFO MainThread:4410 [wandb_init.py:init():789] starting run threads in backend | |||
2023-03-09 23:55:25,970 INFO MainThread:4410 [wandb_run.py:_console_start():2112] atexit reg | |||
2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW | |||
2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():2032] Wrapping output streams. | |||
2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():2057] Redirects installed. | |||
2023-03-09 23:55:25,972 INFO MainThread:4410 [wandb_init.py:init():831] run started, returning control to user process | |||
2023-03-09 23:55:25,972 INFO MainThread:4410 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} |
@@ -0,0 +1,39 @@ | |||
wandb_version: 1 | |||
_wandb: | |||
desc: null | |||
value: | |||
cli_version: 0.13.11 | |||
framework: torch | |||
is_jupyter_run: false | |||
is_kaggle_kernel: false | |||
python_version: 3.7.11 | |||
start_time: 1678377605.403295 | |||
t: | |||
1: | |||
- 1 | |||
- 55 | |||
2: | |||
- 1 | |||
- 55 | |||
3: | |||
- 23 | |||
4: 3.7.11 | |||
5: 0.13.11 | |||
8: | |||
- 5 | |||
batch_size: | |||
desc: null | |||
value: 1 | |||
epochs: | |||
desc: null | |||
value: 5 | |||
img_scale: | |||
desc: null | |||
value: 0.5 | |||
learning_rate: | |||
desc: null | |||
value: 1.0e-05 | |||
save_checkpoint: | |||
desc: null | |||
value: true |
@@ -0,0 +1,8 @@ | |||
INFO: Starting training: | |||
Epochs: 5 | |||
Batch size: 1 | |||
Learning rate: 1e-05 | |||
Training size: 5088 | |||
Checkpoints: True | |||
Images scaling: 0.5 | |||
Epoch 1/5: 0%| | 0/5088 [00:00<?, ?img/s] |
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》