diff --git a/official/cv/unet2d/2dunet-adapter/README.md b/official/cv/unet2d/2dunet-adapter/README.md new file mode 100644 index 0000000..cb4d4c9 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/README.md @@ -0,0 +1,189 @@ +# U-Net: Semantic segmentation with PyTorch + + + + + +![input and output for a random image in the test dataset](https://i.imgur.com/GD8FcB7.png) + + +Customized implementation of the [U-Net](https://arxiv.org/abs/1505.04597) in PyTorch for Kaggle's [Carvana Image Masking Challenge](https://www.kaggle.com/c/carvana-image-masking-challenge) from high definition images. + +- [Quick start](#quick-start) + - [Without Docker](#without-docker) + - [With Docker](#with-docker) +- [Description](#description) +- [Usage](#usage) + - [Docker](#docker) + - [Training](#training) + - [Prediction](#prediction) +- [Weights & Biases](#weights--biases) +- [Pretrained model](#pretrained-model) +- [Data](#data) + +## Quick start + +### Without Docker + +1. [Install CUDA](https://developer.nvidia.com/cuda-downloads) + +2. [Install PyTorch 1.13 or later](https://pytorch.org/get-started/locally/) + +3. Install dependencies +```bash +pip install -r requirements.txt +``` + +4. Download the data and run training: +```bash +bash scripts/download_data.sh +python train.py --amp +``` + +### With Docker + +1. [Install Docker 19.03 or later:](https://docs.docker.com/get-docker/) +```bash +curl https://get.docker.com | sh && sudo systemctl --now enable docker +``` +2. [Install the NVIDIA container toolkit:](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) +```bash +distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ + && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ + && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list +sudo apt-get update +sudo apt-get install -y nvidia-docker2 +sudo systemctl restart docker +``` +3. [Download and run the image:](https://hub.docker.com/repository/docker/milesial/unet) +```bash +sudo docker run --rm --shm-size=8g --ulimit memlock=-1 --gpus all -it milesial/unet +``` + +4. Download the data and run training: +```bash +bash scripts/download_data.sh +python train.py --amp +``` + +## Description +This model was trained from scratch with 5k images and scored a [Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) of 0.988423 on over 100k test images. + +It can be easily used for multiclass segmentation, portrait segmentation, medical segmentation, ... + + +## Usage +**Note : Use Python 3.6 or newer** + +### Docker + +A docker image containing the code and the dependencies is available on [DockerHub](https://hub.docker.com/repository/docker/milesial/unet). +You can download and jump in the container with ([docker >=19.03](https://docs.docker.com/get-docker/)): + +```console +docker run -it --rm --shm-size=8g --ulimit memlock=-1 --gpus all milesial/unet +``` + + +### Training + +```console +> python train.py -h +usage: train.py [-h] [--epochs E] [--batch-size B] [--learning-rate LR] + [--load LOAD] [--scale SCALE] [--validation VAL] [--amp] + +Train the UNet on images and target masks + +optional arguments: + -h, --help show this help message and exit + --epochs E, -e E Number of epochs + --batch-size B, -b B Batch size + --learning-rate LR, -l LR + Learning rate + --load LOAD, -f LOAD Load model from a .pth file + --scale SCALE, -s SCALE + Downscaling factor of the images + --validation VAL, -v VAL + Percent of the data that is used as validation (0-100) + --amp Use mixed precision +``` + +By default, the `scale` is 0.5, so if you wish to obtain better results (but use more memory), set it to 1. + +Automatic mixed precision is also available with the `--amp` flag. [Mixed precision](https://arxiv.org/abs/1710.03740) allows the model to use less memory and to be faster on recent GPUs by using FP16 arithmetic. Enabling AMP is recommended. + + +### Prediction + +After training your model and saving it to `MODEL.pth`, you can easily test the output masks on your images via the CLI. + +To predict a single image and save it: + +`python predict.py -i image.jpg -o output.jpg` + +To predict a multiple images and show them without saving them: + +`python predict.py -i image1.jpg image2.jpg --viz --no-save` + +```console +> python predict.py -h +usage: predict.py [-h] [--model FILE] --input INPUT [INPUT ...] + [--output INPUT [INPUT ...]] [--viz] [--no-save] + [--mask-threshold MASK_THRESHOLD] [--scale SCALE] + +Predict masks from input images + +optional arguments: + -h, --help show this help message and exit + --model FILE, -m FILE + Specify the file in which the model is stored + --input INPUT [INPUT ...], -i INPUT [INPUT ...] + Filenames of input images + --output INPUT [INPUT ...], -o INPUT [INPUT ...] + Filenames of output images + --viz, -v Visualize the images as they are processed + --no-save, -n Do not save the output masks + --mask-threshold MASK_THRESHOLD, -t MASK_THRESHOLD + Minimum probability value to consider a mask pixel white + --scale SCALE, -s SCALE + Scale factor for the input images +``` +You can specify which model file to use with `--model MODEL.pth`. + +## Weights & Biases + +The training progress can be visualized in real-time using [Weights & Biases](https://wandb.ai/). Loss curves, validation curves, weights and gradient histograms, as well as predicted masks are logged to the platform. + +When launching a training, a link will be printed in the console. Click on it to go to your dashboard. If you have an existing W&B account, you can link it + by setting the `WANDB_API_KEY` environment variable. If not, it will create an anonymous run which is automatically deleted after 7 days. + + +## Pretrained model +A [pretrained model](https://github.com/milesial/Pytorch-UNet/releases/tag/v3.0) is available for the Carvana dataset. It can also be loaded from torch.hub: + +```python +net = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana', pretrained=True, scale=0.5) +``` +Available scales are 0.5 and 1.0. + +## Data +The Carvana data is available on the [Kaggle website](https://www.kaggle.com/c/carvana-image-masking-challenge/data). + +You can also download it using the helper script: + +``` +bash scripts/download_data.sh +``` + +The input images and target masks should be in the `data/imgs` and `data/masks` folders respectively (note that the `imgs` and `masks` folder should not contain any sub-folder or any other files, due to the greedy data-loader). For Carvana, images are RGB and masks are black and white. + +You can use your own dataset as long as you make sure it is loaded properly in `utils/data_loading.py`. + + +--- + +Original paper by Olaf Ronneberger, Philipp Fischer, Thomas Brox: + +[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597) + +![network architecture](https://i.imgur.com/jeDVpqF.png) diff --git a/official/cv/unet2d/2dunet-adapter/data/imgs/.keep b/official/cv/unet2d/2dunet-adapter/data/imgs/.keep new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/data/masks/.keep b/official/cv/unet2d/2dunet-adapter/data/masks/.keep new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/requirements.txt b/official/cv/unet2d/2dunet-adapter/requirements.txt new file mode 100644 index 0000000..256fb59 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/requirements.txt @@ -0,0 +1,5 @@ +matplotlib==3.6.2 +numpy==1.23.5 +Pillow==9.3.0 +tqdm==4.64.1 +wandb==0.13.5 diff --git a/official/cv/unet2d/2dunet-adapter/scripts/download_data.sh b/official/cv/unet2d/2dunet-adapter/scripts/download_data.sh new file mode 100644 index 0000000..a151ede --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/scripts/download_data.sh @@ -0,0 +1,27 @@ +#!/bin/bash + +if [[ ! -f ~/.kaggle/kaggle.json ]]; then + echo -n "Kaggle username: " + read USERNAME + echo + echo -n "Kaggle API key: " + read APIKEY + + mkdir -p ~/.kaggle + echo "{\"username\":\"$USERNAME\",\"key\":\"$APIKEY\"}" > ~/.kaggle/kaggle.json + chmod 600 ~/.kaggle/kaggle.json +fi + +pip install kaggle --upgrade + +kaggle competitions download -c carvana-image-masking-challenge -f train_hq.zip +unzip train_hq.zip +mv train_hq/* data/imgs/ +rm -d train_hq +rm train_hq.zip + +kaggle competitions download -c carvana-image-masking-challenge -f train_masks.zip +unzip train_masks.zip +mv train_masks/* data/masks/ +rm -d train_masks +rm train_masks.zip diff --git a/official/cv/unet2d/2dunet-adapter/train.py b/official/cv/unet2d/2dunet-adapter/train.py new file mode 100644 index 0000000..0ac2e07 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/train.py @@ -0,0 +1,164 @@ +import argparse +import logging +import os +os.environ['CUDA_VISIBLE_DEVICES'] = '1' +import random +import sys +import ms_adapter.pytorch as torch +import ms_adapter.pytorch.nn as nn +import ms_adapter.pytorch.nn.functional as F +import mindspore as ms + +import ms_adapter.torchvision.transforms as transforms +import ms_adapter.torchvision.transforms.functional as TF +from pathlib import Path +from ms_adapter.pytorch import optim +from ms_adapter.pytorch.utils.data import DataLoader, random_split +from tqdm import tqdm + +from unet import UNet +from utils.data_loading import BasicDataset, CarvanaDataset +from utils.dice_score import dice_loss + +dir_img = Path('/code/UNet-master/2dunet-pytorch/data/imgs/') +dir_mask = Path('/code/UNet-master/2dunet-pytorch/data/masks/') +dir_checkpoint = Path('./checkpoints/') + +ms.set_context(device_target='GPU') + + +def train_model( + model, + epochs: int = 5, + batch_size: int = 1, + learning_rate: float = 1e-5, + save_checkpoint: bool = True, + img_scale: float = 0.5, + weight_decay: float = 1e-8, + momentum: float = 0.999, +): + # 1. Create dataset + try: + dataset = CarvanaDataset(dir_img, dir_mask, img_scale) + print('Load CarvanaDataset') + except (AssertionError, RuntimeError, IndexError): + dataset = BasicDataset(dir_img, dir_mask, img_scale) + print('Load BasicDataset') + + n_train = len(dataset) + + # 3. Create data loaders + loader_args = dict(batch_size=batch_size, num_workers=os.cpu_count()) + train_loader = DataLoader(dataset, shuffle=True, **loader_args) + #val_loader = DataLoader(val_set, shuffle=False, drop_last=True, **loader_args) + + # (Initialize logging) + + logging.info(f'''Starting training: + Epochs: {epochs} + Batch size: {batch_size} + Learning rate: {learning_rate} + Training size: {n_train} + Checkpoints: {save_checkpoint} + Images scaling: {img_scale} + ''') + + # 4. Set up the optimizer, the loss, the learning rate scheduler and the loss scaling for AMP + optimizer = ms.nn.RMSProp(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay) + criterion = nn.CrossEntropyLoss() if model.n_classes > 1 else nn.BCEWithLogitsLoss() + global_step = 0 + + + def forward_fn(data, label): + masks_pred = model(data) + + if model.n_classes == 1: + loss = criterion(masks_pred.squeeze(1), true_masks.float()) + loss += dice_loss(F.sigmoid(masks_pred.squeeze(1)), true_masks.float(), multiclass=False) + else: + loss = criterion(masks_pred, true_masks) + loss += dice_loss(F.softmax(masks_pred, dim=1).float(), + F.one_hot(true_masks, model.n_classes).permute(0, 3, 1, 2).float(), multiclass=True) + + return loss, masks_pred + + grad_fn = ms.ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True) + + def train_step(data, label): + (loss, _), grads = grad_fn(data, label) + loss = ms.ops.depend(loss, optimizer(grads)) + return loss + + + + # 5. Begin training + for epoch in range(1, epochs + 1): + model.train() + epoch_loss = 0 + print('train_loader', len(train_loader), train_loader) + with tqdm(total=n_train, desc=f'Epoch {epoch}/{epochs}', unit='img') as pbar: + for batch in train_loader: + images, true_masks = batch['image'], batch['mask'] + assert images.shape[1] == model.n_channels, \ + f'Network has been defined with {model.n_channels} input channels, ' \ + f'but loaded images have {images.shape[1]} channels. Please check that ' \ + 'the images are loaded correctly.' + + images = images.to(dtype=torch.float32) + true_masks = true_masks.to(dtype=torch.long) + + #masks_pred = model(images) + loss = train_step(images, true_masks) + + pbar.update(images.shape[0]) + global_step += 1 + epoch_loss += loss.item() + print( + 'train loss: %f' % loss.item(), + 'step: %d' % global_step, + 'epoch: %d' % epoch + ) + pbar.set_postfix(**{'loss (batch)': loss.item()}) + + + + +def get_args(): + parser = argparse.ArgumentParser(description='Train the UNet on images and target masks') + parser.add_argument('--epochs', '-e', metavar='E', type=int, default=5, help='Number of epochs') + parser.add_argument('--batch-size', '-b', dest='batch_size', metavar='B', type=int, default=1, help='Batch size') + parser.add_argument('--learning-rate', '-l', metavar='LR', type=float, default=1e-5, + help='Learning rate', dest='lr') + parser.add_argument('--load', '-f', type=str, default=False, help='Load model from a .pth file') + parser.add_argument('--scale', '-s', type=float, default=0.5, help='Downscaling factor of the images') + parser.add_argument('--validation', '-v', dest='val', type=float, default=10.0, + help='Percent of the data that is used as validation (0-100)') + parser.add_argument('--bilinear', action='store_true', default=False, help='Use bilinear upsampling') + parser.add_argument('--classes', '-c', type=int, default=2, help='Number of classes') + + return parser.parse_args() + + +if __name__ == '__main__': + args = get_args() + + logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s') + + # Change here to adapt to your data + # n_channels=3 for RGB images + # n_classes is the number of probabilities you want to get per pixel + model = UNet(n_channels=3, n_classes=args.classes, bilinear=args.bilinear) + + logging.info(f'Network:\n' + f'\t{model.n_channels} input channels\n' + f'\t{model.n_classes} output channels (classes)\n' + f'\t{"Bilinear" if model.bilinear else "Transposed conv"} upscaling') + + + train_model( + model=model, + epochs=args.epochs, + batch_size=args.batch_size, + learning_rate=args.lr, + img_scale=args.scale, + ) diff --git a/official/cv/unet2d/2dunet-adapter/unet/__init__.py b/official/cv/unet2d/2dunet-adapter/unet/__init__.py new file mode 100644 index 0000000..2e9b63b --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/unet/__init__.py @@ -0,0 +1 @@ +from .unet_model import UNet diff --git a/official/cv/unet2d/2dunet-adapter/unet/unet_model.py b/official/cv/unet2d/2dunet-adapter/unet/unet_model.py new file mode 100644 index 0000000..caf79f4 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/unet/unet_model.py @@ -0,0 +1,48 @@ +""" Full assembly of the parts to form the complete network """ + +from .unet_parts import * + + +class UNet(nn.Module): + def __init__(self, n_channels, n_classes, bilinear=False): + super(UNet, self).__init__() + self.n_channels = n_channels + self.n_classes = n_classes + self.bilinear = bilinear + + self.inc = (DoubleConv(n_channels, 64)) + self.down1 = (Down(64, 128)) + self.down2 = (Down(128, 256)) + self.down3 = (Down(256, 512)) + factor = 2 if bilinear else 1 + self.down4 = (Down(512, 1024 // factor)) + self.up1 = (Up(1024, 512 // factor, bilinear)) + self.up2 = (Up(512, 256 // factor, bilinear)) + self.up3 = (Up(256, 128 // factor, bilinear)) + self.up4 = (Up(128, 64, bilinear)) + self.outc = (OutConv(64, n_classes)) + + def forward(self, x): + x1 = self.inc(x) + x2 = self.down1(x1) + x3 = self.down2(x2) + x4 = self.down3(x3) + x5 = self.down4(x4) + x = self.up1(x5, x4) + x = self.up2(x, x3) + x = self.up3(x, x2) + x = self.up4(x, x1) + logits = self.outc(x) + return logits + + def use_checkpointing(self): + self.inc = torch.utils.checkpoint(self.inc) + self.down1 = torch.utils.checkpoint(self.down1) + self.down2 = torch.utils.checkpoint(self.down2) + self.down3 = torch.utils.checkpoint(self.down3) + self.down4 = torch.utils.checkpoint(self.down4) + self.up1 = torch.utils.checkpoint(self.up1) + self.up2 = torch.utils.checkpoint(self.up2) + self.up3 = torch.utils.checkpoint(self.up3) + self.up4 = torch.utils.checkpoint(self.up4) + self.outc = torch.utils.checkpoint(self.outc) \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/unet/unet_parts.py b/official/cv/unet2d/2dunet-adapter/unet/unet_parts.py new file mode 100644 index 0000000..cf93dc1 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/unet/unet_parts.py @@ -0,0 +1,77 @@ +""" Parts of the U-Net model """ + +import ms_adapter.pytorch as torch +import ms_adapter.pytorch.nn as nn +import ms_adapter.pytorch.nn.functional as F + + +class DoubleConv(nn.Module): + """(convolution => [BN] => ReLU) * 2""" + + def __init__(self, in_channels, out_channels, mid_channels=None): + super().__init__() + if not mid_channels: + mid_channels = out_channels + self.double_conv = nn.Sequential( + nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(mid_channels), + nn.ReLU(inplace=True), + nn.Conv2d(mid_channels, out_channels, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(out_channels), + nn.ReLU(inplace=True) + ) + + def forward(self, x): + return self.double_conv(x) + + +class Down(nn.Module): + """Downscaling with maxpool then double conv""" + + def __init__(self, in_channels, out_channels): + super().__init__() + self.maxpool_conv = nn.Sequential( + nn.MaxPool2d(2), + DoubleConv(in_channels, out_channels) + ) + + def forward(self, x): + return self.maxpool_conv(x) + + +class Up(nn.Module): + """Upscaling then double conv""" + + def __init__(self, in_channels, out_channels, bilinear=True): + super().__init__() + + # if bilinear, use the normal convolutions to reduce the number of channels + if bilinear: + self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True) + self.conv = DoubleConv(in_channels, out_channels, in_channels // 2) + else: + self.up = nn.ConvTranspose2d(in_channels, in_channels // 2, kernel_size=2, stride=2) + self.conv = DoubleConv(in_channels, out_channels) + + def forward(self, x1, x2): + x1 = self.up(x1) + # input is CHW + diffY = x2.size()[2] - x1.size()[2] + diffX = x2.size()[3] - x1.size()[3] + + x1 = F.pad(x1, [diffX // 2, diffX - diffX // 2, + diffY // 2, diffY - diffY // 2]) + # if you have padding issues, see + # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a + # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd + x = torch.cat([x2, x1], dim=1) + return self.conv(x) + + +class OutConv(nn.Module): + def __init__(self, in_channels, out_channels): + super(OutConv, self).__init__() + self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1) + + def forward(self, x): + return self.conv(x) diff --git a/official/cv/unet2d/2dunet-adapter/utils/__init__.py b/official/cv/unet2d/2dunet-adapter/utils/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/utils/data_loading.py b/official/cv/unet2d/2dunet-adapter/utils/data_loading.py new file mode 100644 index 0000000..c5b0495 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/utils/data_loading.py @@ -0,0 +1,117 @@ +import logging +import numpy as np +import ms_adapter.pytorch as torch +from PIL import Image +from functools import lru_cache +from functools import partial +from itertools import repeat +from multiprocessing import Pool +from os import listdir +from os.path import splitext, isfile, join +from pathlib import Path +from ms_adapter.pytorch.utils.data import Dataset +from tqdm import tqdm + + +def load_image(filename): + ext = splitext(filename)[1] + if ext == '.npy': + return Image.fromarray(np.load(filename)) + elif ext in ['.pt', '.pth']: + return Image.fromarray(torch.load(filename).numpy()) + else: + return Image.open(filename) + + +def unique_mask_values(idx, mask_dir, mask_suffix): + mask_file = list(mask_dir.glob(idx + mask_suffix + '.*'))[0] + mask = np.asarray(load_image(mask_file)) + if mask.ndim == 2: + return np.unique(mask) + elif mask.ndim == 3: + mask = mask.reshape(-1, mask.shape[-1]) + return np.unique(mask, axis=0) + else: + raise ValueError(f'Loaded masks should have 2 or 3 dimensions, found {mask.ndim}') + + +class BasicDataset(Dataset): + def __init__(self, images_dir: str, mask_dir: str, scale: float = 1.0, mask_suffix: str = ''): + self.images_dir = Path(images_dir) + self.mask_dir = Path(mask_dir) + assert 0 < scale <= 1, 'Scale must be between 0 and 1' + self.scale = scale + self.mask_suffix = mask_suffix + + self.ids = [splitext(file)[0] for file in listdir(images_dir) if isfile(join(images_dir, file)) and not file.startswith('.')] + if not self.ids: + raise RuntimeError(f'No input file found in {images_dir}, make sure you put your images there') + + logging.info(f'Creating dataset with {len(self.ids)} examples') + logging.info('Scanning mask files to determine unique values') + with Pool() as p: + unique = list(tqdm( + p.imap(partial(unique_mask_values, mask_dir=self.mask_dir, mask_suffix=self.mask_suffix), self.ids), + total=len(self.ids) + )) + + self.mask_values = list(sorted(np.unique(np.concatenate(unique), axis=0).tolist())) + logging.info(f'Unique mask values: {self.mask_values}') + + def __len__(self): + return len(self.ids) + + @staticmethod + def preprocess(mask_values, pil_img, scale, is_mask): + w, h = pil_img.size + newW, newH = int(scale * w), int(scale * h) + assert newW > 0 and newH > 0, 'Scale is too small, resized images would have no pixel' + pil_img = pil_img.resize((newW, newH), resample=Image.NEAREST if is_mask else Image.BICUBIC) + img = np.asarray(pil_img) + + if is_mask: + mask = np.zeros((newH, newW), dtype=np.int64) + for i, v in enumerate(mask_values): + if img.ndim == 2: + mask[img == v] = i + else: + mask[(img == v).all(-1)] = i + + return mask + + else: + if img.ndim == 2: + img = img[np.newaxis, ...] + else: + img = img.transpose((2, 0, 1)) + + if (img > 1).any(): + img = img / 255.0 + + return img + + def __getitem__(self, idx): + name = self.ids[idx] + mask_file = list(self.mask_dir.glob(name + self.mask_suffix + '.*')) + img_file = list(self.images_dir.glob(name + '.*')) + + assert len(img_file) == 1, f'Either no image or multiple images found for the ID {name}: {img_file}' + assert len(mask_file) == 1, f'Either no mask or multiple masks found for the ID {name}: {mask_file}' + mask = load_image(mask_file[0]) + img = load_image(img_file[0]) + + assert img.size == mask.size, \ + f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}' + + img = self.preprocess(self.mask_values, img, self.scale, is_mask=False) + mask = self.preprocess(self.mask_values, mask, self.scale, is_mask=True) + + return { + 'image': torch.as_tensor(img.copy()).float().contiguous(), + 'mask': torch.as_tensor(mask.copy()).long().contiguous() + } + + +class CarvanaDataset(BasicDataset): + def __init__(self, images_dir, mask_dir, scale=1): + super().__init__(images_dir, mask_dir, scale, mask_suffix='_mask') diff --git a/official/cv/unet2d/2dunet-adapter/utils/dice_score.py b/official/cv/unet2d/2dunet-adapter/utils/dice_score.py new file mode 100644 index 0000000..51574f0 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/utils/dice_score.py @@ -0,0 +1,28 @@ +import ms_adapter.pytorch as torch +from ms_adapter.pytorch import Tensor + + +def dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): + # Average of Dice coefficient for all batches, or for a single mask + assert input.size() == target.size() + assert input.dim() == 3 or not reduce_batch_first + + sum_dim = (-1, -2) if input.dim() == 2 or not reduce_batch_first else (-1, -2, -3) + + inter = 2 * (input * target).sum(dim=sum_dim) + sets_sum = input.sum(dim=sum_dim) + target.sum(dim=sum_dim) + sets_sum = torch.where(sets_sum == 0, inter, sets_sum) + + dice = (inter + epsilon) / (sets_sum + epsilon) + return dice.mean() + + +def multiclass_dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): + # Average of Dice coefficient for all classes + return dice_coeff(input.flatten(0, 1), target.flatten(0, 1), reduce_batch_first, epsilon) + + +def dice_loss(input: Tensor, target: Tensor, multiclass: bool = False): + # Dice loss (objective to minimize) between 0 and 1 + fn = multiclass_dice_coeff if multiclass else dice_coeff + return 1 - fn(input, target, reduce_batch_first=True) diff --git a/official/cv/unet2d/2dunet-adapter/utils/utils.py b/official/cv/unet2d/2dunet-adapter/utils/utils.py new file mode 100644 index 0000000..6ed8d8d --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/utils/utils.py @@ -0,0 +1,13 @@ +import matplotlib.pyplot as plt + + +def plot_img_and_mask(img, mask): + classes = mask.max() + 1 + fig, ax = plt.subplots(1, classes + 1) + ax[0].set_title('Input image') + ax[0].imshow(img) + for i in range(classes): + ax[i + 1].set_title(f'Mask (class {i + 1})') + ax[i + 1].imshow(mask == i) + plt.xticks([]), plt.yticks([]) + plt.show() diff --git a/official/cv/unet2d/2dunet-adapter/wandb/debug-cli.root.log b/official/cv/unet2d/2dunet-adapter/wandb/debug-cli.root.log new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/debug-internal.log new file mode 100644 index 0000000..4a38ce3 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/debug-internal.log @@ -0,0 +1,312 @@ +2023-03-10 00:10:05,030 INFO StreamThr :17139 [internal.py:wandb_internal():90] W&B internal server running at pid: 17139, started at: 2023-03-10 00:10:05.028238 +2023-03-10 00:10:05,031 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status +2023-03-10 00:10:05,033 INFO WriterThread:17139 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb +2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: header +2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: run +2023-03-10 00:10:05,174 INFO SenderThread:17139 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/215re05m +2023-03-10 00:10:06,363 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: check_version +2023-03-10 00:10:06,364 INFO SenderThread:17139 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files +2023-03-10 00:10:06,364 INFO SenderThread:17139 [sender.py:_start_run_threads():1081] run started: 215re05m with start time 1678378205.024339 +2023-03-10 00:10:06,364 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: summary_record +2023-03-10 00:10:06,422 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:10:06,422 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: check_version +2023-03-10 00:10:07,368 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:10:11,365 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:11,423 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:11,444 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: run_start +2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():31] System info init +2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():46] System info init done +2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:start():183] Starting system monitor +2023-03-10 00:10:11,449 INFO SystemMonitor:17139 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:probe():204] Collecting system info +2023-03-10 00:10:11,450 INFO SystemMonitor:17139 [interfaces.py:start():187] Started cpu monitoring +2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started disk monitoring +2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started gpu monitoring +2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started memory monitoring +2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started network monitoring +2023-03-10 00:10:11,489 DEBUG HandlerThread:17139 [system_info.py:probe():195] Probing system +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [git.py:repo():40] git repository is invalid +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_info.py:probe():240] Probing system done +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T16:10:11.489682', 'startedAt': '2023-03-09T16:10:04.972918', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.301262500001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.047, 'min': 0.0, 'max': 0.0}, {'current': 2505.411, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2497.435, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.275, 'min': 0.0, 'max': 0.0}, {'current': 2497.673, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2501.929, 'min': 0.0, 'max': 0.0}, {'current': 2500.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.824, 'min': 0.0, 'max': 0.0}, {'current': 2498.743, 'min': 0.0, 'max': 0.0}, {'current': 2498.804, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2501.118, 'min': 0.0, 'max': 0.0}, {'current': 2502.038, 'min': 0.0, 'max': 0.0}, {'current': 2499.891, 'min': 0.0, 'max': 0.0}, {'current': 2500.327, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2499.951, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.433, 'min': 0.0, 'max': 0.0}, {'current': 2501.418, 'min': 0.0, 'max': 0.0}, {'current': 2501.066, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.969, 'min': 0.0, 'max': 0.0}, {'current': 2499.963, 'min': 0.0, 'max': 0.0}, {'current': 2499.319, 'min': 0.0, 'max': 0.0}, {'current': 2499.62, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2502.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2504.876, 'min': 0.0, 'max': 0.0}, {'current': 2499.546, 'min': 0.0, 'max': 0.0}, {'current': 2504.283, 'min': 0.0, 'max': 0.0}, {'current': 2506.421, 'min': 0.0, 'max': 0.0}, {'current': 2499.741, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2498.673, 'min': 0.0, 'max': 0.0}, {'current': 2505.326, 'min': 0.0, 'max': 0.0}, {'current': 2504.107, 'min': 0.0, 'max': 0.0}, {'current': 2505.212, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2509.28, 'min': 0.0, 'max': 0.0}, {'current': 2500.431, 'min': 0.0, 'max': 0.0}, {'current': 2499.538, 'min': 0.0, 'max': 0.0}, {'current': 2500.876, 'min': 0.0, 'max': 0.0}, {'current': 2503.961, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2500.3, 'min': 0.0, 'max': 0.0}, {'current': 2500.458, 'min': 0.0, 'max': 0.0}, {'current': 2499.894, 'min': 0.0, 'max': 0.0}, {'current': 2502.906, 'min': 0.0, 'max': 0.0}, {'current': 2499.885, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2504.513, 'min': 0.0, 'max': 0.0}, {'current': 2504.644, 'min': 0.0, 'max': 0.0}, {'current': 2503.685, 'min': 0.0, 'max': 0.0}, {'current': 2500.655, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.238, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.672, 'min': 0.0, 'max': 0.0}, {'current': 2504.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.59, 'min': 0.0, 'max': 0.0}, {'current': 2500.478, 'min': 0.0, 'max': 0.0}, {'current': 2500.85, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6511535644531}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():214] Finished collecting system info +2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():217] Publishing system info +2023-03-10 00:10:11,517 DEBUG HandlerThread:17139 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-10 00:10:12,268 DEBUG HandlerThread:17139 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-10 00:10:12,302 INFO HandlerThread:17139 [system_monitor.py:probe():219] Finished publishing system info +2023-03-10 00:10:12,305 DEBUG SenderThread:17139 [sender.py:send():336] send: files +2023-03-10 00:10:12,306 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-10 00:10:12,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: stop_status +2023-03-10 00:10:12,333 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: stop_status +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json +2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: telemetry +2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: config +2023-03-10 00:10:13,506 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /tmp/tmpf905ug1_wandb/11iue9v6-wandb-metadata.json +2023-03-10 00:10:13,547 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:10:15,549 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:10:16,804 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:21,805 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:26,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:31,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:32,316 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:36,812 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:37,317 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:37,569 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml +2023-03-10 00:10:42,303 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:42,319 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:47,304 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:47,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:52,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:52,322 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:57,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:57,323 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:02,306 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:02,324 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:07,307 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:07,325 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:11,452 DEBUG SystemMonitor:17139 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-10 00:11:11,458 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:11:12,326 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:12,459 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:17,327 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:17,460 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:22,329 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:22,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:27,330 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:27,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:32,331 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:32,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:37,332 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:37,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:41,460 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:11:42,333 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:43,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:47,334 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:48,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:52,336 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:53,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:57,337 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:58,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:02,338 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:03,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:07,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:08,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:11,463 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:12:12,340 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:14,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:17,342 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:19,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:22,343 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:24,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:27,344 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:29,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:32,345 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:34,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:37,346 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:39,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:41,466 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:12:42,348 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:45,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:47,349 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:50,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:52,350 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:55,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:57,351 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:00,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:02,352 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:05,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:07,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:10,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:11,469 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:13:12,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:16,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:17,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:21,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:22,357 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:26,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:27,358 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:31,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:32,359 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:36,473 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:37,360 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:41,548 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:41,550 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:13:42,362 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:46,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:47,373 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:51,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:52,375 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:56,553 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:57,376 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:01,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:02,377 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:06,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:07,378 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:11,553 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:14:12,379 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:12,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:17,381 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:17,555 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:22,382 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:22,556 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:27,383 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:27,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:32,384 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:32,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:37,386 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:37,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:41,555 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:14:42,387 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:43,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:47,388 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:48,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:52,389 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:53,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:57,390 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:58,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:02,391 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:03,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:07,393 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:08,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:11,558 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:15:12,394 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:14,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:17,409 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:18,419 DEBUG SenderThread:17139 [sender.py:send():336] send: exit +2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():559] handling exit code: 1 +2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():561] handling runtime: 306 +2023-03-10 00:15:18,723 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:15:18,724 INFO SenderThread:17139 [sender.py:send_exit():567] send defer +2023-03-10 00:15:18,724 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,725 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-10 00:15:18,725 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 1 +2023-03-10 00:15:18,726 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,726 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-10 00:15:18,726 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 2 +2023-03-10 00:15:18,808 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,808 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-10 00:15:18,808 INFO HandlerThread:17139 [system_monitor.py:finish():193] Stopping system monitor +2023-03-10 00:15:19,020 DEBUG SystemMonitor:17139 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-10 00:15:19,021 DEBUG SystemMonitor:17139 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined cpu monitor +2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined disk monitor +2023-03-10 00:15:19,110 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:20,110 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined gpu monitor +2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined memory monitor +2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined network monitor +2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:20,113 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 3 +2023-03-10 00:15:20,114 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,114 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-10 00:15:20,208 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:15:20,209 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:20,210 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 4 +2023-03-10 00:15:20,211 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:20,211 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-10 00:15:20,211 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,211 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-10 00:15:20,212 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 5 +2023-03-10 00:15:20,212 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,212 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-10 00:15:20,212 DEBUG SenderThread:17139 [sender.py:send():336] send: summary +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:15:20,233 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 6 +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-10 00:15:20,234 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 7 +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-10 00:15:20,235 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,235 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-10 00:15:20,511 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:21,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:23,339 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 8 +2023-03-10 00:15:23,339 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:23,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:23,340 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-10 00:15:23,340 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:23,340 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 9 +2023-03-10 00:15:23,341 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:23,341 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-10 00:15:23,341 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-10 00:15:23,341 INFO SenderThread:17139 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-10 00:15:24,213 INFO SenderThread:17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml conda-environment.yaml +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log output.log +2023-03-10 00:15:24,215 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt requirements.txt +2023-03-10 00:15:24,220 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml config.yaml +2023-03-10 00:15:24,241 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json wandb-summary.json +2023-03-10 00:15:24,256 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json wandb-metadata.json +2023-03-10 00:15:24,261 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 10 +2023-03-10 00:15:24,277 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:24,287 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-10 00:15:24,419 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:24,424 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-10 00:15:24,424 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher +2023-03-10 00:15:25,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:25,536 INFO wandb-upload_4:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:25,612 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:25,889 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:25,909 INFO wandb-upload_2:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt +2023-03-10 00:15:26,005 INFO wandb-upload_3:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml +2023-03-10 00:15:26,205 INFO Thread-12 :17139 [sender.py:transition_state():587] send defer: 11 +2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-10 00:15:26,206 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-10 00:15:26,206 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher +2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 12 +2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-10 00:15:26,207 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:26,207 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-10 00:15:28,352 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 13 +2023-03-10 00:15:28,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:28,353 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-10 00:15:28,353 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 14 +2023-03-10 00:15:28,354 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:28,354 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: final +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: footer +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:28,354 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-10 00:15:28,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:28,355 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:28,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: server_info +2023-03-10 00:15:28,356 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: server_info +2023-03-10 00:15:28,392 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: get_summary +2023-03-10 00:15:28,438 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-10 00:15:29,089 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: shutdown +2023-03-10 00:15:29,090 INFO HandlerThread:17139 [handler.py:finish():842] shutting down handler +2023-03-10 00:15:29,356 INFO WriterThread:17139 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb +2023-03-10 00:15:30,089 INFO SenderThread:17139 [sender.py:finish():1504] shutting down sender +2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher +2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-adapter/wandb/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/debug.log new file mode 100644 index 0000000..a86b536 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/debug.log @@ -0,0 +1,28 @@ +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Configure stats pid to 15417 +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():546] calling init triggers +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():602] starting backend +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():606] setting up manager +2023-03-10 00:10:05,022 INFO MainThread:15417 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-10 00:10:05,024 INFO MainThread:15417 [wandb_init.py:init():613] backend started and connected +2023-03-10 00:10:05,028 INFO MainThread:15417 [wandb_init.py:init():701] updated telemetry +2023-03-10 00:10:05,029 INFO MainThread:15417 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-10 00:10:06,363 INFO MainThread:15417 [wandb_run.py:_on_init():2131] communicating current version +2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_run.py:_on_init():2140] got version response +2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_init.py:init():789] starting run threads in backend +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_console_start():2112] atexit reg +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-10 00:10:12,314 INFO MainThread:15417 [wandb_init.py:init():831] run started, returning control to user process +2023-03-10 00:10:12,315 INFO MainThread:15417 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} +2023-03-10 00:15:30,109 WARNING MsgRouterThr:15417 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml new file mode 100644 index 0000000..083d3a5 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml @@ -0,0 +1,40 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678374242.245928 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/output.log new file mode 100644 index 0000000..59c115b --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/output.log @@ -0,0 +1,6 @@ +Traceback (most recent call last): + File "train.py", line 170, in + img_scale=args.scale, + File "train.py", line 68, in train_model + ''') +NameError: name 'n_val' is not defined \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json new file mode 100644 index 0000000..45a5ed8 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:04:08.672479", + "startedAt": "2023-03-09T15:04:02.165764", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.5035375, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2501.774, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.632, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.844, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.227, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.25, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.66, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.269, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.584, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.412, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.957, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.437, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.395, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.827, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.381, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.589, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.571, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.823, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.653, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2494.369, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.905, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.341, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.224, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.123, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.567, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.922, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.487, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.288, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.937, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.938, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.099, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.616, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.366, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.679, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.537, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.211, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.78, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.362, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.673, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.016, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.354, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.742, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.466, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.896, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.131, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.496, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.341, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.252, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.01, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.34, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.367, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.053, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.115, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.91, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.852, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.801, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.094, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.488, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.156, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.755, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.669, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.714, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.401, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.973, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.132, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.65414810180664 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log new file mode 100644 index 0000000..7a98e5a --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log @@ -0,0 +1,209 @@ +2023-03-09 23:04:02,252 INFO StreamThr :5402 [internal.py:wandb_internal():90] W&B internal server running at pid: 5402, started at: 2023-03-09 23:04:02.250393 +2023-03-09 23:04:02,254 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:04:02,255 INFO WriterThread:5402 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb +2023-03-09 23:04:02,261 DEBUG SenderThread:5402 [sender.py:send():336] send: header +2023-03-09 23:04:02,261 DEBUG SenderThread:5402 [sender.py:send():336] send: run +2023-03-09 23:04:02,318 INFO SenderThread:5402 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/tfw61kps +2023-03-09 23:04:03,555 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:04:03,556 INFO SenderThread:5402 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files +2023-03-09 23:04:03,556 INFO SenderThread:5402 [sender.py:_start_run_threads():1081] run started: tfw61kps with start time 1678374242.245928 +2023-03-09 23:04:03,556 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:04:03,606 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:04:03,606 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:04:04,560 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json +2023-03-09 23:04:08,556 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:04:08,608 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:04:08,626 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:04:08,633 DEBUG HandlerThread:5402 [system_info.py:__init__():31] System info init +2023-03-09 23:04:08,633 DEBUG HandlerThread:5402 [system_info.py:__init__():46] System info init done +2023-03-09 23:04:08,633 INFO HandlerThread:5402 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:04:08,634 INFO SystemMonitor:5402 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:04:08,634 INFO HandlerThread:5402 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:04:08,634 INFO SystemMonitor:5402 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:04:08,635 INFO SystemMonitor:5402 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:04:08,635 INFO SystemMonitor:5402 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:04:08,636 INFO SystemMonitor:5402 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:04:08,636 INFO SystemMonitor:5402 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:04:08,672 DEBUG HandlerThread:5402 [system_info.py:probe():195] Probing system +2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [git.py:repo():40] git repository is invalid +2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_info.py:probe():240] Probing system done +2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:04:08.672479', 'startedAt': '2023-03-09T15:04:02.165764', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5035375, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2501.774, 'min': 0.0, 'max': 0.0}, {'current': 2499.632, 'min': 0.0, 'max': 0.0}, {'current': 2500.844, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.227, 'min': 0.0, 'max': 0.0}, {'current': 2498.25, 'min': 0.0, 'max': 0.0}, {'current': 2498.66, 'min': 0.0, 'max': 0.0}, {'current': 2501.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.584, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.412, 'min': 0.0, 'max': 0.0}, {'current': 2500.957, 'min': 0.0, 'max': 0.0}, {'current': 2500.437, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.381, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2499.571, 'min': 0.0, 'max': 0.0}, {'current': 2500.823, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2494.369, 'min': 0.0, 'max': 0.0}, {'current': 2499.905, 'min': 0.0, 'max': 0.0}, {'current': 2500.341, 'min': 0.0, 'max': 0.0}, {'current': 2500.224, 'min': 0.0, 'max': 0.0}, {'current': 2500.123, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2499.922, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.487, 'min': 0.0, 'max': 0.0}, {'current': 2500.288, 'min': 0.0, 'max': 0.0}, {'current': 2499.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.938, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2501.366, 'min': 0.0, 'max': 0.0}, {'current': 2499.679, 'min': 0.0, 'max': 0.0}, {'current': 2505.537, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2505.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.78, 'min': 0.0, 'max': 0.0}, {'current': 2504.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2505.673, 'min': 0.0, 'max': 0.0}, {'current': 2499.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.354, 'min': 0.0, 'max': 0.0}, {'current': 2503.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.466, 'min': 0.0, 'max': 0.0}, {'current': 2500.896, 'min': 0.0, 'max': 0.0}, {'current': 2500.131, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.496, 'min': 0.0, 'max': 0.0}, {'current': 2505.341, 'min': 0.0, 'max': 0.0}, {'current': 2505.252, 'min': 0.0, 'max': 0.0}, {'current': 2505.01, 'min': 0.0, 'max': 0.0}, {'current': 2506.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2503.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.115, 'min': 0.0, 'max': 0.0}, {'current': 2505.91, 'min': 0.0, 'max': 0.0}, {'current': 2504.027, 'min': 0.0, 'max': 0.0}, {'current': 2501.852, 'min': 0.0, 'max': 0.0}, {'current': 2504.801, 'min': 0.0, 'max': 0.0}, {'current': 2499.094, 'min': 0.0, 'max': 0.0}, {'current': 2504.488, 'min': 0.0, 'max': 0.0}, {'current': 2504.156, 'min': 0.0, 'max': 0.0}, {'current': 2497.755, 'min': 0.0, 'max': 0.0}, {'current': 2504.669, 'min': 0.0, 'max': 0.0}, {'current': 2504.714, 'min': 0.0, 'max': 0.0}, {'current': 2504.401, 'min': 0.0, 'max': 0.0}, {'current': 2504.973, 'min': 0.0, 'max': 0.0}, {'current': 2501.132, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.65414810180664}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:04:08,697 INFO HandlerThread:5402 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:04:08,697 INFO HandlerThread:5402 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:04:08,697 DEBUG HandlerThread:5402 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:04:08,730 DEBUG HandlerThread:5402 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:04:08,731 DEBUG HandlerThread:5402 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:04:09,530 DEBUG HandlerThread:5402 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:04:09,551 INFO HandlerThread:5402 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:04:09,555 DEBUG SenderThread:5402 [sender.py:send():336] send: files +2023-03-09 23:04:09,556 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:04:09,574 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:04:09,607 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:04:09,801 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml +2023-03-09 23:04:09,802 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt +2023-03-09 23:04:09,802 INFO Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json +2023-03-09 23:04:10,151 DEBUG SenderThread:5402 [sender.py:send():336] send: telemetry +2023-03-09 23:04:10,156 DEBUG SenderThread:5402 [sender.py:send():336] send: config +2023-03-09 23:04:10,168 DEBUG SenderThread:5402 [sender.py:send():336] send: exit +2023-03-09 23:04:10,168 INFO SenderThread:5402 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:04:10,168 INFO SenderThread:5402 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:04:10,227 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:04:10,237 INFO SenderThread:5402 [sender.py:send_exit():567] send defer +2023-03-09 23:04:10,253 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,258 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:04:10,278 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,283 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:04:10,283 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:04:10,283 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,283 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:04:10,284 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,284 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:04:10,284 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:04:10,284 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,284 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:04:10,284 INFO HandlerThread:5402 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:04:10,285 DEBUG SystemMonitor:5402 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:04:10,286 DEBUG SystemMonitor:5402 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:04:10,286 DEBUG SystemMonitor:5402 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:04:10,288 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:04:10,288 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:04:10,302 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:04:10,303 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:04:10,303 INFO HandlerThread:5402 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:04:10,304 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,304 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:04:10,304 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:04:10,305 DEBUG SenderThread:5402 [sender.py:send():336] send: stats +2023-03-09 23:04:10,305 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,305 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:04:10,306 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:04:10,306 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,306 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:04:10,306 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:04:10,306 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:04:10,307 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,307 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:04:10,307 DEBUG SenderThread:5402 [sender.py:send():336] send: summary +2023-03-09 23:04:10,334 INFO SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:04:10,335 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:04:10,335 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,335 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:04:10,335 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,335 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:04:10,341 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:04:10,671 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:10,767 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:04:10,767 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:10,767 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:10,767 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:04:10,768 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:10,768 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:04:10,784 INFO Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml +2023-03-09 23:04:10,784 INFO Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json +2023-03-09 23:04:10,863 INFO wandb-upload_0:5402 [upload_job.py:push():138] Uploaded file /tmp/tmpbm6fg7powandb/0dy5vwa2-wandb-metadata.json +2023-03-09 23:04:11,672 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:12,197 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:04:12,202 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:12,203 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:12,208 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:04:12,229 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:12,229 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:04:12,265 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:04:12,280 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:12,280 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:04:12,286 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:12,286 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:04:12,286 INFO SenderThread:5402 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:04:12,684 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:12,786 INFO SenderThread:5402 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files +2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt requirements.txt +2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml config.yaml +2023-03-09 23:04:12,787 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json wandb-summary.json +2023-03-09 23:04:12,792 INFO SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:04:12,793 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:04:12,793 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:12,818 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:12,818 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:04:12,940 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:12,946 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:04:12,946 INFO SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:04:13,726 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:13,747 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:13,947 INFO wandb-upload_3:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json +2023-03-09 23:04:14,378 INFO wandb-upload_2:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml +2023-03-09 23:04:14,732 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:14,733 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:15,734 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:15,734 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:16,734 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:16,735 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:17,736 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:17,736 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:18,736 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:18,737 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:19,244 ERROR wandb-upload_1:5402 [internal_api.py:execute():245] 502 response executing GraphQL. +2023-03-09 23:04:19,245 ERROR wandb-upload_1:5402 [internal_api.py:execute():246] + + +502 Server Error + + +

Error: Server Error

+

The server encountered a temporary error and could not complete your request.

Please try again in 30 seconds.

+

+ + +2023-03-09 23:04:19,738 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:04:19,738 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:19,739 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:20,739 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:20,740 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:21,423 INFO wandb-upload_1:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt +2023-03-09 23:04:21,623 INFO Thread-12 :5402 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:04:21,623 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:21,623 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:04:21,624 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:04:21,624 INFO SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:04:21,624 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:21,624 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:04:21,624 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:21,624 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:04:21,809 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:22,109 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:04:22,109 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:22,109 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:22,110 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:04:22,110 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:22,110 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:04:22,110 INFO SenderThread:5402 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:04:22,110 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:04:22,111 INFO HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send():336] send: final +2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send():336] send: footer +2023-03-09 23:04:22,111 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: defer +2023-03-09 23:04:22,111 INFO SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:04:22,112 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:22,112 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:22,113 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:04:22,113 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:04:22,113 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:04:22,114 DEBUG SenderThread:5402 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:04:22,150 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:04:22,165 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:04:23,210 INFO MainThread:5402 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:04:23,211 DEBUG HandlerThread:5402 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:04:23,212 INFO HandlerThread:5402 [handler.py:finish():842] shutting down handler +2023-03-09 23:04:24,208 INFO WriterThread:5402 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb +2023-03-09 23:04:24,209 INFO SenderThread:5402 [sender.py:finish():1504] shutting down sender +2023-03-09 23:04:24,210 INFO SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:04:24,210 INFO SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log new file mode 100644 index 0000000..7c11276 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log @@ -0,0 +1,29 @@ +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Configure stats pid to 3604 +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():602] starting backend +2023-03-09 23:04:02,230 INFO MainThread:3604 [wandb_init.py:init():606] setting up manager +2023-03-09 23:04:02,242 INFO MainThread:3604 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:04:02,245 INFO MainThread:3604 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:04:02,249 INFO MainThread:3604 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:04:02,251 INFO MainThread:3604 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:04:03,555 INFO MainThread:3604 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:04:08,608 INFO MainThread:3604 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:04:08,608 INFO MainThread:3604 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:04:09,562 INFO MainThread:3604 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:04:09,563 INFO MainThread:3604 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:04:09,564 INFO MainThread:3604 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} +2023-03-09 23:04:24,411 WARNING MsgRouterThr:3604 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb new file mode 100644 index 0000000000000000000000000000000000000000..71079b7d0eae5630de48872691a8589715c3e6dc GIT binary patch literal 3921 zcmbW4UuYaf9LF=s<#L%`noFWaH!UnzHGfE!{l9lXE%c8fYG15~f}DH1lVmS@yO-U~ zrKt#dDAYoI=u@?Vu|BlwgCHn;kT`ns~)KsZhDB8AGEL-@as+Nw*Q;`|Yi&NrhuqU&+ zesLw8`+DWV;=AAg?9PD8U4ZMl3_w1u2!cxr8CH~@Q{>TD^rOc&qYB?{Wzm-x>U*>@aVV!EOXW;1IW*l&Dsrjt8xJTNU( zHm!Mntm=mmA0P{P5szncdDjf1@o9MX^Bq2S^u8l_zMjfnGT-;B-=4u6?+&)@7P|I> zYyZB5Q)Md8l2n)8{~4ElnCBmNT*T+2z;ryJww#~jUB^SbWQdczVu3o-&5~qTdBgEQ`4}Qni)a|V4g_g6H z;}W)q+?Kh~(b4f;c#%JRW3}mJPtwb8H+SR3HQcC*Q)Md#@p9_Tzp5lSF$a{Xo+h_(an!Sti*RL*o(k#C(QT|UqHs0RFE<65F#@UBc?QMWu zK09}!8dMGKgj+AlW8f@EWk3#$bLXF4=>?ho>GMO_;(Duzu3frv3UK$ptcRi{KbRLn z+&rx=6_ca5sp^}fdtjGUTipEe035(y1W~~MOEnZUTMtW;Aaa|M#7H{FZA8girIe%z zT02bHs^L>ZZcFO+l152NoKUBvL_%YGsX$3`Y^+F0qENub2Aen}nuj7ZO>by4kYpT1 z1iMyA98E+L45HCUnq{`|#S5=cVDP3O z9u%5OQe(NuZSM-ph199+z(rxXkcC%RE^(bZa8Xz;Bu=HdBsQg6mJ9K&?!ZN5 zxx{fSmsq2@B$`v*fs0CWkp!i^Mk>ukjvHQMxsU@!qq#_NA89NX!c1eikhRrlE=s&V zHJVFe?`kxc#QJG07s6Acxg_IgE{TIlXSoo6I?W}~w>ry(9IiUch3rqA=91_?o#vv& zeXG-4ibAoy?mEpyiXR?2%Y`u0X)atN7a zk{Zn=sj*z*I$16%Hw<&Nz_*%@a3PutkZI!|Je>uN>w_uV3Fq-|7koyF+(U4!#zHWfwhjr~!HvL)OHiAQ!p4FuidHe)nm%#(p@E3=6`r8PFpcMMF?zZXX;< v%H+K%g}1q}_UWy&Df}jv<2v^LAp{U#4&d6xzy;9p=a&clZagsf8R+;AF>YTG literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml new file mode 100644 index 0000000..b3ecb08 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml @@ -0,0 +1,40 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678374396.698895 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log new file mode 100644 index 0000000..01e492d --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log @@ -0,0 +1,14 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Device: cuda + Images scaling: 0.5 +Traceback (most recent call last): + File "train.py", line 169, in + img_scale=args.scale, + File "train.py", line 70, in train_model + optimizer = ms.nn.RMSprop(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay) +AttributeError: module 'mindspore.nn' has no attribute 'RMSprop' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json new file mode 100644 index 0000000..441adae --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:06:43.530245", + "startedAt": "2023-03-09T15:06:36.660035", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2499.997949999998, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.979, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.99, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.988, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.985, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.987, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.981, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.985, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.984, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.012, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.006, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.008, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.006, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.981, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.987, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.049, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.006, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.984, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.063, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.004, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.014, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6544303894043 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log new file mode 100644 index 0000000..ce6f366 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log @@ -0,0 +1,177 @@ +2023-03-09 23:06:36,705 INFO StreamThr :7419 [internal.py:wandb_internal():90] W&B internal server running at pid: 7419, started at: 2023-03-09 23:06:36.703095 +2023-03-09 23:06:36,706 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:06:36,708 INFO WriterThread:7419 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb +2023-03-09 23:06:36,712 DEBUG SenderThread:7419 [sender.py:send():336] send: header +2023-03-09 23:06:36,713 DEBUG SenderThread:7419 [sender.py:send():336] send: run +2023-03-09 23:06:36,804 INFO SenderThread:7419 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/3p0zeu2z +2023-03-09 23:06:38,422 INFO SenderThread:7419 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files +2023-03-09 23:06:38,422 INFO SenderThread:7419 [sender.py:_start_run_threads():1081] run started: 3p0zeu2z with start time 1678374396.698895 +2023-03-09 23:06:38,423 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:06:38,426 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:06:38,444 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:06:38,444 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:06:39,426 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json +2023-03-09 23:06:43,427 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:06:43,446 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:06:43,504 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:06:43,509 DEBUG HandlerThread:7419 [system_info.py:__init__():31] System info init +2023-03-09 23:06:43,509 DEBUG HandlerThread:7419 [system_info.py:__init__():46] System info init done +2023-03-09 23:06:43,509 INFO HandlerThread:7419 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:06:43,510 INFO SystemMonitor:7419 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:06:43,510 INFO HandlerThread:7419 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:06:43,510 INFO SystemMonitor:7419 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:06:43,511 INFO SystemMonitor:7419 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:06:43,511 INFO SystemMonitor:7419 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:06:43,512 INFO SystemMonitor:7419 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:06:43,512 INFO SystemMonitor:7419 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:06:43,530 DEBUG HandlerThread:7419 [system_info.py:probe():195] Probing system +2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [git.py:repo():40] git repository is invalid +2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_info.py:probe():240] Probing system done +2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:06:43.530245', 'startedAt': '2023-03-09T15:06:36.660035', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2499.997949999998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.979, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.049, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.014, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6544303894043}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:06:43,549 INFO HandlerThread:7419 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:06:43,549 INFO HandlerThread:7419 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:06:43,549 DEBUG HandlerThread:7419 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:06:43,576 DEBUG HandlerThread:7419 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:06:43,577 DEBUG HandlerThread:7419 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:06:44,293 DEBUG HandlerThread:7419 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:06:44,346 INFO HandlerThread:7419 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:06:44,349 DEBUG SenderThread:7419 [sender.py:send():336] send: files +2023-03-09 23:06:44,349 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:06:44,368 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:06:44,384 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml +2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json +2023-03-09 23:06:44,593 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt +2023-03-09 23:06:44,866 DEBUG SenderThread:7419 [sender.py:send():336] send: telemetry +2023-03-09 23:06:44,871 DEBUG SenderThread:7419 [sender.py:send():336] send: config +2023-03-09 23:06:44,914 DEBUG SenderThread:7419 [sender.py:send():336] send: exit +2023-03-09 23:06:44,920 INFO SenderThread:7419 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:06:44,920 INFO SenderThread:7419 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:06:45,056 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:06:45,067 INFO SenderThread:7419 [sender.py:send_exit():567] send defer +2023-03-09 23:06:45,068 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,069 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:06:45,069 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,069 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:06:45,069 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:06:45,069 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,069 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:06:45,070 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,070 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:06:45,070 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:06:45,070 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,070 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:06:45,070 INFO HandlerThread:7419 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:06:45,070 DEBUG SystemMonitor:7419 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:06:45,071 DEBUG SystemMonitor:7419 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:06:45,071 DEBUG SystemMonitor:7419 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:06:45,072 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:06:45,072 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:06:45,086 INFO HandlerThread:7419 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:06:45,087 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,087 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:06:45,087 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:06:45,087 DEBUG SenderThread:7419 [sender.py:send():336] send: stats +2023-03-09 23:06:45,087 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,087 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:06:45,088 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,088 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:06:45,088 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:06:45,088 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,088 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:06:45,089 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,089 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:06:45,089 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:06:45,089 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,089 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:06:45,089 DEBUG SenderThread:7419 [sender.py:send():336] send: summary +2023-03-09 23:06:45,107 INFO SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:06:45,108 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:06:45,108 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,108 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:06:45,108 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,108 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:06:45,114 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:06:45,573 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:06:45,573 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:45,573 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:06:45,574 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:45,574 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:06:45,591 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json +2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml +2023-03-09 23:06:45,595 INFO Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log +2023-03-09 23:06:45,608 INFO wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /tmp/tmph6vgygn9wandb/2ibjar1l-wandb-metadata.json +2023-03-09 23:06:46,926 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:06:46,927 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:06:46,927 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:46,927 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:06:46,928 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:46,928 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:06:46,928 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:06:46,928 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:46,928 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:06:46,928 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:46,929 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:06:46,929 INFO SenderThread:7419 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:06:47,597 INFO SenderThread:7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log +2023-03-09 23:06:47,597 INFO SenderThread:7419 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files +2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log output.log +2023-03-09 23:06:47,598 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt requirements.txt +2023-03-09 23:06:47,603 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml config.yaml +2023-03-09 23:06:47,629 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json wandb-summary.json +2023-03-09 23:06:47,720 INFO SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:06:47,720 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:06:47,857 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:47,858 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:06:47,891 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:47,896 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:06:47,896 INFO SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:06:48,926 INFO wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log +2023-03-09 23:06:49,240 INFO wandb-upload_3:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml +2023-03-09 23:06:49,315 INFO wandb-upload_4:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json +2023-03-09 23:06:50,115 INFO wandb-upload_2:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt +2023-03-09 23:06:50,316 INFO Thread-12 :7419 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:06:50,316 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:50,316 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:06:50,317 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:50,317 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:06:50,317 INFO SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:06:50,317 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:06:50,317 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:50,317 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:06:50,317 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:50,318 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:06:50,626 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:06:50,662 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:06:50,662 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:50,662 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:06:50,662 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:50,663 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:06:50,663 INFO SenderThread:7419 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:06:50,663 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:06:50,663 INFO HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:06:50,663 DEBUG SenderThread:7419 [sender.py:send():336] send: final +2023-03-09 23:06:50,664 DEBUG SenderThread:7419 [sender.py:send():336] send: footer +2023-03-09 23:06:50,664 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: defer +2023-03-09 23:06:50,664 INFO SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:06:50,664 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:06:50,665 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:06:50,665 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:06:50,666 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:06:50,666 DEBUG SenderThread:7419 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:06:50,717 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:06:51,503 INFO MainThread:7419 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:06:51,504 DEBUG HandlerThread:7419 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:06:51,504 INFO HandlerThread:7419 [handler.py:finish():842] shutting down handler +2023-03-09 23:06:51,666 INFO WriterThread:7419 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb +2023-03-09 23:06:52,503 INFO SenderThread:7419 [sender.py:finish():1504] shutting down sender +2023-03-09 23:06:52,503 INFO SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:06:52,503 INFO SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log new file mode 100644 index 0000000..35dbef8 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Configure stats pid to 5620 +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:06:36,685 INFO MainThread:5620 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log +2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log +2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():602] starting backend +2023-03-09 23:06:36,686 INFO MainThread:5620 [wandb_init.py:init():606] setting up manager +2023-03-09 23:06:36,697 INFO MainThread:5620 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:06:36,698 INFO MainThread:5620 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:06:36,703 INFO MainThread:5620 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:06:36,704 INFO MainThread:5620 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:06:38,425 INFO MainThread:5620 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:06:43,446 INFO MainThread:5620 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:06:43,447 INFO MainThread:5620 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:06:44,356 INFO MainThread:5620 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:06:44,356 INFO MainThread:5620 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:06:44,357 INFO MainThread:5620 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:06:44,357 INFO MainThread:5620 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:06:44,358 INFO MainThread:5620 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:06:44,358 INFO MainThread:5620 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} +2023-03-09 23:06:52,524 WARNING MsgRouterThr:5620 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb new file mode 100644 index 0000000000000000000000000000000000000000..8cc6fc66939c3d417b859f2eb854434dac377cfa GIT binary patch literal 2100 zcmbW2UuYaf9LI0+=dzhzl1rkEo)yL`HJ6aAyLY>Hxx=(HZ3;!(Qd1E`d(Q4o?)Gx~ zhuyivL_~U3O6rSL14a6f+6To{ihYngXbD0U6{Y%;A}FYc#cB~j^Pm)GH<#_@LE7Hk z%gp@d_viDS-^}KodU))IOQA;=SGY_FaG(m}F_g+?6)`EvNkK|U;-spGSxr@CNfA?$ zAWvrA1)+~45OOJ7m?M?s+>K9mzVRvZZmI&|_TKSGVI-|vX~o9IZ2BZP25v=)wLUp~ zkkEJ+?116?3^p~T3YuPcv6Lt(liEl+sicS?OJa5;t7&ppQQ2RT)HG3sDR!rXVc{f* zaQ|L^dodh6y>Rx+rZ>TM*vf(BW)N#$1|S|z@flte2ipW*OtHaOOB5u)CYooh&$tH| zO>?pj-V1yEB8Ck@v4$z;ORi;J{$+mYIkuza>A_m?p4;cRBV{)g(l-A5)-x)<4e zz%a2bwdhs>kYN-;1yi+{cl9~KRz%nbJIVw*rfwGU4yJ6eRV2fL4DW+oE}kZNwMf({ z+tN*11ufL65ZDddb)%4XRa|DfuAkcU?@i-GC!8Yl8NJ?lE?MwBD9 zG=Z}$jEn@%djc8U^}t6wDK>ofl4)Jt$mlJq6%7a0{RS=&7kPqt7zi=Rd?UPj;m}LF zcIfPE+e)cGw<|ZlV7hH*04hZm|A!&?quo-Zc3q4oujNWhZRxV8b2txAOjkP~y0W z0j5^{9Uk2@8$yOE` z&?6gp)-~<#k=ez{YO`*Uu9Q)y0*Fl_OWJB zd4m|1Gt1*?te3G;CI`l_2)pau9M|<6+Qdv&=4C0i z4ffWXZQ^)u3;QAnyp;L=@~+ogSmE8V(ACv*k3lx + img_scale=args.scale, + File "train.py", line 88, in train_model + grad_fn = ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True) +NameError: name 'ops' is not defined \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json new file mode 100644 index 0000000..24f138c --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:10:50.243518", + "startedAt": "2023-03-09T15:10:43.500810", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.419575, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.322, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.664, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.917, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.586, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.207, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.024, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.054, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.294, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.793, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.641, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.912, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.087, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.927, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.004, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.109, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.008, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.666, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.422, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.34, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.446, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.667, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.071, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.954, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.346, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.336, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.318, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.379, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.717, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.668, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.274, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.083, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.294, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.426, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.706, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.578, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.8, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.034, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.781, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.461, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.789, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.345, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.362, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.448, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.053, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.154, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.213, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.574, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.754, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.345, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.409, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.953, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.981, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.045, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.47, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.703, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.13, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.197, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.917, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.623, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.99, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.272, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.349, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.464, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.376, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.544, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.74, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.053, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6549377441406 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log new file mode 100644 index 0000000..c565b74 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log @@ -0,0 +1,183 @@ +2023-03-09 23:10:43,546 INFO StreamThr :9187 [internal.py:wandb_internal():90] W&B internal server running at pid: 9187, started at: 2023-03-09 23:10:43.544652 +2023-03-09 23:10:43,548 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:10:43,549 INFO WriterThread:9187 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb +2023-03-09 23:10:43,554 DEBUG SenderThread:9187 [sender.py:send():336] send: header +2023-03-09 23:10:43,554 DEBUG SenderThread:9187 [sender.py:send():336] send: run +2023-03-09 23:10:43,617 INFO SenderThread:9187 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/osho06z5 +2023-03-09 23:10:45,099 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:10:45,100 INFO SenderThread:9187 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files +2023-03-09 23:10:45,100 INFO SenderThread:9187 [sender.py:_start_run_threads():1081] run started: osho06z5 with start time 1678374643.540464 +2023-03-09 23:10:45,100 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:10:45,131 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:10:45,132 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:10:46,104 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json +2023-03-09 23:10:50,100 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:10:50,133 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:10:50,192 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:10:50,205 DEBUG HandlerThread:9187 [system_info.py:__init__():31] System info init +2023-03-09 23:10:50,205 DEBUG HandlerThread:9187 [system_info.py:__init__():46] System info init done +2023-03-09 23:10:50,205 INFO HandlerThread:9187 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:10:50,206 INFO SystemMonitor:9187 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:10:50,206 INFO HandlerThread:9187 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:10:50,206 INFO SystemMonitor:9187 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:10:50,207 INFO SystemMonitor:9187 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:10:50,207 INFO SystemMonitor:9187 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:10:50,208 INFO SystemMonitor:9187 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:10:50,208 INFO SystemMonitor:9187 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:10:50,243 DEBUG HandlerThread:9187 [system_info.py:probe():195] Probing system +2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [git.py:repo():40] git repository is invalid +2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [system_info.py:probe():240] Probing system done +2023-03-09 23:10:50,266 DEBUG HandlerThread:9187 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:10:50.243518', 'startedAt': '2023-03-09T15:10:43.500810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.419575, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.664, 'min': 0.0, 'max': 0.0}, {'current': 2499.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.586, 'min': 0.0, 'max': 0.0}, {'current': 2500.207, 'min': 0.0, 'max': 0.0}, {'current': 2500.024, 'min': 0.0, 'max': 0.0}, {'current': 2500.054, 'min': 0.0, 'max': 0.0}, {'current': 2501.294, 'min': 0.0, 'max': 0.0}, {'current': 2497.793, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.641, 'min': 0.0, 'max': 0.0}, {'current': 2503.912, 'min': 0.0, 'max': 0.0}, {'current': 2502.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2502.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2503.109, 'min': 0.0, 'max': 0.0}, {'current': 2501.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.666, 'min': 0.0, 'max': 0.0}, {'current': 2501.422, 'min': 0.0, 'max': 0.0}, {'current': 2500.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.446, 'min': 0.0, 'max': 0.0}, {'current': 2499.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.071, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.954, 'min': 0.0, 'max': 0.0}, {'current': 2500.346, 'min': 0.0, 'max': 0.0}, {'current': 2500.336, 'min': 0.0, 'max': 0.0}, {'current': 2501.318, 'min': 0.0, 'max': 0.0}, {'current': 2500.379, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.668, 'min': 0.0, 'max': 0.0}, {'current': 2499.274, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.083, 'min': 0.0, 'max': 0.0}, {'current': 2499.294, 'min': 0.0, 'max': 0.0}, {'current': 2505.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.706, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.578, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.8, 'min': 0.0, 'max': 0.0}, {'current': 2499.034, 'min': 0.0, 'max': 0.0}, {'current': 2504.781, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2500.789, 'min': 0.0, 'max': 0.0}, {'current': 2501.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.448, 'min': 0.0, 'max': 0.0}, {'current': 2499.053, 'min': 0.0, 'max': 0.0}, {'current': 2503.154, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.213, 'min': 0.0, 'max': 0.0}, {'current': 2504.574, 'min': 0.0, 'max': 0.0}, {'current': 2506.754, 'min': 0.0, 'max': 0.0}, {'current': 2500.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.409, 'min': 0.0, 'max': 0.0}, {'current': 2504.953, 'min': 0.0, 'max': 0.0}, {'current': 2501.981, 'min': 0.0, 'max': 0.0}, {'current': 2500.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.47, 'min': 0.0, 'max': 0.0}, {'current': 2500.703, 'min': 0.0, 'max': 0.0}, {'current': 2503.13, 'min': 0.0, 'max': 0.0}, {'current': 2500.197, 'min': 0.0, 'max': 0.0}, {'current': 2505.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.99, 'min': 0.0, 'max': 0.0}, {'current': 2503.272, 'min': 0.0, 'max': 0.0}, {'current': 2502.349, 'min': 0.0, 'max': 0.0}, {'current': 2504.464, 'min': 0.0, 'max': 0.0}, {'current': 2503.376, 'min': 0.0, 'max': 0.0}, {'current': 2503.997, 'min': 0.0, 'max': 0.0}, {'current': 2505.544, 'min': 0.0, 'max': 0.0}, {'current': 2505.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6549377441406}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:10:50,267 INFO HandlerThread:9187 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:10:50,267 INFO HandlerThread:9187 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:10:50,267 DEBUG HandlerThread:9187 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:10:50,309 DEBUG HandlerThread:9187 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:10:50,309 DEBUG HandlerThread:9187 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:10:51,019 DEBUG HandlerThread:9187 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:10:51,065 INFO HandlerThread:9187 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:10:51,068 DEBUG SenderThread:9187 [sender.py:send():336] send: files +2023-03-09 23:10:51,068 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:10:51,084 DEBUG SenderThread:9187 [sender.py:send():336] send: telemetry +2023-03-09 23:10:51,084 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:10:51,094 DEBUG SenderThread:9187 [sender.py:send():336] send: config +2023-03-09 23:10:51,130 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:10:51,233 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json +2023-03-09 23:10:51,262 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml +2023-03-09 23:10:51,268 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt +2023-03-09 23:10:51,721 DEBUG SenderThread:9187 [sender.py:send():336] send: exit +2023-03-09 23:10:51,721 INFO SenderThread:9187 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:10:51,721 INFO SenderThread:9187 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:10:51,797 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:10:51,808 INFO SenderThread:9187 [sender.py:send_exit():567] send defer +2023-03-09 23:10:51,828 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,832 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:10:51,833 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,833 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:10:51,833 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:10:51,833 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,833 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:10:51,834 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,834 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:10:51,834 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:10:51,834 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,834 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:10:51,834 INFO HandlerThread:9187 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:10:51,834 DEBUG SystemMonitor:9187 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:10:51,835 DEBUG SystemMonitor:9187 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:10:51,835 DEBUG SystemMonitor:9187 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:10:51,837 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:10:51,838 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:10:51,850 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:10:51,851 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:10:51,851 INFO HandlerThread:9187 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:10:51,852 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,852 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:10:51,852 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:10:51,852 DEBUG SenderThread:9187 [sender.py:send():336] send: stats +2023-03-09 23:10:51,853 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,853 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:10:51,853 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,853 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:10:51,853 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:10:51,854 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,854 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:10:51,854 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,854 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:10:51,854 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:10:51,854 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,854 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:10:51,854 DEBUG SenderThread:9187 [sender.py:send():336] send: summary +2023-03-09 23:10:51,873 INFO SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:10:51,874 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:10:51,874 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:51,874 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:10:51,874 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:51,874 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:10:51,879 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:10:52,221 INFO Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json +2023-03-09 23:10:52,221 INFO Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log +2023-03-09 23:10:52,323 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:10:52,323 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:52,323 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:10:52,324 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:52,324 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:10:52,324 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:10:52,402 INFO wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /tmp/tmpfner7v0awandb/mcayg890-wandb-metadata.json +2023-03-09 23:10:53,222 INFO Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml +2023-03-09 23:10:53,747 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:10:53,753 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:10:53,753 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:53,758 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:10:53,789 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:53,801 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:10:53,827 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:10:53,840 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:53,840 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:10:53,840 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:53,840 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:10:53,840 INFO SenderThread:9187 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:10:54,228 INFO SenderThread:9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log +2023-03-09 23:10:54,228 INFO SenderThread:9187 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files +2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log output.log +2023-03-09 23:10:54,229 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt requirements.txt +2023-03-09 23:10:54,235 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml config.yaml +2023-03-09 23:10:54,255 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json wandb-summary.json +2023-03-09 23:10:54,304 INFO SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:10:54,304 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:10:54,523 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:10:54,529 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:10:54,542 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:10:54,542 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:10:54,542 INFO SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:10:55,533 INFO wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt +2023-03-09 23:10:55,794 INFO wandb-upload_3:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml +2023-03-09 23:10:55,904 INFO wandb-upload_1:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log +2023-03-09 23:10:57,354 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():1997] upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) +2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():1999] upload_file request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '26'} +2023-03-09 23:10:59,356 ERROR wandb-upload_4:9187 [internal_api.py:upload_file():2001] upload_file response body: +2023-03-09 23:11:00,543 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:11:02,356 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:11:03,317 INFO wandb-upload_4:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json +2023-03-09 23:11:03,517 INFO Thread-12 :9187 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:11:03,518 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:11:03,518 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:11:03,518 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:11:03,518 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:11:03,518 INFO SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:11:03,518 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:11:03,519 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:11:03,519 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:11:03,519 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:11:03,519 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:11:04,731 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:11:04,731 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:11:04,731 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:11:04,731 INFO SenderThread:9187 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:11:04,732 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:11:04,732 INFO HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send():336] send: final +2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send():336] send: footer +2023-03-09 23:11:04,732 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: defer +2023-03-09 23:11:04,732 INFO SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:11:04,733 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:11:04,733 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:11:04,734 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:11:04,734 DEBUG SenderThread:9187 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:11:04,770 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:11:04,800 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:11:05,697 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:11:05,697 INFO MainThread:9187 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:11:05,697 INFO MainThread:9187 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:11:05,698 INFO MainThread:9187 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:11:05,698 DEBUG HandlerThread:9187 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:11:05,698 INFO HandlerThread:9187 [handler.py:finish():842] shutting down handler +2023-03-09 23:11:06,697 INFO SenderThread:9187 [sender.py:finish():1504] shutting down sender +2023-03-09 23:11:06,697 INFO SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:11:06,697 INFO SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:11:06,697 INFO WriterThread:9187 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log new file mode 100644 index 0000000..706d6a1 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Configure stats pid to 7468 +2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:10:43,526 INFO MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():602] starting backend +2023-03-09 23:10:43,527 INFO MainThread:7468 [wandb_init.py:init():606] setting up manager +2023-03-09 23:10:43,538 INFO MainThread:7468 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:10:43,540 INFO MainThread:7468 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:10:43,544 INFO MainThread:7468 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:10:43,545 INFO MainThread:7468 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:10:45,099 INFO MainThread:7468 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:10:50,134 INFO MainThread:7468 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:10:50,134 INFO MainThread:7468 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:10:51,075 INFO MainThread:7468 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:10:51,076 INFO MainThread:7468 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:10:51,076 INFO MainThread:7468 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} +2023-03-09 23:11:06,747 WARNING MsgRouterThr:7468 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb new file mode 100644 index 0000000000000000000000000000000000000000..a8e212b5b9b5650fc533185a43425fa59b5f39d3 GIT binary patch literal 4195 zcmbW4UuYaf9LF=sCA~~8?e(g(UJ4FZYST1X_wW6~Dx_(lDDk0{f*|MI-dyf3m)q;^ zCeez}o=|KNAM8sj7NQjkJ}3%8@ZUp4)GFeG&}yL|h!pY9L-AkjgR__0+3k!?+q;8f zZ@&Bc&F}k}+28Kw58OBT!^b_gH8q&&0a4HbsU#}u#avM~G11hrZeSg2aynbirFD`k zDXLn06ZD*mK^V21iY2EH8AreSz3>^0?rj12*3qdLwrjR_utFS7wT^=$Ajp*Le{5o! zxT$q;7!DK`aidghf!=4Ht&LWSH^u7$X!2T#$?gPobWdKqGnv{_gWo(TsDH=VPjKskxxWUa6))&zc z>ZVWK2=9g)+C@yv5f_)Ri=~=lH3skf{5ap5wO2-cZPwA4R5Kq0{t_ zm0=7Hkfvo;oE88Ej6+z&u30HK)kB1iC~zYjt`lrGs*Ra~ja@p}uaF7ZfScet2QQF< zSs~_Z)2cSy7Kpg^Jb~-sV0CV$;F!2hPw$f@ z^t0^mOv8c=Ih$p{yBGy$CGmcbgdaOa7OJK{qc>>ImoQWJ&fyv2Adi=410hPREy6qJ z_TCGxPFCsCuBmB>SO?qE(aR@6atk$Jk|H)|(deAzxX30ZX}E~aT1R#4xZCqx%=cCP zygNnbi`Le%`3@hye)9IN_ZxPlxpoIWMpF&e_OVpU*PcHGlDiPvQ>_z}YA?0ad^$Cb zC@O-KbZ#6~8|Z;Kt2AFH54NeXy|)wFGmg~WhT~mW`}68szj)SQ*xI=hr@FBA`EQ#O z=V_{;+6lnCEq!vDdTW83nagI|-rAlE)F7$1?akP@R46ylPGmJ5X#v;gNrASrf;T=| zw(Lc0d-^zGNHBv-@4x+GUvdccna%lji1Sy+L!h}ad5piUg#`DmmAdUy3 zQXqa#k%-E)|3l)(@l0g8G`_||@tpgbNX1|pkHuRZKThPr=2Jy3EKcRQglbWFE@4gN z!s=8-E{b2LDso}oRgp^&Cvpiio{JK2QF$)(iOa7=}y@*!mepu4Ii(G;@o{J{2I78%;^XoKtE;?;l9S?@ch4mLh`Y= z)4pH&weVaNi7jqg1^PUm8u;!Clg-eF#5?a3u z&n46(Gd!1Yojey$+X&-L+cLddxa3xBf=l#GvNvla?}D)gaThInR&rJzWn_8}4vo_H zkA=>&&oGRXs?IEZfz)(KzWnJ!CnEHrYJIZj;)OF0LwcB?IY+^3py$~4&t2%J0~_82 GJ^ul&__Hwp literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml new file mode 100644 index 0000000..fefb4f3 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml @@ -0,0 +1,40 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678374750.437312 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log new file mode 100644 index 0000000..0a4ff46 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log @@ -0,0 +1,8 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Device: cuda + Images scaling: 0.5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json new file mode 100644 index 0000000..0568fa2 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:12:37.409031", + "startedAt": "2023-03-09T15:12:30.397388", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.8709499999995, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2503.176, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.665, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.337, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.575, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.932, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.175, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.232, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.76, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.386, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.645, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.742, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.602, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.547, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.785, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.892, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.913, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.64, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.636, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.065, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.447, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.937, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.397, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.317, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.778, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.691, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.442, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.403, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.367, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.909, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.332, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.151, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.712, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.725, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.245, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.616, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.382, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.774, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2509.295, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.964, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.445, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.114, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.765, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.667, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.401, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.018, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.418, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.384, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.864, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.165, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.535, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.818, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.705, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.344, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.589, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.395, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.818, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.757, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.398, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.645, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.011, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.031, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.391, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.45, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.348, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.345, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.644, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.544, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.622, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.011, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.421, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6551322937012 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log new file mode 100644 index 0000000..5d443d7 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log @@ -0,0 +1,155 @@ +2023-03-09 23:12:30,443 INFO StreamThr :11017 [internal.py:wandb_internal():90] W&B internal server running at pid: 11017, started at: 2023-03-09 23:12:30.441643 +2023-03-09 23:12:30,445 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:12:30,446 INFO WriterThread:11017 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/run-qbg0lcfa.wandb +2023-03-09 23:12:30,452 DEBUG SenderThread:11017 [sender.py:send():336] send: header +2023-03-09 23:12:30,452 DEBUG SenderThread:11017 [sender.py:send():336] send: run +2023-03-09 23:12:30,506 INFO SenderThread:11017 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/qbg0lcfa +2023-03-09 23:12:32,285 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:12:32,285 INFO SenderThread:11017 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files +2023-03-09 23:12:32,285 INFO SenderThread:11017 [sender.py:_start_run_threads():1081] run started: qbg0lcfa with start time 1678374750.437312 +2023-03-09 23:12:32,285 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:12:32,305 INFO SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:12:32,305 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:12:33,289 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json +2023-03-09 23:12:37,286 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:12:37,306 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:12:37,368 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:12:37,373 DEBUG HandlerThread:11017 [system_info.py:__init__():31] System info init +2023-03-09 23:12:37,373 DEBUG HandlerThread:11017 [system_info.py:__init__():46] System info init done +2023-03-09 23:12:37,373 INFO HandlerThread:11017 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:12:37,374 INFO SystemMonitor:11017 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:12:37,374 INFO HandlerThread:11017 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:12:37,374 INFO SystemMonitor:11017 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:12:37,375 INFO SystemMonitor:11017 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:12:37,375 INFO SystemMonitor:11017 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:12:37,376 INFO SystemMonitor:11017 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:12:37,376 INFO SystemMonitor:11017 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:12:37,408 DEBUG HandlerThread:11017 [system_info.py:probe():195] Probing system +2023-03-09 23:12:37,420 DEBUG HandlerThread:11017 [git.py:repo():40] git repository is invalid +2023-03-09 23:12:37,420 DEBUG HandlerThread:11017 [system_info.py:probe():240] Probing system done +2023-03-09 23:12:37,421 DEBUG HandlerThread:11017 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:12:37.409031', 'startedAt': '2023-03-09T15:12:30.397388', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.8709499999995, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2503.176, 'min': 0.0, 'max': 0.0}, {'current': 2499.665, 'min': 0.0, 'max': 0.0}, {'current': 2500.337, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2502.932, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.175, 'min': 0.0, 'max': 0.0}, {'current': 2499.232, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.76, 'min': 0.0, 'max': 0.0}, {'current': 2500.386, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.602, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.547, 'min': 0.0, 'max': 0.0}, {'current': 2498.785, 'min': 0.0, 'max': 0.0}, {'current': 2503.892, 'min': 0.0, 'max': 0.0}, {'current': 2499.913, 'min': 0.0, 'max': 0.0}, {'current': 2499.64, 'min': 0.0, 'max': 0.0}, {'current': 2500.636, 'min': 0.0, 'max': 0.0}, {'current': 2505.065, 'min': 0.0, 'max': 0.0}, {'current': 2498.447, 'min': 0.0, 'max': 0.0}, {'current': 2500.937, 'min': 0.0, 'max': 0.0}, {'current': 2505.397, 'min': 0.0, 'max': 0.0}, {'current': 2499.317, 'min': 0.0, 'max': 0.0}, {'current': 2500.778, 'min': 0.0, 'max': 0.0}, {'current': 2500.691, 'min': 0.0, 'max': 0.0}, {'current': 2501.442, 'min': 0.0, 'max': 0.0}, {'current': 2501.403, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.909, 'min': 0.0, 'max': 0.0}, {'current': 2502.332, 'min': 0.0, 'max': 0.0}, {'current': 2501.151, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.712, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2503.245, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2503.382, 'min': 0.0, 'max': 0.0}, {'current': 2504.774, 'min': 0.0, 'max': 0.0}, {'current': 2504.997, 'min': 0.0, 'max': 0.0}, {'current': 2509.295, 'min': 0.0, 'max': 0.0}, {'current': 2503.964, 'min': 0.0, 'max': 0.0}, {'current': 2500.445, 'min': 0.0, 'max': 0.0}, {'current': 2501.114, 'min': 0.0, 'max': 0.0}, {'current': 2499.765, 'min': 0.0, 'max': 0.0}, {'current': 2500.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2506.018, 'min': 0.0, 'max': 0.0}, {'current': 2500.418, 'min': 0.0, 'max': 0.0}, {'current': 2500.384, 'min': 0.0, 'max': 0.0}, {'current': 2505.864, 'min': 0.0, 'max': 0.0}, {'current': 2505.165, 'min': 0.0, 'max': 0.0}, {'current': 2507.535, 'min': 0.0, 'max': 0.0}, {'current': 2506.818, 'min': 0.0, 'max': 0.0}, {'current': 2504.705, 'min': 0.0, 'max': 0.0}, {'current': 2500.344, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.818, 'min': 0.0, 'max': 0.0}, {'current': 2503.757, 'min': 0.0, 'max': 0.0}, {'current': 2504.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.031, 'min': 0.0, 'max': 0.0}, {'current': 2500.391, 'min': 0.0, 'max': 0.0}, {'current': 2500.45, 'min': 0.0, 'max': 0.0}, {'current': 2505.348, 'min': 0.0, 'max': 0.0}, {'current': 2505.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.644, 'min': 0.0, 'max': 0.0}, {'current': 2507.027, 'min': 0.0, 'max': 0.0}, {'current': 2499.544, 'min': 0.0, 'max': 0.0}, {'current': 2500.622, 'min': 0.0, 'max': 0.0}, {'current': 2501.011, 'min': 0.0, 'max': 0.0}, {'current': 2501.421, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6551322937012}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:12:37,421 INFO HandlerThread:11017 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:12:37,421 INFO HandlerThread:11017 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:12:37,421 DEBUG HandlerThread:11017 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:12:37,486 DEBUG HandlerThread:11017 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:12:37,486 DEBUG HandlerThread:11017 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:12:38,197 DEBUG HandlerThread:11017 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:12:38,229 INFO HandlerThread:11017 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:12:38,232 DEBUG SenderThread:11017 [sender.py:send():336] send: files +2023-03-09 23:12:38,232 INFO SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:12:38,248 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:12:38,268 DEBUG SenderThread:11017 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/conda-environment.yaml +2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json +2023-03-09 23:12:38,472 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt +2023-03-09 23:12:38,720 DEBUG SenderThread:11017 [sender.py:send():336] send: telemetry +2023-03-09 23:12:38,721 DEBUG SenderThread:11017 [sender.py:send():336] send: config +2023-03-09 23:12:39,475 INFO Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log +2023-03-09 23:12:39,490 INFO wandb-upload_0:11017 [upload_job.py:push():138] Uploaded file /tmp/tmp03hehnd5wandb/0g7l9wyo-wandb-metadata.json +2023-03-09 23:12:41,477 INFO Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log +2023-03-09 23:12:42,727 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:12:47,727 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:12:52,728 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:12:53,241 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:12:53,242 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:12:57,729 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:12:58,243 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:02,735 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:03,257 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:03,497 INFO Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml +2023-03-09 23:13:08,259 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:08,259 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:13,260 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:13,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:18,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:18,262 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:23,261 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:23,263 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:28,262 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:28,265 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:33,263 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:33,266 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:37,377 DEBUG SystemMonitor:11017 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:13:37,381 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:13:38,267 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:38,382 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:43,269 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:43,383 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:48,270 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:48,384 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:53,272 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:53,384 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:13:58,273 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:13:58,385 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:03,274 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:03,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:07,384 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:14:08,275 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:09,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:13,277 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:14,386 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:18,278 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:19,387 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:23,294 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:24,409 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:28,309 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:29,410 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:33,509 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:34,411 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:37,414 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:14:38,609 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:39,426 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:43,622 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:44,608 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:48,623 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:49,609 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:53,709 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:54,610 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:14:58,711 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:14:59,611 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:03,712 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:04,611 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:07,416 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:15:08,713 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:10,418 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:13,715 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:15,419 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:18,716 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:20,420 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:23,717 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:25,420 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:28,718 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:30,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:33,720 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:35,422 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:37,419 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:15:38,722 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:41,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:41,969 WARNING FileStreamThread:11017 [file_stream.py:request_with_retry():675] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream. func: >, args: ('https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 4, 'content': ['{"system.disk": 34.0, "system.network.sent": 648043.93, "system.network.recv": 292258.87, "system.proc.memory.availableMB": 329995.35, "system.memory": 14.43, "system.proc.memory.rssMB": 22483.33, "system.proc.memory.percent": 5.83, "system.cpu": 0.01, "system.cpu.0.cpu_percent": 13.63, "system.cpu.1.cpu_percent": 14.26, "system.cpu.2.cpu_percent": 13.02, "system.cpu.3.cpu_percent": 11.75, "system.cpu.4.cpu_percent": 11.36, "system.cpu.5.cpu_percent": 13.88, "system.cpu.6.cpu_percent": 11.97, "system.cpu.7.cpu_percent": 11.55, "system.cpu.8.cpu_percent": 10.73, "system.cpu.9.cpu_percent": 11.03, "system.cpu.10.cpu_percent": 12.05, "system.cpu.11.cpu_percent": 12.08, "system.cpu.12.cpu_percent": 12.5, "system.cpu.13.cpu_percent": 10.96, "system.cpu.14.cpu_percent": 11.1, "system.cpu.15.cpu_percent": 11.49, "system.cpu.16.cpu_percent": 11.22, "system.cpu.17.cpu_percent": 11.7, "system.cpu.18.cpu_percent": 12.28, "system.cpu.19.cpu_percent": 10.81, "system.cpu.20.cpu_percent": 13.6, "system.cpu.21.cpu_percent": 13.91, "system.cpu.22.cpu_percent": 13.94, "system.cpu.23.cpu_percent": 11.47, "system.cpu.24.cpu_percent": 10.25, "system.cpu.25.cpu_percent": 12.91, "system.cpu.26.cpu_percent": 10.18, "system.cpu.27.cpu_percent": 10.71, "system.cpu.28.cpu_percent": 10.51, "system.cpu.29.cpu_percent": 9.33, "system.cpu.30.cpu_percent": 10.98, "system.cpu.31.cpu_percent": 10.93, "system.cpu.32.cpu_percent": 9.93, "system.cpu.33.cpu_percent": 10.65, "system.cpu.34.cpu_percent": 11.46, "system.cpu.35.cpu_percent": 9.93, "system.cpu.36.cpu_percent": 10.47, "system.cpu.37.cpu_percent": 10.19, "system.cpu.38.cpu_percent": 9.21, "system.cpu.39.cpu_percent": 10.19, "system.cpu.40.cpu_percent": 12.19, "system.cpu.41.cpu_percent": 12.61, "system.cpu.42.cpu_percent": 18.98, "system.cpu.43.cpu_percent": 11.38, "system.cpu.44.cpu_percent": 11.84, "system.cpu.45.cpu_percent": 11.58, "system.cpu.46.cpu_percent": 10.05, "system.cpu.47.cpu_percent": 11.54, "system.cpu.48.cpu_percent": 11.37, "system.cpu.49.cpu_percent": 12.76, "system.cpu.50.cpu_percent": 11.16, "system.cpu.51.cpu_percent": 11.69, "system.cpu.52.cpu_percent": 10.77, "system.cpu.53.cpu_percent": 10.6, "system.cpu.54.cpu_percent": 10.82, "system.cpu.55.cpu_percent": 11.67, "system.cpu.56.cpu_percent": 17.88, "system.cpu.57.cpu_percent": 14.37, "system.cpu.58.cpu_percent": 10.75, "system.cpu.59.cpu_percent": 11.79, "system.cpu.60.cpu_percent": 11.43, "system.cpu.61.cpu_percent": 10.05, "system.cpu.62.cpu_percent": 9.11, "system.cpu.63.cpu_percent": 9.93, "system.cpu.64.cpu_percent": 11.24, "system.cpu.65.cpu_percent": 16.09, "system.cpu.66.cpu_percent": 12.53, "system.cpu.67.cpu_percent": 17.75, "system.cpu.68.cpu_percent": 12.15, "system.cpu.69.cpu_percent": 10.11, "system.cpu.70.cpu_percent": 9.68, "system.cpu.71.cpu_percent": 10.84, "system.cpu.72.cpu_percent": 16.69, "system.cpu.73.cpu_percent": 10.49, "system.cpu.74.cpu_percent": 10.3, "system.cpu.75.cpu_percent": 12.14, "system.cpu.76.cpu_percent": 12.36, "system.cpu.77.cpu_percent": 9.46, "system.cpu.78.cpu_percent": 10.52, "system.cpu.79.cpu_percent": 11.16, "system.proc.cpu.threads": 88, "system.gpu.0.memoryAllocated": 32.36, "system.gpu.1.memoryAllocated": 0.02, "system.gpu.0.memory": 0.0, "system.gpu.1.memory": 0.0, "system.gpu.0.gpu": 0.0, "system.gpu.1.gpu": 0.0, "system.gpu.0.temp": 56.0, "system.gpu.1.temp": 39.0, "system.gpu.0.powerWatts": 29.95, "system.gpu.1.powerWatts": 9.91, "system.gpu.0.powerPercent": 42.78, "system.gpu.1.powerPercent": 14.18, "_wandb": true, "_timestamp": 1678374937.415186, "_runtime": 186.977874}']}}, 'dropped': 0}} +2023-03-09 23:15:43,723 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:46,421 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:48,724 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:51,422 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:53,725 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:56,423 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:58,726 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:01,423 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:03,728 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:06,424 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:07,490 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:16:08,729 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:11,492 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:13,730 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:16,492 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:18,732 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:21,493 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:23,733 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:26,494 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:28,734 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:31,494 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:33,736 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:36,495 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:37,493 DEBUG SenderThread:11017 [sender.py:send():336] send: stats +2023-03-09 23:16:38,737 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:16:42,508 DEBUG HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:16:45,728 INFO MainThread:11017 [internal.py:handle_exit():77] Internal process exited diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log new file mode 100644 index 0000000..ce507d7 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Configure stats pid to 9232 +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:12:30,423 INFO MainThread:9232 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():602] starting backend +2023-03-09 23:12:30,424 INFO MainThread:9232 [wandb_init.py:init():606] setting up manager +2023-03-09 23:12:30,435 INFO MainThread:9232 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:12:30,437 INFO MainThread:9232 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:12:30,440 INFO MainThread:9232 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:12:30,441 INFO MainThread:9232 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:12:32,284 INFO MainThread:9232 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:12:37,307 INFO MainThread:9232 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:12:37,307 INFO MainThread:9232 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:12:38,240 INFO MainThread:9232 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:12:38,241 INFO MainThread:9232 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:12:38,242 INFO MainThread:9232 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:12:38,243 INFO MainThread:9232 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/run-qbg0lcfa.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/run-qbg0lcfa.wandb new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml new file mode 100644 index 0000000..3457afe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml @@ -0,0 +1,40 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678375959.443294 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/output.log new file mode 100644 index 0000000..0a4ff46 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/output.log @@ -0,0 +1,8 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Device: cuda + Images scaling: 0.5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json new file mode 100644 index 0000000..036767f --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:32:46.785082", + "startedAt": "2023-03-09T15:32:39.401025", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.3775624999994, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.614, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.081, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.009, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.73, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.934, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.158, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.351, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.211, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.412, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.243, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.648, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.74, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.191, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.465, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.182, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.405, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.933, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.646, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.576, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.25, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.654, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.366, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.578, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.947, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.823, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.483, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.788, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.937, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.939, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.089, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.405, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.625, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.759, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.618, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.609, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.887, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.558, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.769, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.432, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.978, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.278, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.876, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.435, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.423, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.059, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.103, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.258, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.683, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.349, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.848, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.132, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.036, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.927, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.827, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.608, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.265, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.142, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.334, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.609, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.275, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.014, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.764, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.409, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.183, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.052, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.383, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.035, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.466, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.834, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.654, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.935, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6485137939453 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log new file mode 100644 index 0000000..c99609b --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log @@ -0,0 +1,94 @@ +2023-03-09 23:32:39,448 INFO StreamThr :21832 [internal.py:wandb_internal():90] W&B internal server running at pid: 21832, started at: 2023-03-09 23:32:39.446798 +2023-03-09 23:32:39,450 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:32:39,451 INFO WriterThread:21832 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/run-58pb6znt.wandb +2023-03-09 23:32:39,456 DEBUG SenderThread:21832 [sender.py:send():336] send: header +2023-03-09 23:32:39,456 DEBUG SenderThread:21832 [sender.py:send():336] send: run +2023-03-09 23:32:39,536 INFO SenderThread:21832 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/58pb6znt +2023-03-09 23:32:41,675 INFO SenderThread:21832 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files +2023-03-09 23:32:41,675 INFO SenderThread:21832 [sender.py:_start_run_threads():1081] run started: 58pb6znt with start time 1678375959.443294 +2023-03-09 23:32:41,676 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:32:41,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:32:41,715 INFO SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:32:41,716 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:32:42,679 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json +2023-03-09 23:32:46,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:32:46,717 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:32:46,737 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:32:46,742 DEBUG HandlerThread:21832 [system_info.py:__init__():31] System info init +2023-03-09 23:32:46,742 DEBUG HandlerThread:21832 [system_info.py:__init__():46] System info init done +2023-03-09 23:32:46,742 INFO HandlerThread:21832 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:32:46,742 INFO SystemMonitor:21832 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:32:46,742 INFO HandlerThread:21832 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:32:46,743 INFO SystemMonitor:21832 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:32:46,744 INFO SystemMonitor:21832 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:32:46,744 INFO SystemMonitor:21832 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:32:46,745 INFO SystemMonitor:21832 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:32:46,746 INFO SystemMonitor:21832 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:32:46,785 DEBUG HandlerThread:21832 [system_info.py:probe():195] Probing system +2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [git.py:repo():40] git repository is invalid +2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [system_info.py:probe():240] Probing system done +2023-03-09 23:32:46,806 DEBUG HandlerThread:21832 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:32:46.785082', 'startedAt': '2023-03-09T15:32:39.401025', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.3775624999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.614, 'min': 0.0, 'max': 0.0}, {'current': 2501.081, 'min': 0.0, 'max': 0.0}, {'current': 2500.009, 'min': 0.0, 'max': 0.0}, {'current': 2500.73, 'min': 0.0, 'max': 0.0}, {'current': 2502.934, 'min': 0.0, 'max': 0.0}, {'current': 2500.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.351, 'min': 0.0, 'max': 0.0}, {'current': 2500.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.412, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2499.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.191, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.465, 'min': 0.0, 'max': 0.0}, {'current': 2500.182, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2501.933, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.646, 'min': 0.0, 'max': 0.0}, {'current': 2499.576, 'min': 0.0, 'max': 0.0}, {'current': 2499.25, 'min': 0.0, 'max': 0.0}, {'current': 2499.654, 'min': 0.0, 'max': 0.0}, {'current': 2500.366, 'min': 0.0, 'max': 0.0}, {'current': 2501.578, 'min': 0.0, 'max': 0.0}, {'current': 2499.947, 'min': 0.0, 'max': 0.0}, {'current': 2503.823, 'min': 0.0, 'max': 0.0}, {'current': 2501.483, 'min': 0.0, 'max': 0.0}, {'current': 2499.788, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2500.089, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2499.625, 'min': 0.0, 'max': 0.0}, {'current': 2504.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2500.887, 'min': 0.0, 'max': 0.0}, {'current': 2504.558, 'min': 0.0, 'max': 0.0}, {'current': 2500.769, 'min': 0.0, 'max': 0.0}, {'current': 2500.432, 'min': 0.0, 'max': 0.0}, {'current': 2504.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.278, 'min': 0.0, 'max': 0.0}, {'current': 2503.876, 'min': 0.0, 'max': 0.0}, {'current': 2500.435, 'min': 0.0, 'max': 0.0}, {'current': 2504.423, 'min': 0.0, 'max': 0.0}, {'current': 2500.059, 'min': 0.0, 'max': 0.0}, {'current': 2505.103, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2500.258, 'min': 0.0, 'max': 0.0}, {'current': 2503.683, 'min': 0.0, 'max': 0.0}, {'current': 2504.349, 'min': 0.0, 'max': 0.0}, {'current': 2499.848, 'min': 0.0, 'max': 0.0}, {'current': 2505.132, 'min': 0.0, 'max': 0.0}, {'current': 2505.036, 'min': 0.0, 'max': 0.0}, {'current': 2504.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2505.608, 'min': 0.0, 'max': 0.0}, {'current': 2501.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.142, 'min': 0.0, 'max': 0.0}, {'current': 2506.334, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2504.275, 'min': 0.0, 'max': 0.0}, {'current': 2501.014, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.409, 'min': 0.0, 'max': 0.0}, {'current': 2505.183, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2500.383, 'min': 0.0, 'max': 0.0}, {'current': 2503.035, 'min': 0.0, 'max': 0.0}, {'current': 2504.466, 'min': 0.0, 'max': 0.0}, {'current': 2503.834, 'min': 0.0, 'max': 0.0}, {'current': 2500.654, 'min': 0.0, 'max': 0.0}, {'current': 2503.935, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6485137939453}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:32:46,807 INFO HandlerThread:21832 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:32:46,807 INFO HandlerThread:21832 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:32:46,807 DEBUG HandlerThread:21832 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:32:46,861 DEBUG HandlerThread:21832 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:32:46,861 DEBUG HandlerThread:21832 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:32:47,586 DEBUG HandlerThread:21832 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:32:47,646 INFO HandlerThread:21832 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:32:47,649 DEBUG SenderThread:21832 [sender.py:send():336] send: files +2023-03-09 23:32:47,649 INFO SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:32:47,666 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:32:47,666 DEBUG SenderThread:21832 [sender.py:send():336] send: telemetry +2023-03-09 23:32:47,682 DEBUG SenderThread:21832 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:32:47,868 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/conda-environment.yaml +2023-03-09 23:32:47,878 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt +2023-03-09 23:32:47,888 INFO Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json +2023-03-09 23:32:48,228 DEBUG SenderThread:21832 [sender.py:send():336] send: config +2023-03-09 23:32:48,916 INFO wandb-upload_0:21832 [upload_job.py:push():138] Uploaded file /tmp/tmpckj69pbswandb/vyd13lcr-wandb-metadata.json +2023-03-09 23:32:52,258 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:32:57,259 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:02,260 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:02,657 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:02,657 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:07,260 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:07,659 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:12,266 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:12,660 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:13,878 INFO Thread-13 :21832 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml +2023-03-09 23:33:17,661 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:18,617 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:22,662 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:23,618 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:27,663 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:28,619 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:32,665 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:33,619 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:37,666 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:38,620 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:42,667 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:43,621 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:46,746 DEBUG SystemMonitor:21832 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:33:46,751 DEBUG SenderThread:21832 [sender.py:send():336] send: stats +2023-03-09 23:33:47,668 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:48,752 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:52,669 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:53,753 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:33:57,671 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:33:58,754 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:02,672 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:34:03,754 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:07,673 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:34:08,755 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:12,674 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:34:13,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:16,754 DEBUG SenderThread:21832 [sender.py:send():336] send: stats +2023-03-09 23:34:17,676 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:34:19,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:22,677 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:34:24,756 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:28,873 ERROR memory :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) +2023-03-09 23:34:28,873 ERROR memory :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) +2023-03-09 23:34:28,915 ERROR cpu :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) +2023-03-09 23:34:28,917 ERROR cpu :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104) +2023-03-09 23:34:29,757 DEBUG HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:34:29,823 INFO MainThread:21832 [internal.py:handle_exit():77] Internal process exited diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log new file mode 100644 index 0000000..88d31f0 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Configure stats pid to 20104 +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log +2023-03-09 23:32:39,428 INFO MainThread:20104 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log +2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():602] starting backend +2023-03-09 23:32:39,429 INFO MainThread:20104 [wandb_init.py:init():606] setting up manager +2023-03-09 23:32:39,440 INFO MainThread:20104 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:32:39,442 INFO MainThread:20104 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:32:39,447 INFO MainThread:20104 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:32:39,448 INFO MainThread:20104 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:32:41,675 INFO MainThread:20104 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:32:46,718 INFO MainThread:20104 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:32:46,718 INFO MainThread:20104 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:32:47,656 INFO MainThread:20104 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:32:47,657 INFO MainThread:20104 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:32:47,658 INFO MainThread:20104 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/run-58pb6znt.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/run-58pb6znt.wandb new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/config.yaml new file mode 100644 index 0000000..1f01caa --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/config.yaml @@ -0,0 +1,22 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678376153.802776 + t: + 1: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log new file mode 100644 index 0000000..3926efa --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log @@ -0,0 +1,7 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json new file mode 100644 index 0000000..2634e3d --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:36:00.197376", + "startedAt": "2023-03-09T15:35:53.764072", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.5297500000006, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.096, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.615, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.893, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.591, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.521, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.243, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.638, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.144, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.868, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.649, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.109, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.04, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.822, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.83, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.476, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.859, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.572, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.283, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.012, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.618, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.178, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.045, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.773, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.834, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.236, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.061, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.96, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.082, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.189, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.323, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.69, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.152, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.988, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.041, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.322, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.397, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.099, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.921, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.55, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.546, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.646, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.091, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.889, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.029, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.008, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.061, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.807, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.396, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.189, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.804, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.96, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.432, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.638, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.016, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.045, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.726, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.212, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.968, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.15, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.153, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.227, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.381, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.1, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.225, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.803, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.722, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.718, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.523, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.426, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.206, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.864, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.673, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.64935302734375 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log new file mode 100644 index 0000000..e96eaf7 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log @@ -0,0 +1,51 @@ +2023-03-09 23:35:53,808 INFO StreamThr :24054 [internal.py:wandb_internal():90] W&B internal server running at pid: 24054, started at: 2023-03-09 23:35:53.806812 +2023-03-09 23:35:53,810 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:35:53,811 INFO WriterThread:24054 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/run-yc1qtvil.wandb +2023-03-09 23:35:53,828 DEBUG SenderThread:24054 [sender.py:send():336] send: header +2023-03-09 23:35:53,828 DEBUG SenderThread:24054 [sender.py:send():336] send: run +2023-03-09 23:35:53,864 INFO SenderThread:24054 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/yc1qtvil +2023-03-09 23:35:55,068 INFO SenderThread:24054 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files +2023-03-09 23:35:55,069 INFO SenderThread:24054 [sender.py:_start_run_threads():1081] run started: yc1qtvil with start time 1678376153.802776 +2023-03-09 23:35:55,069 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:35:55,069 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:35:55,122 INFO SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:35:55,123 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:35:56,073 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json +2023-03-09 23:36:00,070 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:36:00,124 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:36:00,144 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:36:00,154 DEBUG HandlerThread:24054 [system_info.py:__init__():31] System info init +2023-03-09 23:36:00,154 DEBUG HandlerThread:24054 [system_info.py:__init__():46] System info init done +2023-03-09 23:36:00,155 INFO HandlerThread:24054 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:36:00,155 INFO SystemMonitor:24054 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:36:00,155 INFO HandlerThread:24054 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:36:00,156 INFO SystemMonitor:24054 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:36:00,156 INFO SystemMonitor:24054 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:36:00,157 INFO SystemMonitor:24054 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:36:00,157 INFO SystemMonitor:24054 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:36:00,158 INFO SystemMonitor:24054 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:36:00,197 DEBUG HandlerThread:24054 [system_info.py:probe():195] Probing system +2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [git.py:repo():40] git repository is invalid +2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [system_info.py:probe():240] Probing system done +2023-03-09 23:36:00,234 DEBUG HandlerThread:24054 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:36:00.197376', 'startedAt': '2023-03-09T15:35:53.764072', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5297500000006, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.096, 'min': 0.0, 'max': 0.0}, {'current': 2502.615, 'min': 0.0, 'max': 0.0}, {'current': 2500.893, 'min': 0.0, 'max': 0.0}, {'current': 2499.591, 'min': 0.0, 'max': 0.0}, {'current': 2499.521, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2499.638, 'min': 0.0, 'max': 0.0}, {'current': 2499.144, 'min': 0.0, 'max': 0.0}, {'current': 2501.868, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2501.109, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.822, 'min': 0.0, 'max': 0.0}, {'current': 2498.83, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.572, 'min': 0.0, 'max': 0.0}, {'current': 2499.283, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2501.178, 'min': 0.0, 'max': 0.0}, {'current': 2501.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.773, 'min': 0.0, 'max': 0.0}, {'current': 2501.834, 'min': 0.0, 'max': 0.0}, {'current': 2506.236, 'min': 0.0, 'max': 0.0}, {'current': 2501.061, 'min': 0.0, 'max': 0.0}, {'current': 2502.96, 'min': 0.0, 'max': 0.0}, {'current': 2501.082, 'min': 0.0, 'max': 0.0}, {'current': 2499.189, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.323, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2502.152, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.397, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2503.921, 'min': 0.0, 'max': 0.0}, {'current': 2500.55, 'min': 0.0, 'max': 0.0}, {'current': 2500.546, 'min': 0.0, 'max': 0.0}, {'current': 2505.646, 'min': 0.0, 'max': 0.0}, {'current': 2501.091, 'min': 0.0, 'max': 0.0}, {'current': 2499.889, 'min': 0.0, 'max': 0.0}, {'current': 2500.029, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2504.061, 'min': 0.0, 'max': 0.0}, {'current': 2505.807, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2500.189, 'min': 0.0, 'max': 0.0}, {'current': 2506.804, 'min': 0.0, 'max': 0.0}, {'current': 2500.96, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.432, 'min': 0.0, 'max': 0.0}, {'current': 2506.638, 'min': 0.0, 'max': 0.0}, {'current': 2507.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.726, 'min': 0.0, 'max': 0.0}, {'current': 2504.212, 'min': 0.0, 'max': 0.0}, {'current': 2499.968, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2504.15, 'min': 0.0, 'max': 0.0}, {'current': 2505.153, 'min': 0.0, 'max': 0.0}, {'current': 2499.227, 'min': 0.0, 'max': 0.0}, {'current': 2500.381, 'min': 0.0, 'max': 0.0}, {'current': 2504.1, 'min': 0.0, 'max': 0.0}, {'current': 2504.225, 'min': 0.0, 'max': 0.0}, {'current': 2500.803, 'min': 0.0, 'max': 0.0}, {'current': 2504.722, 'min': 0.0, 'max': 0.0}, {'current': 2504.718, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2500.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.206, 'min': 0.0, 'max': 0.0}, {'current': 2501.864, 'min': 0.0, 'max': 0.0}, {'current': 2504.673, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64935302734375}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:36:00,235 INFO HandlerThread:24054 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:36:00,235 INFO HandlerThread:24054 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:36:00,235 DEBUG HandlerThread:24054 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:36:00,274 DEBUG HandlerThread:24054 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:36:00,274 DEBUG HandlerThread:24054 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:36:00,993 DEBUG HandlerThread:24054 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:36:01,023 INFO HandlerThread:24054 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:36:01,027 DEBUG SenderThread:24054 [sender.py:send():336] send: files +2023-03-09 23:36:01,027 INFO SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:36:01,036 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:36:01,049 DEBUG SenderThread:24054 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:36:01,270 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/conda-environment.yaml +2023-03-09 23:36:01,271 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json +2023-03-09 23:36:01,271 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt +2023-03-09 23:36:01,524 DEBUG SenderThread:24054 [sender.py:send():336] send: telemetry +2023-03-09 23:36:01,524 DEBUG SenderThread:24054 [sender.py:send():336] send: config +2023-03-09 23:36:02,264 INFO Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log +2023-03-09 23:36:02,406 INFO wandb-upload_0:24054 [upload_job.py:push():138] Uploaded file /tmp/tmpzwt8zs5awandb/qutsputw-wandb-metadata.json +2023-03-09 23:36:04,279 INFO Thread-13 :24054 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log +2023-03-09 23:36:05,528 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:36:10,529 DEBUG HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log new file mode 100644 index 0000000..27a274c --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Configure stats pid to 22330 +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():602] starting backend +2023-03-09 23:35:53,789 INFO MainThread:22330 [wandb_init.py:init():606] setting up manager +2023-03-09 23:35:53,800 INFO MainThread:22330 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:35:53,802 INFO MainThread:22330 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:35:53,806 INFO MainThread:22330 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:35:53,807 INFO MainThread:22330 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:35:55,067 INFO MainThread:22330 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:36:00,125 INFO MainThread:22330 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:36:00,125 INFO MainThread:22330 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:36:01,034 INFO MainThread:22330 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:36:01,035 INFO MainThread:22330 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:36:01,036 INFO MainThread:22330 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/run-yc1qtvil.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/run-yc1qtvil.wandb new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml new file mode 100644 index 0000000..5415cd5 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml @@ -0,0 +1,39 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: false + python_version: 3.7.11 + start_time: 1678376631.533235 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log new file mode 100644 index 0000000..3926efa --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log @@ -0,0 +1,7 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json new file mode 100644 index 0000000..8496190 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:44:01.780995", + "startedAt": "2023-03-09T15:43:51.492578", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "n1d6fa00775e48a18f80acc1355070be-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.7918, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.007, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.205, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.476, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.178, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.693, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.211, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.485, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.073, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.962, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.188, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.333, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.473, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.557, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.576, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.927, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.451, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.543, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.573, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.677, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.653, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.368, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.888, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.582, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.338, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.318, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.259, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.245, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.213, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.943, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.789, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.016, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.944, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.978, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.631, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.601, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.717, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.733, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.492, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.656, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.187, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.269, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.918, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.545, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.833, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.211, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.345, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.134, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.032, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.317, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.961, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.426, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.555, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.063, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.827, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.429, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.572, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.604, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.86, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.01, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.523, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.704, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.206, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.531, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.875, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.19, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.872, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.063, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.075, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.623, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.85, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.509, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.271, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6454200744629 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log new file mode 100644 index 0000000..64470f0 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log @@ -0,0 +1,114 @@ +2023-03-09 23:43:51,539 INFO StreamThr :1879 [internal.py:wandb_internal():90] W&B internal server running at pid: 1879, started at: 2023-03-09 23:43:51.537061 +2023-03-09 23:43:51,540 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:43:51,541 INFO WriterThread:1879 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/run-w9y25r7k.wandb +2023-03-09 23:43:51,546 DEBUG SenderThread:1879 [sender.py:send():336] send: header +2023-03-09 23:43:51,546 DEBUG SenderThread:1879 [sender.py:send():336] send: run +2023-03-09 23:43:51,616 INFO SenderThread:1879 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/w9y25r7k +2023-03-09 23:43:56,543 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:43:56,651 INFO SenderThread:1879 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files +2023-03-09 23:43:56,651 INFO SenderThread:1879 [sender.py:_start_run_threads():1081] run started: w9y25r7k with start time 1678376631.533235 +2023-03-09 23:43:56,652 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:43:56,652 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:43:56,654 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:43:56,675 INFO SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:43:56,676 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:43:57,655 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json +2023-03-09 23:44:01,654 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:01,677 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:01,736 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:44:01,741 DEBUG HandlerThread:1879 [system_info.py:__init__():31] System info init +2023-03-09 23:44:01,741 DEBUG HandlerThread:1879 [system_info.py:__init__():46] System info init done +2023-03-09 23:44:01,741 INFO HandlerThread:1879 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:44:01,741 INFO SystemMonitor:1879 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:44:01,742 INFO HandlerThread:1879 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:44:01,742 INFO SystemMonitor:1879 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:44:01,743 INFO SystemMonitor:1879 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:44:01,743 INFO SystemMonitor:1879 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:44:01,744 INFO SystemMonitor:1879 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:44:01,744 INFO SystemMonitor:1879 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:44:01,780 DEBUG HandlerThread:1879 [system_info.py:probe():195] Probing system +2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [git.py:repo():40] git repository is invalid +2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [system_info.py:probe():240] Probing system done +2023-03-09 23:44:01,790 DEBUG HandlerThread:1879 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:44:01.780995', 'startedAt': '2023-03-09T15:43:51.492578', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.7918, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2503.205, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.178, 'min': 0.0, 'max': 0.0}, {'current': 2498.693, 'min': 0.0, 'max': 0.0}, {'current': 2499.211, 'min': 0.0, 'max': 0.0}, {'current': 2502.485, 'min': 0.0, 'max': 0.0}, {'current': 2500.073, 'min': 0.0, 'max': 0.0}, {'current': 2503.962, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2502.188, 'min': 0.0, 'max': 0.0}, {'current': 2503.333, 'min': 0.0, 'max': 0.0}, {'current': 2500.473, 'min': 0.0, 'max': 0.0}, {'current': 2499.557, 'min': 0.0, 'max': 0.0}, {'current': 2501.576, 'min': 0.0, 'max': 0.0}, {'current': 2501.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.451, 'min': 0.0, 'max': 0.0}, {'current': 2499.543, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2499.677, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2502.888, 'min': 0.0, 'max': 0.0}, {'current': 2503.582, 'min': 0.0, 'max': 0.0}, {'current': 2499.338, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.318, 'min': 0.0, 'max': 0.0}, {'current': 2501.259, 'min': 0.0, 'max': 0.0}, {'current': 2501.245, 'min': 0.0, 'max': 0.0}, {'current': 2501.213, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2499.789, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2499.944, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.631, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.601, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2505.733, 'min': 0.0, 'max': 0.0}, {'current': 2504.492, 'min': 0.0, 'max': 0.0}, {'current': 2505.656, 'min': 0.0, 'max': 0.0}, {'current': 2498.187, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2499.918, 'min': 0.0, 'max': 0.0}, {'current': 2504.545, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.833, 'min': 0.0, 'max': 0.0}, {'current': 2506.211, 'min': 0.0, 'max': 0.0}, {'current': 2506.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.032, 'min': 0.0, 'max': 0.0}, {'current': 2507.317, 'min': 0.0, 'max': 0.0}, {'current': 2505.961, 'min': 0.0, 'max': 0.0}, {'current': 2506.426, 'min': 0.0, 'max': 0.0}, {'current': 2498.555, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2503.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.429, 'min': 0.0, 'max': 0.0}, {'current': 2502.572, 'min': 0.0, 'max': 0.0}, {'current': 2503.604, 'min': 0.0, 'max': 0.0}, {'current': 2504.86, 'min': 0.0, 'max': 0.0}, {'current': 2500.01, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2503.704, 'min': 0.0, 'max': 0.0}, {'current': 2503.206, 'min': 0.0, 'max': 0.0}, {'current': 2500.531, 'min': 0.0, 'max': 0.0}, {'current': 2502.875, 'min': 0.0, 'max': 0.0}, {'current': 2500.19, 'min': 0.0, 'max': 0.0}, {'current': 2500.872, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2505.075, 'min': 0.0, 'max': 0.0}, {'current': 2503.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.85, 'min': 0.0, 'max': 0.0}, {'current': 2504.509, 'min': 0.0, 'max': 0.0}, {'current': 2504.271, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6454200744629}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:44:01,791 INFO HandlerThread:1879 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:44:01,791 INFO HandlerThread:1879 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:44:01,791 DEBUG HandlerThread:1879 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:44:01,819 DEBUG HandlerThread:1879 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:44:01,819 DEBUG HandlerThread:1879 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:44:02,534 DEBUG HandlerThread:1879 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:44:02,581 INFO HandlerThread:1879 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:44:02,585 DEBUG SenderThread:1879 [sender.py:send():336] send: files +2023-03-09 23:44:02,585 INFO SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:44:02,600 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:44:02,616 DEBUG SenderThread:1879 [sender.py:send():336] send: telemetry +2023-03-09 23:44:02,626 DEBUG SenderThread:1879 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:44:02,790 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/conda-environment.yaml +2023-03-09 23:44:02,803 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json +2023-03-09 23:44:02,819 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt +2023-03-09 23:44:03,640 DEBUG SenderThread:1879 [sender.py:send():336] send: config +2023-03-09 23:44:03,732 INFO wandb-upload_0:1879 [upload_job.py:push():138] Uploaded file /tmp/tmp01qe7zccwandb/qvio2pk7-wandb-metadata.json +2023-03-09 23:44:03,793 INFO Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log +2023-03-09 23:44:05,810 INFO Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log +2023-03-09 23:44:07,644 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:12,645 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:17,593 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:17,593 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:17,646 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:22,595 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:22,652 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:23,826 INFO Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml +2023-03-09 23:44:27,596 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:28,126 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:32,598 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:33,126 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:37,599 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:38,127 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:42,601 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:43,128 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:47,602 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:48,128 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:52,603 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:53,129 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:44:57,604 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:44:58,130 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:01,745 DEBUG SystemMonitor:1879 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:45:01,750 DEBUG SenderThread:1879 [sender.py:send():336] send: stats +2023-03-09 23:45:02,606 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:03,751 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:07,607 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:08,752 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:12,608 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:13,752 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:17,609 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:18,753 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:22,610 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:23,754 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:27,612 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:28,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:31,752 DEBUG SenderThread:1879 [sender.py:send():336] send: stats +2023-03-09 23:45:32,613 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:34,754 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:37,614 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:39,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:42,617 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:44,755 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:47,618 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:49,756 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:52,619 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:54,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:45:57,620 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:45:59,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:46:01,755 DEBUG SenderThread:1879 [sender.py:send():336] send: stats +2023-03-09 23:46:02,621 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:46:05,757 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:46:07,623 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:46:10,758 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:46:12,624 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:46:15,758 DEBUG HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:46:15,881 ERROR memory :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) +2023-03-09 23:46:15,881 ERROR memory :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) +2023-03-09 23:46:15,955 ERROR cpu :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) +2023-03-09 23:46:15,957 ERROR cpu :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77) +2023-03-09 23:46:16,690 WARNING StreamThr :1879 [internal.py:is_dead():416] Internal process exiting, parent pid 77 disappeared +2023-03-09 23:46:16,690 ERROR StreamThr :1879 [internal.py:wandb_internal():153] Internal process shutdown. +2023-03-09 23:46:16,744 INFO MainThread:1879 [internal.py:handle_exit():77] Internal process exited diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log new file mode 100644 index 0000000..42429d4 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Configure stats pid to 77 +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:43:51,519 INFO MainThread:77 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():602] starting backend +2023-03-09 23:43:51,520 INFO MainThread:77 [wandb_init.py:init():606] setting up manager +2023-03-09 23:43:51,531 INFO MainThread:77 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:43:51,532 INFO MainThread:77 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:43:51,540 INFO MainThread:77 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:43:51,541 INFO MainThread:77 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:43:56,652 INFO MainThread:77 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:44:01,678 INFO MainThread:77 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:44:01,678 INFO MainThread:77 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:44:02,591 INFO MainThread:77 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:44:02,592 INFO MainThread:77 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:44:02,593 INFO MainThread:77 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:44:02,594 INFO MainThread:77 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/run-w9y25r7k.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/run-w9y25r7k.wandb new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml new file mode 100644 index 0000000..05a1704 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml @@ -0,0 +1,39 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: false + python_version: 3.7.11 + start_time: 1678377055.792902 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log new file mode 100644 index 0000000..d034b38 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log @@ -0,0 +1,8 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 +Epoch 1/5: 0%| | 0/5088 [00:00=7eTyulU@`lUK9n7B2@8Gui{1Y83_^eL2DOZm1H893YEg?AX_I2C zX)04Plw`~drn*j2s){vSVlhtb6&%AI5RkvzdGg3V`Ta-dxAy@!27NNvlt5V81R&y1 z#kDvgGvhc;QsO7?je=cK3e7k2Q4vK-PQyd+K-WU7!8kQ2r}3KIw2mM7P+t}|#+IV} z)}g&7{1en&HZ~ScKYJ*K_-0HPfPU6$>J{4oK!uYKmMGUNMZ3Dj#0i1ZFj!}_ZB?yu zu}!%+^br=rDm)A)YKs{(bj2QT$gZM9o^#2+Vb{6 z0Azn{8-xP#P7nHvds|ZY9hjRxe=&y^IBj!b3-LCsy8dzy^}lo7M7hBhA$3^FyYY}4 zuU{s^(uICb;Oiu|kyePL{~7*VSHxx O6q*%pCt(*x=kf>5(iP|c literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml new file mode 100644 index 0000000..4e0df3c --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml @@ -0,0 +1,39 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: false + python_version: 3.7.11 + start_time: 1678377318.650455 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log new file mode 100644 index 0000000..3926efa --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log @@ -0,0 +1,7 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json new file mode 100644 index 0000000..e2efc37 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:55:25.128694", + "startedAt": "2023-03-09T15:55:18.607606", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "n1d6fa00775e48a18f80acc1355070be-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2502.0496374999993, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.767, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.475, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.398, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.156, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.385, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.678, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.643, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.163, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.395, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.117, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.927, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.953, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.177, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.15, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.575, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.649, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.6, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.068, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.553, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.434, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.396, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.163, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.428, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.283, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.769, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.975, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.573, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.008, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.441, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.077, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.185, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.401, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.465, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.575, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.949, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.373, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.085, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.672, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.784, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.976, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.28, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.92, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.817, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.072, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.767, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.946, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.257, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.292, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.734, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.093, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.904, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.615, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.676, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.732, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.387, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.376, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.485, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.054, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.539, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.286, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.724, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2508.643, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.17, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.69, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.158, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.016, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.234, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.759, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.361, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.269, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.461, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.742, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.565, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.111, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.64724349975586 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log new file mode 100644 index 0000000..6ee098e --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log @@ -0,0 +1,133 @@ +2023-03-09 23:55:18,660 INFO StreamThr :6152 [internal.py:wandb_internal():90] W&B internal server running at pid: 6152, started at: 2023-03-09 23:55:18.657868 +2023-03-09 23:55:18,661 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:55:18,663 INFO WriterThread:6152 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/run-wqdqlwjv.wandb +2023-03-09 23:55:18,671 DEBUG SenderThread:6152 [sender.py:send():336] send: header +2023-03-09 23:55:18,671 DEBUG SenderThread:6152 [sender.py:send():336] send: run +2023-03-09 23:55:18,690 INFO SenderThread:6152 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/wqdqlwjv +2023-03-09 23:55:19,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:55:19,991 INFO SenderThread:6152 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files +2023-03-09 23:55:19,991 INFO SenderThread:6152 [sender.py:_start_run_threads():1081] run started: wqdqlwjv with start time 1678377318.650455 +2023-03-09 23:55:19,991 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:55:20,015 INFO SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:55:20,016 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:55:20,998 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json +2023-03-09 23:55:24,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:25,017 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:25,076 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:55:25,081 DEBUG HandlerThread:6152 [system_info.py:__init__():31] System info init +2023-03-09 23:55:25,081 DEBUG HandlerThread:6152 [system_info.py:__init__():46] System info init done +2023-03-09 23:55:25,081 INFO HandlerThread:6152 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:55:25,082 INFO SystemMonitor:6152 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:55:25,082 INFO HandlerThread:6152 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:55:25,082 INFO SystemMonitor:6152 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:55:25,083 INFO SystemMonitor:6152 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:55:25,083 INFO SystemMonitor:6152 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:55:25,084 INFO SystemMonitor:6152 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:55:25,085 INFO SystemMonitor:6152 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:55:25,128 DEBUG HandlerThread:6152 [system_info.py:probe():195] Probing system +2023-03-09 23:55:25,168 DEBUG HandlerThread:6152 [git.py:repo():40] git repository is invalid +2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_info.py:probe():240] Probing system done +2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:55:25.128694', 'startedAt': '2023-03-09T15:55:18.607606', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2502.0496374999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.475, 'min': 0.0, 'max': 0.0}, {'current': 2500.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.156, 'min': 0.0, 'max': 0.0}, {'current': 2500.385, 'min': 0.0, 'max': 0.0}, {'current': 2501.678, 'min': 0.0, 'max': 0.0}, {'current': 2499.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.163, 'min': 0.0, 'max': 0.0}, {'current': 2500.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.117, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2497.953, 'min': 0.0, 'max': 0.0}, {'current': 2499.177, 'min': 0.0, 'max': 0.0}, {'current': 2501.15, 'min': 0.0, 'max': 0.0}, {'current': 2503.575, 'min': 0.0, 'max': 0.0}, {'current': 2501.649, 'min': 0.0, 'max': 0.0}, {'current': 2500.6, 'min': 0.0, 'max': 0.0}, {'current': 2501.068, 'min': 0.0, 'max': 0.0}, {'current': 2501.553, 'min': 0.0, 'max': 0.0}, {'current': 2500.434, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2499.163, 'min': 0.0, 'max': 0.0}, {'current': 2501.428, 'min': 0.0, 'max': 0.0}, {'current': 2501.283, 'min': 0.0, 'max': 0.0}, {'current': 2501.769, 'min': 0.0, 'max': 0.0}, {'current': 2499.975, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.441, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.077, 'min': 0.0, 'max': 0.0}, {'current': 2499.185, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2500.465, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2499.949, 'min': 0.0, 'max': 0.0}, {'current': 2501.373, 'min': 0.0, 'max': 0.0}, {'current': 2501.085, 'min': 0.0, 'max': 0.0}, {'current': 2500.672, 'min': 0.0, 'max': 0.0}, {'current': 2498.784, 'min': 0.0, 'max': 0.0}, {'current': 2504.976, 'min': 0.0, 'max': 0.0}, {'current': 2505.28, 'min': 0.0, 'max': 0.0}, {'current': 2503.92, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.817, 'min': 0.0, 'max': 0.0}, {'current': 2507.072, 'min': 0.0, 'max': 0.0}, {'current': 2504.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2503.257, 'min': 0.0, 'max': 0.0}, {'current': 2500.292, 'min': 0.0, 'max': 0.0}, {'current': 2502.734, 'min': 0.0, 'max': 0.0}, {'current': 2507.093, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.904, 'min': 0.0, 'max': 0.0}, {'current': 2505.615, 'min': 0.0, 'max': 0.0}, {'current': 2503.676, 'min': 0.0, 'max': 0.0}, {'current': 2503.732, 'min': 0.0, 'max': 0.0}, {'current': 2505.387, 'min': 0.0, 'max': 0.0}, {'current': 2500.376, 'min': 0.0, 'max': 0.0}, {'current': 2501.485, 'min': 0.0, 'max': 0.0}, {'current': 2506.054, 'min': 0.0, 'max': 0.0}, {'current': 2506.539, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2507.286, 'min': 0.0, 'max': 0.0}, {'current': 2500.724, 'min': 0.0, 'max': 0.0}, {'current': 2508.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.17, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2505.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2500.234, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2504.361, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2505.742, 'min': 0.0, 'max': 0.0}, {'current': 2504.565, 'min': 0.0, 'max': 0.0}, {'current': 2506.111, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64724349975586}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:55:25,169 INFO HandlerThread:6152 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:55:25,169 INFO HandlerThread:6152 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:55:25,169 DEBUG HandlerThread:6152 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:55:25,187 DEBUG HandlerThread:6152 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:55:25,187 DEBUG HandlerThread:6152 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:55:25,907 DEBUG HandlerThread:6152 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:55:25,960 INFO HandlerThread:6152 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:55:25,963 DEBUG SenderThread:6152 [sender.py:send():336] send: files +2023-03-09 23:55:25,964 INFO SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:55:25,999 DEBUG SenderThread:6152 [sender.py:send():336] send: telemetry +2023-03-09 23:55:25,999 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:55:26,008 DEBUG SenderThread:6152 [sender.py:send():336] send: config +2023-03-09 23:55:26,055 DEBUG SenderThread:6152 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:55:26,143 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt +2023-03-09 23:55:26,165 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json +2023-03-09 23:55:26,171 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/conda-environment.yaml +2023-03-09 23:55:27,136 INFO Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log +2023-03-09 23:55:27,210 INFO wandb-upload_0:6152 [upload_job.py:push():138] Uploaded file /tmp/tmpcpr9x05xwandb/qhhlrewz-wandb-metadata.json +2023-03-09 23:55:29,138 INFO Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log +2023-03-09 23:55:30,921 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:35,922 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:40,923 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:40,972 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:40,973 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:45,923 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:45,974 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:50,929 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:55:50,991 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:52,159 INFO Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml +2023-03-09 23:55:55,977 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:55:56,728 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:00,979 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:01,728 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:05,980 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:06,729 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:10,982 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:11,730 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:15,983 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:16,731 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:20,984 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:21,731 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:25,085 DEBUG SystemMonitor:6152 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:56:25,090 DEBUG SenderThread:6152 [sender.py:send():336] send: stats +2023-03-09 23:56:25,986 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:27,092 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:30,987 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:32,092 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:35,988 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:37,093 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:40,990 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:42,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:45,991 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:47,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:50,992 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:52,095 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:56:55,093 DEBUG SenderThread:6152 [sender.py:send():336] send: stats +2023-03-09 23:56:55,994 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:56:58,094 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:00,995 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:03,095 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:05,996 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:08,096 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:10,997 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:13,096 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:15,998 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:18,097 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:21,000 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:23,098 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:25,095 DEBUG SenderThread:6152 [sender.py:send():336] send: stats +2023-03-09 23:57:26,001 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:29,097 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:31,003 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:34,098 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:36,004 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:39,099 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:41,005 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:44,099 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:46,007 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:49,100 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:51,008 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:57:54,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:57:55,098 DEBUG SenderThread:6152 [sender.py:send():336] send: stats +2023-03-09 23:57:56,009 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:00,100 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:01,010 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:05,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:06,012 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:10,101 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:11,013 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:15,102 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:16,014 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:20,103 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:21,015 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:25,101 DEBUG SenderThread:6152 [sender.py:send():336] send: stats +2023-03-09 23:58:26,018 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:26,103 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:31,018 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:31,104 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:36,019 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:36,105 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:41,021 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:58:41,108 DEBUG HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:58:45,110 INFO MainThread:6152 [internal.py:handle_exit():77] Internal process exited diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log new file mode 100644 index 0000000..45cfd92 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:55:18,636 INFO MainThread:4410 [wandb_setup.py:_flush():76] Configure stats pid to 4410 +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():602] starting backend +2023-03-09 23:55:18,637 INFO MainThread:4410 [wandb_init.py:init():606] setting up manager +2023-03-09 23:55:18,648 INFO MainThread:4410 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:55:18,650 INFO MainThread:4410 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:55:18,654 INFO MainThread:4410 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:55:18,655 INFO MainThread:4410 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:55:19,986 INFO MainThread:4410 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:55:25,018 INFO MainThread:4410 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:55:25,018 INFO MainThread:4410 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:55:25,970 INFO MainThread:4410 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:55:25,971 INFO MainThread:4410 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:55:25,972 INFO MainThread:4410 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:55:25,972 INFO MainThread:4410 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/run-wqdqlwjv.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/run-wqdqlwjv.wandb new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/conda-environment.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/config.yaml b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/config.yaml new file mode 100644 index 0000000..1515ea5 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/config.yaml @@ -0,0 +1,39 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: false + python_version: 3.7.11 + start_time: 1678377605.403295 + t: + 1: + - 1 + - 55 + 2: + - 1 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 5 +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/output.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/output.log new file mode 100644 index 0000000..90e6336 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/output.log @@ -0,0 +1,8 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 +Epoch 1/5: 0%| | 0/5088 [00:00 +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 5088 + Checkpoints: True + Images scaling: 0.5 +Epoch 1/5: 0%| | 0/5088 [05:05 + img_scale=args.scale, + File "train.py", line 102, in train_model + for batch in train_loader: + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/dataloader.py", line 637, in __next__ + data = self._next_data() + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/dataloader.py", line 920, in _next_data + return self._process_data(data) + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/dataloader.py", line 947, in _process_data + data.reraise() + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/_utils.py", line 33, in reraise + raise exception +TypeError: Caught TypeError in DataLoader worker process 0. +Original Traceback (most recent call last): + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 308, in default_collate + return elem_type({key: default_collate([d[key] for d in batch]) for key in elem}) + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 308, in + return elem_type({key: default_collate([d[key] for d in batch]) for key in elem}) + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 331, in default_collate + raise TypeError(default_collate_err_msg_format.format(elem_type)) +TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/worker.py", line 308, in _worker_loop + data = fetcher.fetch(index) + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/fetch.py", line 52, in fetch + return self.collate_fn(data) + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 311, in default_collate + return {key: default_collate([d[key] for d in batch]) for key in elem} + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 311, in + return {key: default_collate([d[key] for d in batch]) for key in elem} + File "/root/.local/lib/python3.7/site-packages/ms_adapter-0.1.0-py3.7.egg/ms_adapter/pytorch/utils/data/_utils/collate.py", line 331, in default_collate + raise TypeError(default_collate_err_msg_format.format(elem_type)) +TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json new file mode 100644 index 0000000..b40c65f --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T16:10:11.489682", + "startedAt": "2023-03-09T16:10:04.972918", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "n1d6fa00775e48a18f80acc1355070be-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.301262500001, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.047, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.411, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.435, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.764, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.859, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.134, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.275, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.673, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.909, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.007, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.929, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.087, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.824, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.743, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.804, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.567, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.118, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.038, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.891, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.327, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.645, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.951, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.433, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.418, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.066, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.648, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.969, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.963, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.319, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.62, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.649, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.927, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.265, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.725, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.876, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.546, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.283, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.421, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.741, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.052, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.357, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.673, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.326, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.107, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.212, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2509.28, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.431, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.538, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.876, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.961, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.357, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.3, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.458, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.894, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.906, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.885, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.946, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.513, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.644, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.685, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.655, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.238, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.672, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.909, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.59, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.478, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.85, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6511535644531 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json new file mode 100644 index 0000000..6e9da68 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 306}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log new file mode 100644 index 0000000..4a38ce3 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log @@ -0,0 +1,312 @@ +2023-03-10 00:10:05,030 INFO StreamThr :17139 [internal.py:wandb_internal():90] W&B internal server running at pid: 17139, started at: 2023-03-10 00:10:05.028238 +2023-03-10 00:10:05,031 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status +2023-03-10 00:10:05,033 INFO WriterThread:17139 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb +2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: header +2023-03-10 00:10:05,038 DEBUG SenderThread:17139 [sender.py:send():336] send: run +2023-03-10 00:10:05,174 INFO SenderThread:17139 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/215re05m +2023-03-10 00:10:06,363 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: check_version +2023-03-10 00:10:06,364 INFO SenderThread:17139 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files +2023-03-10 00:10:06,364 INFO SenderThread:17139 [sender.py:_start_run_threads():1081] run started: 215re05m with start time 1678378205.024339 +2023-03-10 00:10:06,364 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: summary_record +2023-03-10 00:10:06,422 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:10:06,422 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: check_version +2023-03-10 00:10:07,368 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:10:11,365 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:11,423 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:11,444 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: run_start +2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():31] System info init +2023-03-10 00:10:11,449 DEBUG HandlerThread:17139 [system_info.py:__init__():46] System info init done +2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:start():183] Starting system monitor +2023-03-10 00:10:11,449 INFO SystemMonitor:17139 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-10 00:10:11,449 INFO HandlerThread:17139 [system_monitor.py:probe():204] Collecting system info +2023-03-10 00:10:11,450 INFO SystemMonitor:17139 [interfaces.py:start():187] Started cpu monitoring +2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started disk monitoring +2023-03-10 00:10:11,451 INFO SystemMonitor:17139 [interfaces.py:start():187] Started gpu monitoring +2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started memory monitoring +2023-03-10 00:10:11,452 INFO SystemMonitor:17139 [interfaces.py:start():187] Started network monitoring +2023-03-10 00:10:11,489 DEBUG HandlerThread:17139 [system_info.py:probe():195] Probing system +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [git.py:repo():40] git repository is invalid +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_info.py:probe():240] Probing system done +2023-03-10 00:10:11,516 DEBUG HandlerThread:17139 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T16:10:11.489682', 'startedAt': '2023-03-09T16:10:04.972918', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.301262500001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.047, 'min': 0.0, 'max': 0.0}, {'current': 2505.411, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2497.435, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.275, 'min': 0.0, 'max': 0.0}, {'current': 2497.673, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2501.929, 'min': 0.0, 'max': 0.0}, {'current': 2500.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.824, 'min': 0.0, 'max': 0.0}, {'current': 2498.743, 'min': 0.0, 'max': 0.0}, {'current': 2498.804, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2501.118, 'min': 0.0, 'max': 0.0}, {'current': 2502.038, 'min': 0.0, 'max': 0.0}, {'current': 2499.891, 'min': 0.0, 'max': 0.0}, {'current': 2500.327, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2499.951, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.433, 'min': 0.0, 'max': 0.0}, {'current': 2501.418, 'min': 0.0, 'max': 0.0}, {'current': 2501.066, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.969, 'min': 0.0, 'max': 0.0}, {'current': 2499.963, 'min': 0.0, 'max': 0.0}, {'current': 2499.319, 'min': 0.0, 'max': 0.0}, {'current': 2499.62, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2502.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2504.876, 'min': 0.0, 'max': 0.0}, {'current': 2499.546, 'min': 0.0, 'max': 0.0}, {'current': 2504.283, 'min': 0.0, 'max': 0.0}, {'current': 2506.421, 'min': 0.0, 'max': 0.0}, {'current': 2499.741, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2498.673, 'min': 0.0, 'max': 0.0}, {'current': 2505.326, 'min': 0.0, 'max': 0.0}, {'current': 2504.107, 'min': 0.0, 'max': 0.0}, {'current': 2505.212, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2509.28, 'min': 0.0, 'max': 0.0}, {'current': 2500.431, 'min': 0.0, 'max': 0.0}, {'current': 2499.538, 'min': 0.0, 'max': 0.0}, {'current': 2500.876, 'min': 0.0, 'max': 0.0}, {'current': 2503.961, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2500.3, 'min': 0.0, 'max': 0.0}, {'current': 2500.458, 'min': 0.0, 'max': 0.0}, {'current': 2499.894, 'min': 0.0, 'max': 0.0}, {'current': 2502.906, 'min': 0.0, 'max': 0.0}, {'current': 2499.885, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2504.513, 'min': 0.0, 'max': 0.0}, {'current': 2504.644, 'min': 0.0, 'max': 0.0}, {'current': 2503.685, 'min': 0.0, 'max': 0.0}, {'current': 2500.655, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.238, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.672, 'min': 0.0, 'max': 0.0}, {'current': 2504.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.59, 'min': 0.0, 'max': 0.0}, {'current': 2500.478, 'min': 0.0, 'max': 0.0}, {'current': 2500.85, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6511535644531}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():214] Finished collecting system info +2023-03-10 00:10:11,517 INFO HandlerThread:17139 [system_monitor.py:probe():217] Publishing system info +2023-03-10 00:10:11,517 DEBUG HandlerThread:17139 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-10 00:10:11,546 DEBUG HandlerThread:17139 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-10 00:10:12,268 DEBUG HandlerThread:17139 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-10 00:10:12,302 INFO HandlerThread:17139 [system_monitor.py:probe():219] Finished publishing system info +2023-03-10 00:10:12,305 DEBUG SenderThread:17139 [sender.py:send():336] send: files +2023-03-10 00:10:12,306 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-10 00:10:12,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: stop_status +2023-03-10 00:10:12,333 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: stop_status +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt +2023-03-10 00:10:12,546 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json +2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: telemetry +2023-03-10 00:10:12,800 DEBUG SenderThread:17139 [sender.py:send():336] send: config +2023-03-10 00:10:13,506 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /tmp/tmpf905ug1_wandb/11iue9v6-wandb-metadata.json +2023-03-10 00:10:13,547 INFO Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:10:15,549 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:10:16,804 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:21,805 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:26,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:27,314 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:31,806 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:32,316 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:36,812 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:37,317 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:37,569 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml +2023-03-10 00:10:42,303 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:42,319 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:47,304 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:47,320 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:52,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:52,322 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:10:57,305 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:10:57,323 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:02,306 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:02,324 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:07,307 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:07,325 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:11,452 DEBUG SystemMonitor:17139 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-10 00:11:11,458 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:11:12,326 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:12,459 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:17,327 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:17,460 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:22,329 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:22,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:27,330 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:27,461 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:32,331 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:32,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:37,332 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:37,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:41,460 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:11:42,333 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:43,462 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:47,334 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:48,463 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:52,336 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:53,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:11:57,337 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:11:58,464 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:02,338 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:03,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:07,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:08,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:11,463 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:12:12,340 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:14,465 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:17,342 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:19,466 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:22,343 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:24,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:27,344 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:29,467 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:32,345 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:34,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:37,346 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:39,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:41,466 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:12:42,348 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:45,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:47,349 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:50,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:52,350 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:12:55,469 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:12:57,351 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:00,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:02,352 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:05,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:07,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:10,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:11,469 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:13:12,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:16,470 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:17,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:21,471 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:22,357 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:26,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:27,358 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:31,472 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:32,359 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:36,473 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:37,360 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:41,548 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:41,550 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:13:42,362 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:46,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:47,373 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:51,552 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:52,375 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:13:56,553 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:13:57,376 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:01,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:02,377 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:06,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:07,378 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:11,553 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:14:12,379 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:12,554 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:17,381 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:17,555 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:22,382 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:22,556 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:27,383 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:27,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:32,384 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:32,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:37,386 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:37,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:41,555 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:14:42,387 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:43,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:47,388 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:48,557 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:52,389 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:53,558 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:14:57,390 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:14:58,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:02,391 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:03,559 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:07,393 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:08,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:11,558 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:15:12,394 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:14,560 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:17,409 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:18,419 DEBUG SenderThread:17139 [sender.py:send():336] send: exit +2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():559] handling exit code: 1 +2023-03-10 00:15:18,419 INFO SenderThread:17139 [sender.py:send_exit():561] handling runtime: 306 +2023-03-10 00:15:18,723 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:15:18,724 INFO SenderThread:17139 [sender.py:send_exit():567] send defer +2023-03-10 00:15:18,724 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,725 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-10 00:15:18,725 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-10 00:15:18,725 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 1 +2023-03-10 00:15:18,726 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,726 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-10 00:15:18,726 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-10 00:15:18,726 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 2 +2023-03-10 00:15:18,808 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:18,808 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-10 00:15:18,808 INFO HandlerThread:17139 [system_monitor.py:finish():193] Stopping system monitor +2023-03-10 00:15:19,020 DEBUG SystemMonitor:17139 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-10 00:15:19,021 DEBUG SystemMonitor:17139 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined cpu monitor +2023-03-10 00:15:19,109 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined disk monitor +2023-03-10 00:15:19,110 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:20,110 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined gpu monitor +2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined memory monitor +2023-03-10 00:15:20,111 INFO HandlerThread:17139 [interfaces.py:finish():199] Joined network monitor +2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:20,111 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:20,113 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-10 00:15:20,114 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 3 +2023-03-10 00:15:20,114 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,114 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-10 00:15:20,208 DEBUG SenderThread:17139 [sender.py:send():336] send: stats +2023-03-10 00:15:20,209 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:20,210 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-10 00:15:20,210 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 4 +2023-03-10 00:15:20,211 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:20,211 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-10 00:15:20,211 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,211 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-10 00:15:20,212 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 5 +2023-03-10 00:15:20,212 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,212 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-10 00:15:20,212 DEBUG SenderThread:17139 [sender.py:send():336] send: summary +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-10 00:15:20,233 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-10 00:15:20,233 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 6 +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-10 00:15:20,234 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-10 00:15:20,234 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 7 +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:20,234 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:20,234 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-10 00:15:20,235 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:20,235 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-10 00:15:20,511 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:21,211 INFO Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:23,339 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 8 +2023-03-10 00:15:23,339 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:23,339 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:23,340 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-10 00:15:23,340 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:23,340 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 9 +2023-03-10 00:15:23,341 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:23,341 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-10 00:15:23,341 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:23,341 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-10 00:15:23,341 INFO SenderThread:17139 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-10 00:15:24,213 INFO SenderThread:17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml conda-environment.yaml +2023-03-10 00:15:24,214 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log output.log +2023-03-10 00:15:24,215 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt requirements.txt +2023-03-10 00:15:24,220 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml config.yaml +2023-03-10 00:15:24,241 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json wandb-summary.json +2023-03-10 00:15:24,256 INFO SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json wandb-metadata.json +2023-03-10 00:15:24,261 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 10 +2023-03-10 00:15:24,277 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:24,287 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-10 00:15:24,419 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:24,424 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-10 00:15:24,424 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher +2023-03-10 00:15:25,468 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report +2023-03-10 00:15:25,536 INFO wandb-upload_4:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json +2023-03-10 00:15:25,612 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive +2023-03-10 00:15:25,889 INFO wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log +2023-03-10 00:15:25,909 INFO wandb-upload_2:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt +2023-03-10 00:15:26,005 INFO wandb-upload_3:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml +2023-03-10 00:15:26,205 INFO Thread-12 :17139 [sender.py:transition_state():587] send defer: 11 +2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-10 00:15:26,206 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-10 00:15:26,206 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher +2023-03-10 00:15:26,206 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 12 +2023-03-10 00:15:26,206 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:26,206 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-10 00:15:26,207 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:26,207 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-10 00:15:28,352 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 13 +2023-03-10 00:15:28,353 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:28,353 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-10 00:15:28,353 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-10 00:15:28,353 INFO SenderThread:17139 [sender.py:transition_state():587] send defer: 14 +2023-03-10 00:15:28,354 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer +2023-03-10 00:15:28,354 INFO HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: final +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send():336] send: footer +2023-03-10 00:15:28,354 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: defer +2023-03-10 00:15:28,354 INFO SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-10 00:15:28,355 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-10 00:15:28,355 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit +2023-03-10 00:15:28,356 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: server_info +2023-03-10 00:15:28,356 DEBUG SenderThread:17139 [sender.py:send_request():363] send_request: server_info +2023-03-10 00:15:28,392 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: get_summary +2023-03-10 00:15:28,438 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-10 00:15:29,089 INFO MainThread:17139 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-10 00:15:29,089 DEBUG HandlerThread:17139 [handler.py:handle_request():144] handle_request: shutdown +2023-03-10 00:15:29,090 INFO HandlerThread:17139 [handler.py:finish():842] shutting down handler +2023-03-10 00:15:29,356 INFO WriterThread:17139 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb +2023-03-10 00:15:30,089 INFO SenderThread:17139 [sender.py:finish():1504] shutting down sender +2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher +2023-03-10 00:15:30,089 INFO SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log new file mode 100644 index 0000000..a86b536 --- /dev/null +++ b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Configure stats pid to 15417 +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():546] calling init triggers +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():602] starting backend +2023-03-10 00:10:05,010 INFO MainThread:15417 [wandb_init.py:init():606] setting up manager +2023-03-10 00:10:05,022 INFO MainThread:15417 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-10 00:10:05,024 INFO MainThread:15417 [wandb_init.py:init():613] backend started and connected +2023-03-10 00:10:05,028 INFO MainThread:15417 [wandb_init.py:init():701] updated telemetry +2023-03-10 00:10:05,029 INFO MainThread:15417 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-10 00:10:06,363 INFO MainThread:15417 [wandb_run.py:_on_init():2131] communicating current version +2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_run.py:_on_init():2140] got version response +2023-03-10 00:10:11,424 INFO MainThread:15417 [wandb_init.py:init():789] starting run threads in backend +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_console_start():2112] atexit reg +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-10 00:10:12,313 INFO MainThread:15417 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-10 00:10:12,314 INFO MainThread:15417 [wandb_init.py:init():831] run started, returning control to user process +2023-03-10 00:10:12,315 INFO MainThread:15417 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5} +2023-03-10 00:15:30,109 WARNING MsgRouterThr:15417 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb b/official/cv/unet2d/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb new file mode 100644 index 0000000000000000000000000000000000000000..7edafa8febb7adf4beae44d9e3e605ec954944ba GIT binary patch literal 34601 zcmeI5e~@HVb;q}NW|pB@WLc0Ovud7;=&ad&^M3Tt31R|Bio{=4lCqQ{Q@uTJ=f}=W zkKH}M2DQwprC=%|YXAiaA}J{(#t@B|DkyA~5?8xfR9!xA^usSdjH@>vEJa7Y zdgs>H%(v&G`h0TP{^tHGORu*#ni%_|Jul7rJ5MMdU*0lvX=`P6@4?dM%U`!Rxf0LT z_O+s@Uaw}A2D-AZ5w+UwIIUKrdbBs&({J_mM|+|xOUI4hcg3wYZP|9`r#^q&yGze5 zkByh^94SqV+*vB^+>(a%Fpi&pOccf`{^gv_lcj-?T-^EW_zsL@BssPG^zuo%2YZ&Y zervYXZ-tA!&dOK*YFqP<@fV}7o;0`KexxTnB25Rn@wOlR{AV}dA!ERwnVadh`uK1x&h|vr@@eH0daX;c znf83v-oM&eSm_@ujrO~1S^32Bu?x#{Gre|e34cBkh1H|);?4hb#p_QlmBzO$-dEbb zb^Ng@#|KaH~i;69NIkb&!yw_(1UoYs(KNf`P!lfPtgB=Z4gZ< z9fOOaXgZ3{dg1Qig)p9}-o-~R%u6rddY}!n9$(xuZk74O2fhtucI>hLi|xPS-=@z# z`X_zxO*ehv0zB#1#gm*TUG>f%;Yp(4vmxYBre`o>dW?SFZt@-Vml^!9s7 z6UUcFXBT?=v9{7m7)_jB-nx?YFYR>qhrLWKx$!tk8_inSm^jTgb+h&*2TRAKXi0I= zndMVgyPbBpoGo{{2g24Rt%ar5-lgm%=Nv2@o7SRQTnQ`j#0ljs`Cs5+wXo8hcvksD z_quMcr>+=}YqfeUOluR*D4*oEuV&r0?$c-`Op}SN<&pO48tw&KuTD&qPsD$PQHX!e z*uRX!X0vFESz8pQ)Ru_0W~I0;6>T+oUPZPg#dTHI)(q=$(N<$^aagSvZFSKW({&Be zmQX*NtPNA6`ng@{5;I}oHJcMVOdA&_4l6Zk^e9uOMk!6*-d1Xp#&jhYCk`9-wp^Vk z#3+m2a(OU~wo&R6*=KTraN8z~Obe7SfyR<*SfjBdT%shb+D0xB%+oeXiJH`-gi92M z^=dItbYRey@l0h~d3Tvam`{=~;R-=pQYhgHK?M6qsZfosOqoJhciJv>e^Mq8`eQHT z`k=35Un$oo4ig#)T+*NfSn9Ts>w`zyMy3xs3rAo~DCGjd+t@}f5IRHxrCgv?`7v|b ziWJCRSz!X9L$XyrVS_BF!WF88)s#9U6|&k^q(a)2=;^F5g)p57-IWWZ z{is<+6(-OaKo#EEC{*PN!7}N-RjE*;L03wJs$^hQsZgxEsUAv|DKr9COLvt5rDQ@? zuRz5}m_V3L>)fhRp@e3v$`rztb|f_^P(&T7ae?9xjb@lNCXnmkYf_&k$zEgnpg-uW zna&y$Xbf>ArAeuAh2XyIeWgOplW^ z$b5>eS^l^9=ioxoiw|MlP;p!Sm#dviv+gTf{XT{? z7T3d6k&SMP!@;(~uR7PgoWKzQS6i=YG#A>&xEjXCedHG}*#S~MZu%NU${*LHeAUmt ze6Jwo^`W;KM|B;qUFdCu%^KA9LGLk1QmrP4;X|3X8aC9J8J-`TM70Dy7O})Z_ngni zYjM5NRA%F#GH_@JDkq8(OdDTD$vUW9RUyz^#}`o&CTY9HNvrtC?uv*AlPhir8J<5ZDa;C3QVq1f0#fc z#_g)e9zKSp8vz6#JDGth69Q889KG5cO8@ zvAa(fe2hjiX~DK2~nTUda{0m9~*9v&-T z4d-K>h$-5*E?}|3X9+yUZioDbz+;Sqd{)f0(cLkwP~b7FikwT#oD3LK=AZ*k(xu9|OrrTEWMaybO^exwIa8A=k$ZYyyvA z|G@S+8v9?AtOLjMv@aU zMo9$Y0c2;C%mD$At?C{?wsgn=8Kbn`fDJn9;d;A7Xdi~R8b8uI)xYTKlM;A3}61Ruk??3FBn!SoQN%t$yN zW0Wcvr58NMW3Hj&HjAecyNla9jZ#M@V2QtD!7!p2zuV1Pb*1HC({SuSTFG*XDw9P8C0tVJ>;x~&O1LMI20KLV?k;RT4^7n8<`6(LZ5D9E+%g?{bXwh3zMLmtsr%r zg%+kDWW9|xtDO_F-WwEcT$sdW@sQ#l@P+6ojG;hTDUEe$9zwR5;vr;PJ{E-RR^J67 zYsHHWaeZ7g5`>I1Iy4kc$WYeyRv>cKNl*wxZn$SDxI%a)xo&~TI9q8Ox%D9QTMU-w zV`oLAkkj0lwGZZutbMSt0}mpTJ_3>5xaUD+eX2-5ks~smNYwyN$Y>j$KX$Fa1ITu7 zCXQLSDI*UcTPhHM3@d;e(Bt5MjJ{f|^tzqmLcYA*Y%X73rr?!7kr@vh0Wn;Kc^=rS zK2(-p$ZSV@5VJ%$9m7N7#9PtGT#xcEl&0fHEWokIADcmPJcc@C%7Tvd)}pvjszcBy z;26u<-j%6?zS`B}VPove3an=GjA5rA*@w!tWlc-aF*fhEN8I~p2V=&`=@_LF^4tQB z>+2n`aMp-mW9)co$#OP^>vl54+tk#&Ku1OnCh_SALDyEHxDWE;E zGxx*V&6w-BmqF2<`og?0RuP0W*GV62d7Q^67zg!Jt|9C41Ro=~k=N&ZJmA%M23ffZ z@=ixb$gZ>W5VDP6f{=k5_L*ECL|byRoRHyxXk*V)$f#D+O$8q}9Ml(ljFa`0uJrJ+ z1JB%cuoCPux$$5i6u<-@H=VqPNFf8&>}|O|Zq8QlF^rkS^YF1c4VIHEk5P(b4LJ!u zMhK@GIp<@9A&RvekJZM*Hm)a+$RNYEXrehFn-2oFlSwc*=2U+v1Gu&WT7MyAlXScJuc9jD_;)WCom(F%a5(@iP{qjr_1(-d84% zYg-(UF%Dupo;^kztvi9oP_9^80*~F*EON*0c^JXR+UyDknM=vF$|0LUILF% z7$oH6ciW2Ewk9@#>IZ6bmVp<`Xd8$!p4a~*Rsc~mrsUKXH`lT~iSn>$7modA*U zmo8(m2~Pv4ny2yggH%)DVFen)z0i^sXzY4e4;tIlA4g+&Cdzq;(-jC5_K{4P+!q?R zZ^cbjZDQOr9RezJaCNLno#0_V+!sZjxiADd@+*;25tMw))!6>=^WRvCE#?7 zfjHe^1~fp9jQ?<wD@OHq^M;6mIR;~0p^pssMn!5)MjClLqUX~^k{JSuAF zW)4`@1wq}$>w6g4E?dFK2*l(`J&bI((wvcV{xJ@SLo&3XZ9mT*qm3ezKx7?fih)RV zY<16r$WE%83534t4N^{rEJUHeNtqu`$Y`U!3P46Bl0u09WGn;n`r`bt-jkV83P2wC zaSU!exRN4`_#)t&6TXp7)3=QlW;*gB~=n7O$*Ua^3 zx;FX#8`pW%WM%EPJKQ^K3D!Ad;atiMNN@g)0dYu1Zz;PfQpoDtFJ>}%3R!(4 zj!f3W$aYEuBLmlH#dAi626j<*$=uyK3FiaE|wR4mDXbuKpsZ6 z0nEe5@FKrajI0>R`}-A`Q}tc7p+}JMRjhR&xfy@$j=K&`0FqCf{tE@kKh7cf^KX5N zfaD$aAVr=;F5^claL58ctLRQpa-){OqaysQe+D_JHgJrYK{ALjUO$bDy3S(&$p~ch zT{{NJSS@PRYrx zN7mK2qPJWi%rJRbo=1kT6iY-J852pz5;-KR!{qjKx4>j1q^Ks%F&QRD2~7_rTapu$ z4BaVaiX#>{uur)fPRV#A9hBfFEYM~p;Dj?kooo+*`La=3Ffuj~WFvx+^_P9jd#f8lK;6jSw9z?duPL9a#D)A7q_CGo6GkHdu*0uy7yZbK=$iPFp z#o~mFLF%MOj@H~WJ5wnL8M`{-CXqx|k-$u|FNsV5D3Ztn=SaBqxcd?g$a#b@&lH3_ zz?Cd);Yw@j9zwP*SP-&~YQ-~o?1Bp>4z@W54#@h~3;jP}Q3;#X?1$TT1_0#k)7L9Ne*2%PL&*31?FZf}0NG(+bIKx4 zqIwOst3oX2<0Puou#f;Y>t~Viv-1rkG?RhfLk0xZ*5_mVh|6QZ$M{iOqEhZ?G?Wbi zWB|R&cxv=yZdiT)gnn^CE+e8@7NS&IiI|%KvVj==G{(nphdSA!+oT7!Z7e>a3~VC_ z2V|^h;xrE*M??&Qk6nWC@UcU=OqqOtV0?k#V@PVB$;45CpMtxOk12e4lF0TjfZ${0 z@l1aNA7lGQ-kJ09x=CcW9?9ug1$Xk!0*_r~1s>yP(?|k=#{=gxxcQ*WL$ROpF@9CU z7D#y#S=*%1N92$Zph$JW$7rNs3O>fVrkG+tuEi6~kN8mAnyX_Cx zN5|n}m~wsDg0{#ZyPHhG$1aKpJ{~wN!9utjY&jmofGiq$_}C6pkjHp5NRnEz-9~PLbD{{Y4d1RHF^yZQEc4EVMjyX(jyI_UcwNv^u+-}HcTI2LtuinF?0+vS3pv%XkhdQC>3cR{A@Bdz;Xl0s zw>oX{71Z_5y!S0PmL|>(g7X)avS6ov3sbmyVCU`t?>Nc={9NGfU|}UVYq>MKwv?T% zUw1_Bg?d4&JJ$>KE24I9U}#t0eCfCv+Ud^FuDbu7H*R2PhtE#u-ho?9EM7!IyYBw? zVrYLjbZAkcho*--gYjicdUX3b-9W!SON~2kTECjme#eFGczmMW&N2MEu{MM-~~Z1TiO@qpQ}$McWuB@?!Bx2Z6)I~9T~s) z@EZ?rz*261;L!ZTxYe^3@3u02|KHz!xHR$ikGG7?BwuY-EiGg>>#ucJjF9T(16i*( zqqS83*odBgaNx$W((|_*J@3Er-5b$!*QbB?{0DKXVDV#C&ksFx3-rA8ao4kA%+eK& zY%}a;co|$T+mJ=R_>`wzsuX;-qu~F3`0qBN;GH*I{>!i9Ry!AOu?jx;gO5YOL%(VT zXY|T;R&tu^g>0nB%}NdpP-30oi1pwX?%s%4wcFnGoo|kp z#-FqJejCkiy5e4lbIfwy(9e)Soeb;rfegDO;ZD zLz2nM_h$#D1NNtr7tLOT&o2&Cz?)V2=%{z`F8u*o)JN(buiJ&Y=6LX{4=+a?@L979 z?S8wnyqdF!@r_Tiv8M6x$P*({&Mf!lW-zPEt$vvQ-z4d?Ygd`B6(%ul zz-M_4^xE#M^zlm0pr5VuI^EvxU}bH2^+3?-c3TJX|Js|WkJWti0@%>fLa*QZ9azuW z%4~2}8^qfSo~QXCyinicd1sf)=dE?s+dk*<+E(=v&7iX{=+9?CYj5Wg$H(jrS}PrV z-nDKg=(O8w-EKBJT`oTnmuT)$JS%i2Z<$%@bXF<$fQ$BJc!z1$4fX#|F09OEmpu*( zO+Ve~bv1`#{juvzG{qSwGqZ1HGAEBGf?tT&St~g>JxcfYgz$c5lz#8`WEkL+*X14k zn`w{v=*~|oUOLm^r7zy|osIC)8{c>M2hYTO0~KuV$KipdWm|~&X7W_ zK1pKQc%~Z%d+BRCyqct6-D0kjwGxzsh8|_4Tk=X04%|`kmlXjpI9rTmId0^X?J*Hv zsYPa(ij=%ii#Domdg`?HaKAXmh?FvYPm|YaaaTHs$B!}UCz@0}uhOE8(&nBbtvwgW zi?jpTcu$qqzV3!sY1R7!slX%-F*emfd(&B7rZqo@ppRhjDy=&ZQKZE;!Du2ykruxz zL8pj3XBa8{&ug^$_Zo|yHq*lwyy|CLMhAZz7oDOR$JUZ%n%Dz$EXlOJSMreKH3)|dMgYF?QcNg<;blUZsU`kv!rk zqYjIvKcYzMe$!4AX)z^qjKp80E&3zv$6mt|Blk8{{xw7-$4KX literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-pytorch/README.md b/official/cv/unet2d/2dunet-pytorch/README.md new file mode 100644 index 0000000..cb4d4c9 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/README.md @@ -0,0 +1,189 @@ +# U-Net: Semantic segmentation with PyTorch + + + + + +![input and output for a random image in the test dataset](https://i.imgur.com/GD8FcB7.png) + + +Customized implementation of the [U-Net](https://arxiv.org/abs/1505.04597) in PyTorch for Kaggle's [Carvana Image Masking Challenge](https://www.kaggle.com/c/carvana-image-masking-challenge) from high definition images. + +- [Quick start](#quick-start) + - [Without Docker](#without-docker) + - [With Docker](#with-docker) +- [Description](#description) +- [Usage](#usage) + - [Docker](#docker) + - [Training](#training) + - [Prediction](#prediction) +- [Weights & Biases](#weights--biases) +- [Pretrained model](#pretrained-model) +- [Data](#data) + +## Quick start + +### Without Docker + +1. [Install CUDA](https://developer.nvidia.com/cuda-downloads) + +2. [Install PyTorch 1.13 or later](https://pytorch.org/get-started/locally/) + +3. Install dependencies +```bash +pip install -r requirements.txt +``` + +4. Download the data and run training: +```bash +bash scripts/download_data.sh +python train.py --amp +``` + +### With Docker + +1. [Install Docker 19.03 or later:](https://docs.docker.com/get-docker/) +```bash +curl https://get.docker.com | sh && sudo systemctl --now enable docker +``` +2. [Install the NVIDIA container toolkit:](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) +```bash +distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \ + && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \ + && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list +sudo apt-get update +sudo apt-get install -y nvidia-docker2 +sudo systemctl restart docker +``` +3. [Download and run the image:](https://hub.docker.com/repository/docker/milesial/unet) +```bash +sudo docker run --rm --shm-size=8g --ulimit memlock=-1 --gpus all -it milesial/unet +``` + +4. Download the data and run training: +```bash +bash scripts/download_data.sh +python train.py --amp +``` + +## Description +This model was trained from scratch with 5k images and scored a [Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) of 0.988423 on over 100k test images. + +It can be easily used for multiclass segmentation, portrait segmentation, medical segmentation, ... + + +## Usage +**Note : Use Python 3.6 or newer** + +### Docker + +A docker image containing the code and the dependencies is available on [DockerHub](https://hub.docker.com/repository/docker/milesial/unet). +You can download and jump in the container with ([docker >=19.03](https://docs.docker.com/get-docker/)): + +```console +docker run -it --rm --shm-size=8g --ulimit memlock=-1 --gpus all milesial/unet +``` + + +### Training + +```console +> python train.py -h +usage: train.py [-h] [--epochs E] [--batch-size B] [--learning-rate LR] + [--load LOAD] [--scale SCALE] [--validation VAL] [--amp] + +Train the UNet on images and target masks + +optional arguments: + -h, --help show this help message and exit + --epochs E, -e E Number of epochs + --batch-size B, -b B Batch size + --learning-rate LR, -l LR + Learning rate + --load LOAD, -f LOAD Load model from a .pth file + --scale SCALE, -s SCALE + Downscaling factor of the images + --validation VAL, -v VAL + Percent of the data that is used as validation (0-100) + --amp Use mixed precision +``` + +By default, the `scale` is 0.5, so if you wish to obtain better results (but use more memory), set it to 1. + +Automatic mixed precision is also available with the `--amp` flag. [Mixed precision](https://arxiv.org/abs/1710.03740) allows the model to use less memory and to be faster on recent GPUs by using FP16 arithmetic. Enabling AMP is recommended. + + +### Prediction + +After training your model and saving it to `MODEL.pth`, you can easily test the output masks on your images via the CLI. + +To predict a single image and save it: + +`python predict.py -i image.jpg -o output.jpg` + +To predict a multiple images and show them without saving them: + +`python predict.py -i image1.jpg image2.jpg --viz --no-save` + +```console +> python predict.py -h +usage: predict.py [-h] [--model FILE] --input INPUT [INPUT ...] + [--output INPUT [INPUT ...]] [--viz] [--no-save] + [--mask-threshold MASK_THRESHOLD] [--scale SCALE] + +Predict masks from input images + +optional arguments: + -h, --help show this help message and exit + --model FILE, -m FILE + Specify the file in which the model is stored + --input INPUT [INPUT ...], -i INPUT [INPUT ...] + Filenames of input images + --output INPUT [INPUT ...], -o INPUT [INPUT ...] + Filenames of output images + --viz, -v Visualize the images as they are processed + --no-save, -n Do not save the output masks + --mask-threshold MASK_THRESHOLD, -t MASK_THRESHOLD + Minimum probability value to consider a mask pixel white + --scale SCALE, -s SCALE + Scale factor for the input images +``` +You can specify which model file to use with `--model MODEL.pth`. + +## Weights & Biases + +The training progress can be visualized in real-time using [Weights & Biases](https://wandb.ai/). Loss curves, validation curves, weights and gradient histograms, as well as predicted masks are logged to the platform. + +When launching a training, a link will be printed in the console. Click on it to go to your dashboard. If you have an existing W&B account, you can link it + by setting the `WANDB_API_KEY` environment variable. If not, it will create an anonymous run which is automatically deleted after 7 days. + + +## Pretrained model +A [pretrained model](https://github.com/milesial/Pytorch-UNet/releases/tag/v3.0) is available for the Carvana dataset. It can also be loaded from torch.hub: + +```python +net = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana', pretrained=True, scale=0.5) +``` +Available scales are 0.5 and 1.0. + +## Data +The Carvana data is available on the [Kaggle website](https://www.kaggle.com/c/carvana-image-masking-challenge/data). + +You can also download it using the helper script: + +``` +bash scripts/download_data.sh +``` + +The input images and target masks should be in the `data/imgs` and `data/masks` folders respectively (note that the `imgs` and `masks` folder should not contain any sub-folder or any other files, due to the greedy data-loader). For Carvana, images are RGB and masks are black and white. + +You can use your own dataset as long as you make sure it is loaded properly in `utils/data_loading.py`. + + +--- + +Original paper by Olaf Ronneberger, Philipp Fischer, Thomas Brox: + +[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597) + +![network architecture](https://i.imgur.com/jeDVpqF.png) diff --git a/official/cv/unet2d/2dunet-pytorch/evaluate.py b/official/cv/unet2d/2dunet-pytorch/evaluate.py new file mode 100644 index 0000000..9a4e3ba --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/evaluate.py @@ -0,0 +1,40 @@ +import torch +import torch.nn.functional as F +from tqdm import tqdm + +from utils.dice_score import multiclass_dice_coeff, dice_coeff + + +@torch.inference_mode() +def evaluate(net, dataloader, device, amp): + net.eval() + num_val_batches = len(dataloader) + dice_score = 0 + + # iterate over the validation set + with torch.autocast(device.type if device.type != 'mps' else 'cpu', enabled=amp): + for batch in tqdm(dataloader, total=num_val_batches, desc='Validation round', unit='batch', leave=False): + image, mask_true = batch['image'], batch['mask'] + + # move images and labels to correct device and type + image = image.to(device=device, dtype=torch.float32, memory_format=torch.channels_last) + mask_true = mask_true.to(device=device, dtype=torch.long) + + # predict the mask + mask_pred = net(image) + + if net.n_classes == 1: + assert mask_true.min() >= 0 and mask_true.max() <= 1, 'True mask indices should be in [0, 1]' + mask_pred = (F.sigmoid(mask_pred) > 0.5).float() + # compute the Dice score + dice_score += dice_coeff(mask_pred, mask_true, reduce_batch_first=False) + else: + assert mask_true.min() >= 0 and mask_true.max() < net.n_classes, 'True mask indices should be in [0, n_classes[' + # convert to one-hot format + mask_true = F.one_hot(mask_true, net.n_classes).permute(0, 3, 1, 2).float() + mask_pred = F.one_hot(mask_pred.argmax(dim=1), net.n_classes).permute(0, 3, 1, 2).float() + # compute the Dice score, ignoring background + dice_score += multiclass_dice_coeff(mask_pred[:, 1:], mask_true[:, 1:], reduce_batch_first=False) + + net.train() + return dice_score / max(num_val_batches, 1) diff --git a/official/cv/unet2d/2dunet-pytorch/hubconf.py b/official/cv/unet2d/2dunet-pytorch/hubconf.py new file mode 100644 index 0000000..46433eb --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/hubconf.py @@ -0,0 +1,23 @@ +import torch +from unet import UNet as _UNet + +def unet_carvana(pretrained=False, scale=0.5): + """ + UNet model trained on the Carvana dataset ( https://www.kaggle.com/c/carvana-image-masking-challenge/data ). + Set the scale to 0.5 (50%) when predicting. + """ + net = _UNet(n_channels=3, n_classes=2, bilinear=False) + if pretrained: + if scale == 0.5: + checkpoint = 'https://github.com/milesial/Pytorch-UNet/releases/download/v3.0/unet_carvana_scale0.5_epoch2.pth' + elif scale == 1.0: + checkpoint = 'https://github.com/milesial/Pytorch-UNet/releases/download/v3.0/unet_carvana_scale1.0_epoch2.pth' + else: + raise RuntimeError('Only 0.5 and 1.0 scales are available') + state_dict = torch.hub.load_state_dict_from_url(checkpoint, progress=True) + if 'mask_values' in state_dict: + state_dict.pop('mask_values') + net.load_state_dict(state_dict) + + return net + diff --git a/official/cv/unet2d/2dunet-pytorch/predict.py b/official/cv/unet2d/2dunet-pytorch/predict.py new file mode 100644 index 0000000..b74c460 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/predict.py @@ -0,0 +1,117 @@ +import argparse +import logging +import os + +import numpy as np +import torch +import torch.nn.functional as F +from PIL import Image +from torchvision import transforms + +from utils.data_loading import BasicDataset +from unet import UNet +from utils.utils import plot_img_and_mask + +def predict_img(net, + full_img, + device, + scale_factor=1, + out_threshold=0.5): + net.eval() + img = torch.from_numpy(BasicDataset.preprocess(None, full_img, scale_factor, is_mask=False)) + img = img.unsqueeze(0) + img = img.to(device=device, dtype=torch.float32) + + with torch.no_grad(): + output = net(img).cpu() + output = F.interpolate(output, (full_img.size[1], full_img.size[0]), mode='bilinear') + if net.n_classes > 1: + mask = output.argmax(dim=1) + else: + mask = torch.sigmoid(output) > out_threshold + + return mask[0].long().squeeze().numpy() + + +def get_args(): + parser = argparse.ArgumentParser(description='Predict masks from input images') + parser.add_argument('--model', '-m', default='MODEL.pth', metavar='FILE', + help='Specify the file in which the model is stored') + parser.add_argument('--input', '-i', metavar='INPUT', nargs='+', help='Filenames of input images', required=True) + parser.add_argument('--output', '-o', metavar='OUTPUT', nargs='+', help='Filenames of output images') + parser.add_argument('--viz', '-v', action='store_true', + help='Visualize the images as they are processed') + parser.add_argument('--no-save', '-n', action='store_true', help='Do not save the output masks') + parser.add_argument('--mask-threshold', '-t', type=float, default=0.5, + help='Minimum probability value to consider a mask pixel white') + parser.add_argument('--scale', '-s', type=float, default=0.5, + help='Scale factor for the input images') + parser.add_argument('--bilinear', action='store_true', default=False, help='Use bilinear upsampling') + parser.add_argument('--classes', '-c', type=int, default=2, help='Number of classes') + + return parser.parse_args() + + +def get_output_filenames(args): + def _generate_name(fn): + return f'{os.path.splitext(fn)[0]}_OUT.png' + + return args.output or list(map(_generate_name, args.input)) + + +def mask_to_image(mask: np.ndarray, mask_values): + if isinstance(mask_values[0], list): + out = np.zeros((mask.shape[-2], mask.shape[-1], len(mask_values[0])), dtype=np.uint8) + elif mask_values == [0, 1]: + out = np.zeros((mask.shape[-2], mask.shape[-1]), dtype=bool) + else: + out = np.zeros((mask.shape[-2], mask.shape[-1]), dtype=np.uint8) + + if mask.ndim == 3: + mask = np.argmax(mask, axis=0) + + for i, v in enumerate(mask_values): + out[mask == i] = v + + return Image.fromarray(out) + + +if __name__ == '__main__': + args = get_args() + logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s') + + in_files = args.input + out_files = get_output_filenames(args) + + net = UNet(n_channels=3, n_classes=args.classes, bilinear=args.bilinear) + + device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') + logging.info(f'Loading model {args.model}') + logging.info(f'Using device {device}') + + net.to(device=device) + state_dict = torch.load(args.model, map_location=device) + mask_values = state_dict.pop('mask_values', [0, 1]) + net.load_state_dict(state_dict) + + logging.info('Model loaded!') + + for i, filename in enumerate(in_files): + logging.info(f'Predicting image {filename} ...') + img = Image.open(filename) + + mask = predict_img(net=net, + full_img=img, + scale_factor=args.scale, + out_threshold=args.mask_threshold, + device=device) + + if not args.no_save: + out_filename = out_files[i] + result = mask_to_image(mask, mask_values) + result.save(out_filename) + logging.info(f'Mask saved to {out_filename}') + + if args.viz: + logging.info(f'Visualizing results for image {filename}, close to continue...') + plot_img_and_mask(img, mask) diff --git a/official/cv/unet2d/2dunet-pytorch/requirements.txt b/official/cv/unet2d/2dunet-pytorch/requirements.txt new file mode 100644 index 0000000..256fb59 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/requirements.txt @@ -0,0 +1,5 @@ +matplotlib==3.6.2 +numpy==1.23.5 +Pillow==9.3.0 +tqdm==4.64.1 +wandb==0.13.5 diff --git a/official/cv/unet2d/2dunet-pytorch/scripts/download_data.sh b/official/cv/unet2d/2dunet-pytorch/scripts/download_data.sh new file mode 100644 index 0000000..a151ede --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/scripts/download_data.sh @@ -0,0 +1,27 @@ +#!/bin/bash + +if [[ ! -f ~/.kaggle/kaggle.json ]]; then + echo -n "Kaggle username: " + read USERNAME + echo + echo -n "Kaggle API key: " + read APIKEY + + mkdir -p ~/.kaggle + echo "{\"username\":\"$USERNAME\",\"key\":\"$APIKEY\"}" > ~/.kaggle/kaggle.json + chmod 600 ~/.kaggle/kaggle.json +fi + +pip install kaggle --upgrade + +kaggle competitions download -c carvana-image-masking-challenge -f train_hq.zip +unzip train_hq.zip +mv train_hq/* data/imgs/ +rm -d train_hq +rm train_hq.zip + +kaggle competitions download -c carvana-image-masking-challenge -f train_masks.zip +unzip train_masks.zip +mv train_masks/* data/masks/ +rm -d train_masks +rm train_masks.zip diff --git a/official/cv/unet2d/2dunet-pytorch/train.py b/official/cv/unet2d/2dunet-pytorch/train.py new file mode 100644 index 0000000..8748b23 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/train.py @@ -0,0 +1,238 @@ +import argparse +import logging +import os +os.environ['CUDA_VISIBLE_DEVICES'] = '1' +import random +import sys +import torch +import torch.nn as nn +import torch.nn.functional as F +import torchvision.transforms as transforms +import torchvision.transforms.functional as TF +from pathlib import Path +from torch import optim +from torch.utils.data import DataLoader, random_split +from tqdm import tqdm + +import wandb +from evaluate import evaluate +from unet import UNet +from utils.data_loading import BasicDataset, CarvanaDataset +from utils.dice_score import dice_loss + +dir_img = Path('./data/imgs/') +dir_mask = Path('./data/masks/') +dir_checkpoint = Path('./checkpoints/') + + +def train_model( + model, + device, + epochs: int = 5, + batch_size: int = 1, + learning_rate: float = 1e-5, + val_percent: float = 0.1, + save_checkpoint: bool = True, + img_scale: float = 0.5, + amp: bool = False, + weight_decay: float = 1e-8, + momentum: float = 0.999, + gradient_clipping: float = 1.0, +): + # 1. Create dataset + try: + dataset = CarvanaDataset(dir_img, dir_mask, img_scale) + except (AssertionError, RuntimeError, IndexError): + dataset = BasicDataset(dir_img, dir_mask, img_scale) + + # 2. Split into train / validation partitions + n_val = int(len(dataset) * val_percent) + n_train = len(dataset) - n_val + train_set, val_set = random_split(dataset, [n_train, n_val], generator=torch.Generator().manual_seed(0)) + + # 3. Create data loaders + loader_args = dict(batch_size=batch_size, num_workers=os.cpu_count(), pin_memory=True) + train_loader = DataLoader(train_set, shuffle=True, **loader_args) + val_loader = DataLoader(val_set, shuffle=False, drop_last=True, **loader_args) + + # (Initialize logging) + experiment = wandb.init(project='U-Net', resume='allow', anonymous='must') + experiment.config.update( + dict(epochs=epochs, batch_size=batch_size, learning_rate=learning_rate, + val_percent=val_percent, save_checkpoint=save_checkpoint, img_scale=img_scale, amp=amp) + ) + + logging.info(f'''Starting training: + Epochs: {epochs} + Batch size: {batch_size} + Learning rate: {learning_rate} + Training size: {n_train} + Validation size: {n_val} + Checkpoints: {save_checkpoint} + Device: {device.type} + Images scaling: {img_scale} + Mixed Precision: {amp} + ''') + + # 4. Set up the optimizer, the loss, the learning rate scheduler and the loss scaling for AMP + optimizer = optim.RMSprop(model.parameters(), + lr=learning_rate, weight_decay=weight_decay, momentum=momentum) + scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'max', patience=5) # goal: maximize Dice score + grad_scaler = torch.cuda.amp.GradScaler(enabled=amp) + criterion = nn.CrossEntropyLoss() if model.n_classes > 1 else nn.BCEWithLogitsLoss() + global_step = 0 + + # 5. Begin training + for epoch in range(1, epochs + 1): + model.train() + epoch_loss = 0 + with tqdm(total=n_train, desc=f'Epoch {epoch}/{epochs}', unit='img') as pbar: + for batch in train_loader: + images, true_masks = batch['image'], batch['mask'] + + assert images.shape[1] == model.n_channels, \ + f'Network has been defined with {model.n_channels} input channels, ' \ + f'but loaded images have {images.shape[1]} channels. Please check that ' \ + 'the images are loaded correctly.' + + images = images.to(device=device, dtype=torch.float32, memory_format=torch.channels_last) + true_masks = true_masks.to(device=device, dtype=torch.long) + + with torch.autocast(device.type if device.type != 'mps' else 'cpu', enabled=amp): + masks_pred = model(images) + if model.n_classes == 1: + loss = criterion(masks_pred.squeeze(1), true_masks.float()) + loss += dice_loss(F.sigmoid(masks_pred.squeeze(1)), true_masks.float(), multiclass=False) + else: + loss = criterion(masks_pred, true_masks) + loss += dice_loss( + F.softmax(masks_pred, dim=1).float(), + F.one_hot(true_masks, model.n_classes).permute(0, 3, 1, 2).float(), + multiclass=True + ) + + optimizer.zero_grad(set_to_none=True) + grad_scaler.scale(loss).backward() + torch.nn.utils.clip_grad_norm_(model.parameters(), gradient_clipping) + grad_scaler.step(optimizer) + grad_scaler.update() + + pbar.update(images.shape[0]) + global_step += 1 + epoch_loss += loss.item() + experiment.log({ + 'train loss': loss.item(), + 'step': global_step, + 'epoch': epoch + }) + pbar.set_postfix(**{'loss (batch)': loss.item()}) + + # Evaluation round + division_step = (n_train // (5 * batch_size)) + if division_step > 0: + if global_step % division_step == 0: + histograms = {} + for tag, value in model.named_parameters(): + tag = tag.replace('/', '.') + if not (torch.isinf(value) | torch.isnan(value)).any(): + histograms['Weights/' + tag] = wandb.Histogram(value.data.cpu()) + if not (torch.isinf(value.grad) | torch.isnan(value.grad)).any(): + histograms['Gradients/' + tag] = wandb.Histogram(value.grad.data.cpu()) + + val_score = evaluate(model, val_loader, device, amp) + scheduler.step(val_score) + + logging.info('Validation Dice score: {}'.format(val_score)) + try: + experiment.log({ + 'learning rate': optimizer.param_groups[0]['lr'], + 'validation Dice': val_score, + 'images': wandb.Image(images[0].cpu()), + 'masks': { + 'true': wandb.Image(true_masks[0].float().cpu()), + 'pred': wandb.Image(masks_pred.argmax(dim=1)[0].float().cpu()), + }, + 'step': global_step, + 'epoch': epoch, + **histograms + }) + except: + pass + + if save_checkpoint: + Path(dir_checkpoint).mkdir(parents=True, exist_ok=True) + state_dict = model.state_dict() + state_dict['mask_values'] = dataset.mask_values + torch.save(state_dict, str(dir_checkpoint / 'checkpoint_epoch{}.pth'.format(epoch))) + logging.info(f'Checkpoint {epoch} saved!') + + +def get_args(): + parser = argparse.ArgumentParser(description='Train the UNet on images and target masks') + parser.add_argument('--epochs', '-e', metavar='E', type=int, default=5, help='Number of epochs') + parser.add_argument('--batch-size', '-b', dest='batch_size', metavar='B', type=int, default=1, help='Batch size') + parser.add_argument('--learning-rate', '-l', metavar='LR', type=float, default=1e-5, + help='Learning rate', dest='lr') + parser.add_argument('--load', '-f', type=str, default=False, help='Load model from a .pth file') + parser.add_argument('--scale', '-s', type=float, default=0.5, help='Downscaling factor of the images') + parser.add_argument('--validation', '-v', dest='val', type=float, default=10.0, + help='Percent of the data that is used as validation (0-100)') + parser.add_argument('--amp', action='store_true', default=False, help='Use mixed precision') + parser.add_argument('--bilinear', action='store_true', default=False, help='Use bilinear upsampling') + parser.add_argument('--classes', '-c', type=int, default=2, help='Number of classes') + + return parser.parse_args() + + +if __name__ == '__main__': + args = get_args() + + logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s') + device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') + logging.info(f'Using device {device}') + + # Change here to adapt to your data + # n_channels=3 for RGB images + # n_classes is the number of probabilities you want to get per pixel + model = UNet(n_channels=3, n_classes=args.classes, bilinear=args.bilinear) + model = model.to(memory_format=torch.channels_last) + + logging.info(f'Network:\n' + f'\t{model.n_channels} input channels\n' + f'\t{model.n_classes} output channels (classes)\n' + f'\t{"Bilinear" if model.bilinear else "Transposed conv"} upscaling') + + if args.load: + state_dict = torch.load(args.load, map_location=device) + del state_dict['mask_values'] + model.load_state_dict(state_dict) + logging.info(f'Model loaded from {args.load}') + + model.to(device=device) + try: + train_model( + model=model, + epochs=args.epochs, + batch_size=args.batch_size, + learning_rate=args.lr, + device=device, + img_scale=args.scale, + val_percent=args.val / 100, + amp=args.amp + ) + except : + logging.error('Detected OutOfMemoryError! ' + 'Enabling checkpointing to reduce memory usage, but this slows down training. ' + 'Consider enabling AMP (--amp) for fast and memory efficient training') + torch.cuda.empty_cache() + #model.use_checkpointing() + train_model( + model=model, + epochs=args.epochs, + batch_size=args.batch_size, + learning_rate=args.lr, + device=device, + img_scale=args.scale, + val_percent=args.val / 100, + amp=args.amp + ) diff --git a/official/cv/unet2d/2dunet-pytorch/unet/__init__.py b/official/cv/unet2d/2dunet-pytorch/unet/__init__.py new file mode 100644 index 0000000..2e9b63b --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/unet/__init__.py @@ -0,0 +1 @@ +from .unet_model import UNet diff --git a/official/cv/unet2d/2dunet-pytorch/unet/unet_model.py b/official/cv/unet2d/2dunet-pytorch/unet/unet_model.py new file mode 100644 index 0000000..caf79f4 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/unet/unet_model.py @@ -0,0 +1,48 @@ +""" Full assembly of the parts to form the complete network """ + +from .unet_parts import * + + +class UNet(nn.Module): + def __init__(self, n_channels, n_classes, bilinear=False): + super(UNet, self).__init__() + self.n_channels = n_channels + self.n_classes = n_classes + self.bilinear = bilinear + + self.inc = (DoubleConv(n_channels, 64)) + self.down1 = (Down(64, 128)) + self.down2 = (Down(128, 256)) + self.down3 = (Down(256, 512)) + factor = 2 if bilinear else 1 + self.down4 = (Down(512, 1024 // factor)) + self.up1 = (Up(1024, 512 // factor, bilinear)) + self.up2 = (Up(512, 256 // factor, bilinear)) + self.up3 = (Up(256, 128 // factor, bilinear)) + self.up4 = (Up(128, 64, bilinear)) + self.outc = (OutConv(64, n_classes)) + + def forward(self, x): + x1 = self.inc(x) + x2 = self.down1(x1) + x3 = self.down2(x2) + x4 = self.down3(x3) + x5 = self.down4(x4) + x = self.up1(x5, x4) + x = self.up2(x, x3) + x = self.up3(x, x2) + x = self.up4(x, x1) + logits = self.outc(x) + return logits + + def use_checkpointing(self): + self.inc = torch.utils.checkpoint(self.inc) + self.down1 = torch.utils.checkpoint(self.down1) + self.down2 = torch.utils.checkpoint(self.down2) + self.down3 = torch.utils.checkpoint(self.down3) + self.down4 = torch.utils.checkpoint(self.down4) + self.up1 = torch.utils.checkpoint(self.up1) + self.up2 = torch.utils.checkpoint(self.up2) + self.up3 = torch.utils.checkpoint(self.up3) + self.up4 = torch.utils.checkpoint(self.up4) + self.outc = torch.utils.checkpoint(self.outc) \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/unet/unet_parts.py b/official/cv/unet2d/2dunet-pytorch/unet/unet_parts.py new file mode 100644 index 0000000..986ba25 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/unet/unet_parts.py @@ -0,0 +1,77 @@ +""" Parts of the U-Net model """ + +import torch +import torch.nn as nn +import torch.nn.functional as F + + +class DoubleConv(nn.Module): + """(convolution => [BN] => ReLU) * 2""" + + def __init__(self, in_channels, out_channels, mid_channels=None): + super().__init__() + if not mid_channels: + mid_channels = out_channels + self.double_conv = nn.Sequential( + nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(mid_channels), + nn.ReLU(inplace=True), + nn.Conv2d(mid_channels, out_channels, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(out_channels), + nn.ReLU(inplace=True) + ) + + def forward(self, x): + return self.double_conv(x) + + +class Down(nn.Module): + """Downscaling with maxpool then double conv""" + + def __init__(self, in_channels, out_channels): + super().__init__() + self.maxpool_conv = nn.Sequential( + nn.MaxPool2d(2), + DoubleConv(in_channels, out_channels) + ) + + def forward(self, x): + return self.maxpool_conv(x) + + +class Up(nn.Module): + """Upscaling then double conv""" + + def __init__(self, in_channels, out_channels, bilinear=True): + super().__init__() + + # if bilinear, use the normal convolutions to reduce the number of channels + if bilinear: + self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True) + self.conv = DoubleConv(in_channels, out_channels, in_channels // 2) + else: + self.up = nn.ConvTranspose2d(in_channels, in_channels // 2, kernel_size=2, stride=2) + self.conv = DoubleConv(in_channels, out_channels) + + def forward(self, x1, x2): + x1 = self.up(x1) + # input is CHW + diffY = x2.size()[2] - x1.size()[2] + diffX = x2.size()[3] - x1.size()[3] + + x1 = F.pad(x1, [diffX // 2, diffX - diffX // 2, + diffY // 2, diffY - diffY // 2]) + # if you have padding issues, see + # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a + # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd + x = torch.cat([x2, x1], dim=1) + return self.conv(x) + + +class OutConv(nn.Module): + def __init__(self, in_channels, out_channels): + super(OutConv, self).__init__() + self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1) + + def forward(self, x): + return self.conv(x) diff --git a/official/cv/unet2d/2dunet-pytorch/utils/__init__.py b/official/cv/unet2d/2dunet-pytorch/utils/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-pytorch/utils/data_loading.py b/official/cv/unet2d/2dunet-pytorch/utils/data_loading.py new file mode 100644 index 0000000..11296e7 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/utils/data_loading.py @@ -0,0 +1,117 @@ +import logging +import numpy as np +import torch +from PIL import Image +from functools import lru_cache +from functools import partial +from itertools import repeat +from multiprocessing import Pool +from os import listdir +from os.path import splitext, isfile, join +from pathlib import Path +from torch.utils.data import Dataset +from tqdm import tqdm + + +def load_image(filename): + ext = splitext(filename)[1] + if ext == '.npy': + return Image.fromarray(np.load(filename)) + elif ext in ['.pt', '.pth']: + return Image.fromarray(torch.load(filename).numpy()) + else: + return Image.open(filename) + + +def unique_mask_values(idx, mask_dir, mask_suffix): + mask_file = list(mask_dir.glob(idx + mask_suffix + '.*'))[0] + mask = np.asarray(load_image(mask_file)) + if mask.ndim == 2: + return np.unique(mask) + elif mask.ndim == 3: + mask = mask.reshape(-1, mask.shape[-1]) + return np.unique(mask, axis=0) + else: + raise ValueError(f'Loaded masks should have 2 or 3 dimensions, found {mask.ndim}') + + +class BasicDataset(Dataset): + def __init__(self, images_dir: str, mask_dir: str, scale: float = 1.0, mask_suffix: str = ''): + self.images_dir = Path(images_dir) + self.mask_dir = Path(mask_dir) + assert 0 < scale <= 1, 'Scale must be between 0 and 1' + self.scale = scale + self.mask_suffix = mask_suffix + + self.ids = [splitext(file)[0] for file in listdir(images_dir) if isfile(join(images_dir, file)) and not file.startswith('.')] + if not self.ids: + raise RuntimeError(f'No input file found in {images_dir}, make sure you put your images there') + + logging.info(f'Creating dataset with {len(self.ids)} examples') + logging.info('Scanning mask files to determine unique values') + with Pool() as p: + unique = list(tqdm( + p.imap(partial(unique_mask_values, mask_dir=self.mask_dir, mask_suffix=self.mask_suffix), self.ids), + total=len(self.ids) + )) + + self.mask_values = list(sorted(np.unique(np.concatenate(unique), axis=0).tolist())) + logging.info(f'Unique mask values: {self.mask_values}') + + def __len__(self): + return len(self.ids) + + @staticmethod + def preprocess(mask_values, pil_img, scale, is_mask): + w, h = pil_img.size + newW, newH = int(scale * w), int(scale * h) + assert newW > 0 and newH > 0, 'Scale is too small, resized images would have no pixel' + pil_img = pil_img.resize((newW, newH), resample=Image.NEAREST if is_mask else Image.BICUBIC) + img = np.asarray(pil_img) + + if is_mask: + mask = np.zeros((newH, newW), dtype=np.int64) + for i, v in enumerate(mask_values): + if img.ndim == 2: + mask[img == v] = i + else: + mask[(img == v).all(-1)] = i + + return mask + + else: + if img.ndim == 2: + img = img[np.newaxis, ...] + else: + img = img.transpose((2, 0, 1)) + + if (img > 1).any(): + img = img / 255.0 + + return img + + def __getitem__(self, idx): + name = self.ids[idx] + mask_file = list(self.mask_dir.glob(name + self.mask_suffix + '.*')) + img_file = list(self.images_dir.glob(name + '.*')) + + assert len(img_file) == 1, f'Either no image or multiple images found for the ID {name}: {img_file}' + assert len(mask_file) == 1, f'Either no mask or multiple masks found for the ID {name}: {mask_file}' + mask = load_image(mask_file[0]) + img = load_image(img_file[0]) + + assert img.size == mask.size, \ + f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}' + + img = self.preprocess(self.mask_values, img, self.scale, is_mask=False) + mask = self.preprocess(self.mask_values, mask, self.scale, is_mask=True) + + return { + 'image': torch.as_tensor(img.copy()).float().contiguous(), + 'mask': torch.as_tensor(mask.copy()).long().contiguous() + } + + +class CarvanaDataset(BasicDataset): + def __init__(self, images_dir, mask_dir, scale=1): + super().__init__(images_dir, mask_dir, scale, mask_suffix='_mask') diff --git a/official/cv/unet2d/2dunet-pytorch/utils/dice_score.py b/official/cv/unet2d/2dunet-pytorch/utils/dice_score.py new file mode 100644 index 0000000..c89eebe --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/utils/dice_score.py @@ -0,0 +1,28 @@ +import torch +from torch import Tensor + + +def dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): + # Average of Dice coefficient for all batches, or for a single mask + assert input.size() == target.size() + assert input.dim() == 3 or not reduce_batch_first + + sum_dim = (-1, -2) if input.dim() == 2 or not reduce_batch_first else (-1, -2, -3) + + inter = 2 * (input * target).sum(dim=sum_dim) + sets_sum = input.sum(dim=sum_dim) + target.sum(dim=sum_dim) + sets_sum = torch.where(sets_sum == 0, inter, sets_sum) + + dice = (inter + epsilon) / (sets_sum + epsilon) + return dice.mean() + + +def multiclass_dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6): + # Average of Dice coefficient for all classes + return dice_coeff(input.flatten(0, 1), target.flatten(0, 1), reduce_batch_first, epsilon) + + +def dice_loss(input: Tensor, target: Tensor, multiclass: bool = False): + # Dice loss (objective to minimize) between 0 and 1 + fn = multiclass_dice_coeff if multiclass else dice_coeff + return 1 - fn(input, target, reduce_batch_first=True) diff --git a/official/cv/unet2d/2dunet-pytorch/utils/utils.py b/official/cv/unet2d/2dunet-pytorch/utils/utils.py new file mode 100644 index 0000000..6ed8d8d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/utils/utils.py @@ -0,0 +1,13 @@ +import matplotlib.pyplot as plt + + +def plot_img_and_mask(img, mask): + classes = mask.max() + 1 + fig, ax = plt.subplots(1, classes + 1) + ax[0].set_title('Input image') + ax[0].imshow(img) + for i in range(classes): + ax[i + 1].set_title(f'Mask (class {i + 1})') + ax[i + 1].imshow(mask == i) + plt.xticks([]), plt.yticks([]) + plt.show() diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/debug-cli.root.log b/official/cv/unet2d/2dunet-pytorch/wandb/debug-cli.root.log new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/debug-internal.log new file mode 100644 index 0000000..4c52d3a --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/debug-internal.log @@ -0,0 +1,778 @@ +2023-03-09 23:27:14,841 INFO StreamThr :13248 [internal.py:wandb_internal():90] W&B internal server running at pid: 13248, started at: 2023-03-09 23:27:14.838951 +2023-03-09 23:27:14,842 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:27:14,843 INFO WriterThread:13248 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb +2023-03-09 23:27:14,848 DEBUG SenderThread:13248 [sender.py:send():336] send: header +2023-03-09 23:27:14,848 DEBUG SenderThread:13248 [sender.py:send():336] send: run +2023-03-09 23:27:14,863 INFO SenderThread:13248 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/76baujih +2023-03-09 23:27:16,098 INFO SenderThread:13248 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files +2023-03-09 23:27:16,098 INFO SenderThread:13248 [sender.py:_start_run_threads():1081] run started: 76baujih with start time 1678375634.838134 +2023-03-09 23:27:16,098 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:16,099 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:27:16,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:16,162 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:27:17,102 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:21,099 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:21,163 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:21,171 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:27:21,178 DEBUG HandlerThread:13248 [system_info.py:__init__():31] System info init +2023-03-09 23:27:21,178 DEBUG HandlerThread:13248 [system_info.py:__init__():46] System info init done +2023-03-09 23:27:21,178 INFO HandlerThread:13248 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:27:21,179 INFO SystemMonitor:13248 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:27:21,179 INFO HandlerThread:13248 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:27:21,179 INFO SystemMonitor:13248 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:27:21,180 INFO SystemMonitor:13248 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:27:21,181 INFO SystemMonitor:13248 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:27:21,181 INFO SystemMonitor:13248 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:27:21,182 INFO SystemMonitor:13248 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:27:21,219 DEBUG HandlerThread:13248 [system_info.py:probe():195] Probing system +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [git.py:repo():40] git repository is invalid +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_info.py:probe():240] Probing system done +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:27:21.219862', 'startedAt': '2023-03-09T15:27:14.806065', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2502.0773624999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.421, 'min': 0.0, 'max': 0.0}, {'current': 2500.46, 'min': 0.0, 'max': 0.0}, {'current': 2501.84, 'min': 0.0, 'max': 0.0}, {'current': 2500.951, 'min': 0.0, 'max': 0.0}, {'current': 2501.019, 'min': 0.0, 'max': 0.0}, {'current': 2499.524, 'min': 0.0, 'max': 0.0}, {'current': 2502.222, 'min': 0.0, 'max': 0.0}, {'current': 2500.672, 'min': 0.0, 'max': 0.0}, {'current': 2499.854, 'min': 0.0, 'max': 0.0}, {'current': 2506.5, 'min': 0.0, 'max': 0.0}, {'current': 2500.023, 'min': 0.0, 'max': 0.0}, {'current': 2499.192, 'min': 0.0, 'max': 0.0}, {'current': 2499.8, 'min': 0.0, 'max': 0.0}, {'current': 2500.343, 'min': 0.0, 'max': 0.0}, {'current': 2499.078, 'min': 0.0, 'max': 0.0}, {'current': 2500.029, 'min': 0.0, 'max': 0.0}, {'current': 2501.541, 'min': 0.0, 'max': 0.0}, {'current': 2498.581, 'min': 0.0, 'max': 0.0}, {'current': 2499.509, 'min': 0.0, 'max': 0.0}, {'current': 2500.341, 'min': 0.0, 'max': 0.0}, {'current': 2499.858, 'min': 0.0, 'max': 0.0}, {'current': 2500.388, 'min': 0.0, 'max': 0.0}, {'current': 2501.575, 'min': 0.0, 'max': 0.0}, {'current': 2500.535, 'min': 0.0, 'max': 0.0}, {'current': 2501.27, 'min': 0.0, 'max': 0.0}, {'current': 2499.294, 'min': 0.0, 'max': 0.0}, {'current': 2500.36, 'min': 0.0, 'max': 0.0}, {'current': 2500.768, 'min': 0.0, 'max': 0.0}, {'current': 2499.017, 'min': 0.0, 'max': 0.0}, {'current': 2505.29, 'min': 0.0, 'max': 0.0}, {'current': 2499.82, 'min': 0.0, 'max': 0.0}, {'current': 2500.378, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2499.261, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2507.621, 'min': 0.0, 'max': 0.0}, {'current': 2506.022, 'min': 0.0, 'max': 0.0}, {'current': 2505.289, 'min': 0.0, 'max': 0.0}, {'current': 2507.416, 'min': 0.0, 'max': 0.0}, {'current': 2506.087, 'min': 0.0, 'max': 0.0}, {'current': 2506.772, 'min': 0.0, 'max': 0.0}, {'current': 2506.081, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2506.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.015, 'min': 0.0, 'max': 0.0}, {'current': 2503.871, 'min': 0.0, 'max': 0.0}, {'current': 2507.194, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2506.906, 'min': 0.0, 'max': 0.0}, {'current': 2506.009, 'min': 0.0, 'max': 0.0}, {'current': 2501.082, 'min': 0.0, 'max': 0.0}, {'current': 2505.269, 'min': 0.0, 'max': 0.0}, {'current': 2508.067, 'min': 0.0, 'max': 0.0}, {'current': 2502.36, 'min': 0.0, 'max': 0.0}, {'current': 2505.23, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.176, 'min': 0.0, 'max': 0.0}, {'current': 2501.354, 'min': 0.0, 'max': 0.0}, {'current': 2503.515, 'min': 0.0, 'max': 0.0}, {'current': 2503.134, 'min': 0.0, 'max': 0.0}, {'current': 2500.134, 'min': 0.0, 'max': 0.0}, {'current': 2500.975, 'min': 0.0, 'max': 0.0}, {'current': 2504.278, 'min': 0.0, 'max': 0.0}, {'current': 2500.387, 'min': 0.0, 'max': 0.0}, {'current': 2504.299, 'min': 0.0, 'max': 0.0}, {'current': 2505.099, 'min': 0.0, 'max': 0.0}, {'current': 2498.784, 'min': 0.0, 'max': 0.0}, {'current': 2504.785, 'min': 0.0, 'max': 0.0}, {'current': 2505.195, 'min': 0.0, 'max': 0.0}, {'current': 2504.921, 'min': 0.0, 'max': 0.0}, {'current': 2499.207, 'min': 0.0, 'max': 0.0}, {'current': 2504.31, 'min': 0.0, 'max': 0.0}, {'current': 2499.904, 'min': 0.0, 'max': 0.0}, {'current': 2502.18, 'min': 0.0, 'max': 0.0}, {'current': 2504.924, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64785385131836}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:27:21,254 INFO HandlerThread:13248 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:27:21,254 INFO HandlerThread:13248 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:27:21,274 DEBUG HandlerThread:13248 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:27:21,274 DEBUG HandlerThread:13248 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:27:21,996 DEBUG HandlerThread:13248 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:27:22,031 INFO HandlerThread:13248 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:27:22,034 DEBUG SenderThread:13248 [sender.py:send():336] send: files +2023-03-09 23:27:22,034 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:27:22,079 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:27:22,085 DEBUG SenderThread:13248 [sender.py:send():336] send: telemetry +2023-03-09 23:27:22,095 DEBUG SenderThread:13248 [sender.py:send():336] send: config +2023-03-09 23:27:22,100 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:27:22,309 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json +2023-03-09 23:27:22,319 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt +2023-03-09 23:27:22,340 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml +2023-03-09 23:27:23,312 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:23,673 INFO wandb-upload_0:13248 [upload_job.py:push():138] Uploaded file /tmp/tmpsqy0gtaqwandb/hoj8mklj-wandb-metadata.json +2023-03-09 23:27:25,318 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:27,115 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:32,116 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:35,417 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:35,424 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:35,426 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:35,426 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:35,446 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:36,417 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:37,043 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:37,043 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:37,420 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:37,570 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:37,859 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:37,860 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:37,860 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:37,943 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:38,421 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:39,423 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:40,294 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:40,295 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:40,295 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:40,309 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:40,424 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:41,427 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:42,046 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:42,740 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:42,741 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:42,741 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:42,742 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:42,759 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:43,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:43,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:45,177 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:45,178 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:45,179 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:45,252 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:45,432 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:47,049 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:47,436 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:47,614 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:47,615 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:47,615 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:47,653 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:48,437 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:48,659 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:49,440 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:49,440 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml +2023-03-09 23:27:50,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:50,052 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:50,053 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:50,103 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:50,441 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:51,443 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:52,049 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:52,491 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:52,492 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:52,492 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:52,519 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:53,453 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:53,453 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:54,526 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:54,935 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:54,937 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:54,937 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:55,020 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:55,454 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:57,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:57,388 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:57,389 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:57,389 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:57,403 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:57,458 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:57,458 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:59,461 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:59,837 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:59,838 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:59,838 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:59,839 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:59,879 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:00,462 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:01,465 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:02,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:02,294 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:02,295 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:02,295 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:02,331 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:02,465 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:03,468 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:04,751 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:04,752 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:04,753 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:04,788 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:05,471 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:05,471 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:05,789 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:07,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:07,208 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:07,209 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:07,209 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:07,244 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:07,473 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:09,477 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:09,669 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:09,670 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:09,671 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:09,689 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:10,478 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:11,480 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:11,704 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:12,052 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:12,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:12,132 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:12,132 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:12,168 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:12,481 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:13,484 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:14,597 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:14,598 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:14,598 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:14,664 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:15,487 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:15,487 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:17,052 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:17,062 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:17,063 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:17,063 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:17,063 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:17,097 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:17,488 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:19,492 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:19,533 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:19,534 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:19,534 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:19,597 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:20,493 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:21,182 DEBUG SystemMonitor:13248 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:28:21,184 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:28:21,495 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:22,005 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:22,006 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:22,006 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:22,038 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:22,053 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:22,496 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:23,041 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:23,499 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:24,483 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:24,484 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:24,484 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:24,523 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:24,523 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:25,525 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:26,957 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:26,958 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:26,959 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:27,020 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:27,054 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:27,527 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:29,021 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:29,432 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:29,432 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:29,433 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:29,474 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:29,530 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:29,530 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:31,533 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:31,907 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:31,909 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:31,909 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:31,930 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:32,054 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:32,534 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:33,537 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:34,384 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:34,384 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:34,385 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:34,385 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:34,421 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:34,538 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:35,540 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:36,865 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:36,866 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:36,866 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:36,885 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:37,055 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:37,543 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:37,544 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:39,344 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:39,345 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:39,345 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:39,417 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:39,418 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:39,545 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:41,548 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:41,826 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:41,827 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:41,827 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:41,846 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:42,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:42,549 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:43,552 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:44,313 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:44,314 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:44,315 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:44,373 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:44,552 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:45,374 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:45,555 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:46,801 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:46,802 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:46,802 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:46,840 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:47,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:47,558 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:47,559 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:49,285 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:49,286 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:49,287 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:49,315 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:49,560 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:51,186 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:28:51,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:51,563 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:51,772 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:51,773 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:51,773 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:51,807 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:52,057 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:52,564 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:53,566 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:54,257 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:54,258 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:54,258 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:54,283 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:54,567 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:55,569 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:56,292 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:56,746 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:56,747 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:56,748 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:56,782 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:57,058 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:57,573 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:57,574 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:59,233 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:59,234 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:59,234 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:59,271 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:59,574 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:01,578 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:01,728 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:01,730 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:01,730 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:01,730 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:01,757 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:02,058 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:02,579 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:03,582 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:04,215 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:04,216 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:04,216 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:04,250 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:04,583 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:05,585 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:06,717 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:06,718 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:06,719 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:06,763 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:06,763 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:07,059 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:07,587 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:07,587 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:09,213 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:09,214 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:09,214 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:09,251 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:09,588 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:11,592 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:11,700 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:11,701 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:11,702 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:11,730 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:12,060 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:12,593 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:12,735 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:13,596 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:14,200 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:14,202 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:14,202 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:14,246 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:14,597 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:15,600 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:16,700 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:16,701 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:16,702 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:16,752 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:17,060 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:17,603 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:17,603 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:17,754 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:19,199 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:19,200 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:19,200 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:19,236 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:19,605 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:21,187 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:29:21,608 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:21,695 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:21,696 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:21,697 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:21,743 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:22,061 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:22,609 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:23,612 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:23,744 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:24,192 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:24,193 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:24,193 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:24,210 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:24,613 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:25,615 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:26,688 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:26,690 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:26,690 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:26,785 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:27,061 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:27,618 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:27,619 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:28,786 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:29,184 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:29,185 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:29,185 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:29,226 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:29,620 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:31,624 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:31,680 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:31,680 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:31,681 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:31,719 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:32,062 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:32,625 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:33,627 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:34,165 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:34,166 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:34,167 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:34,167 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:34,218 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:34,628 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:35,630 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:36,655 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:36,656 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:36,656 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:36,724 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:37,063 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:37,633 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:37,633 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:39,151 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:39,152 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:39,153 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:39,211 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:39,211 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:39,635 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:41,638 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:41,647 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:41,648 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:41,649 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:41,673 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:42,064 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:42,639 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:43,641 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:44,144 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:44,145 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:44,145 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:44,186 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:44,642 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:45,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:45,645 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:46,642 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:46,643 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:46,643 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:46,670 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:46,670 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:47,064 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:47,673 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:49,137 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:49,138 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:49,138 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:49,174 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:49,674 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:51,175 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:51,188 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:29:51,632 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:51,633 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:51,633 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:51,661 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:51,677 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:51,678 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:52,065 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:53,680 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:54,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:54,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:54,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:54,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:54,681 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:55,684 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:56,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:56,625 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:56,626 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:56,626 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:56,649 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:56,685 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:57,066 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:57,687 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:59,122 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:59,123 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:59,123 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:59,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:59,689 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:01,621 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:01,622 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:01,623 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:01,635 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:01,636 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:01,692 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:01,692 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:02,066 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:03,695 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:04,122 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:04,123 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:04,124 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:04,160 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:04,696 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:05,699 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:06,623 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:06,624 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:06,625 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:06,645 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:06,645 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:06,699 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:07,067 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:07,702 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:09,127 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:09,128 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:09,128 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:09,157 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:09,703 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:11,627 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:11,628 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:11,628 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:11,686 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:11,687 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:11,707 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:11,707 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:12,067 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:13,710 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:14,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:14,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:14,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:14,199 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:14,711 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:15,713 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:16,630 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:16,631 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:16,631 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:16,653 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:16,714 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:17,068 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:17,664 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:17,717 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:19,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:19,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:19,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:19,164 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:19,718 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:21,209 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:30:21,629 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:21,630 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:21,631 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:21,773 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:21,773 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:21,774 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:22,069 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:22,774 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:23,777 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:24,127 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:24,128 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:24,129 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:24,166 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:24,777 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:25,780 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:26,623 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:26,624 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:26,625 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:26,682 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:26,781 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:27,069 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:27,783 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:28,683 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:29,119 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:29,120 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:29,120 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:29,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:29,786 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:29,787 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:31,612 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:31,613 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:31,613 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:31,675 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:31,788 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:32,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:33,795 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:34,110 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:34,111 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:34,111 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:34,111 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:34,141 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:34,795 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:35,798 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:36,604 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:36,605 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:36,605 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:36,628 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:36,799 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:37,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:37,815 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:39,101 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:39,102 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:39,102 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:39,178 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:39,180 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:39,831 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:39,831 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:41,602 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:41,603 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:41,603 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:41,684 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:41,832 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:42,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:43,836 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:44,096 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:44,097 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:44,097 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:44,120 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:44,836 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:45,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:45,839 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:46,593 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:46,594 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:46,594 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:46,628 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:46,840 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:47,071 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:47,843 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:49,090 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:49,091 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:49,091 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:49,121 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:49,844 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:49,844 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:51,124 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:51,191 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:30:51,588 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:51,589 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:51,590 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:51,642 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:51,845 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:52,072 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:53,849 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:55,891 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:56,254 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:57,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:57,895 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:01,310 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:01,913 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:02,208 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:04,119 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:06,117 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:06,339 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:07,212 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:08,208 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:10,212 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:11,425 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:12,214 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:12,216 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:14,513 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:16,426 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:16,617 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:17,217 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:18,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:20,512 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:21,210 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:31:21,814 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:22,218 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:22,516 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:24,576 DEBUG SenderThread:13248 [sender.py:send():336] send: exit +2023-03-09 23:31:24,577 INFO SenderThread:13248 [sender.py:send_exit():559] handling exit code: 255 +2023-03-09 23:31:24,577 INFO SenderThread:13248 [sender.py:send_exit():561] handling runtime: 243 +2023-03-09 23:31:24,608 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:24,708 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:31:24,913 INFO SenderThread:13248 [sender.py:send_exit():567] send defer +2023-03-09 23:31:25,019 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,108 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:31:25,118 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,118 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:31:25,118 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:31:25,119 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,119 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:31:25,119 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,119 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:31:25,119 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:31:25,120 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,120 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:31:25,120 INFO HandlerThread:13248 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:31:25,121 DEBUG SystemMonitor:13248 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:31:25,121 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:31:25,121 DEBUG SystemMonitor:13248 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:31:25,124 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:31:25,188 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,188 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:31:25,188 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:31:25,188 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:31:25,189 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,189 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:31:25,189 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,189 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:31:25,189 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:31:25,190 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,190 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:31:25,190 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,190 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:31:25,190 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:31:25,190 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,190 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:31:25,190 DEBUG SenderThread:13248 [sender.py:send():336] send: summary +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:31:25,221 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:31:25,221 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,222 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:31:25,222 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:31:25,222 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:25,222 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,222 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:31:25,222 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:31:25,577 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:25,582 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:31:25,930 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:31:25,930 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:25,931 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,931 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:31:25,931 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,932 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:31:25,932 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:31:25,932 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,932 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:31:25,933 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,933 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:31:25,933 INFO SenderThread:13248 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:31:26,578 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:26,584 INFO SenderThread:13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:26,584 INFO SenderThread:13248 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log output.log +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt requirements.txt +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml config.yaml +2023-03-09 23:31:26,743 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json wandb-summary.json +2023-03-09 23:31:26,800 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:31:26,806 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:31:26,806 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:26,831 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:26,836 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:31:26,853 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:26,858 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:31:26,899 INFO SenderThread:13248 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:31:27,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:27,640 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:27,835 INFO wandb-upload_1:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:28,157 INFO wandb-upload_3:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml +2023-03-09 23:31:28,167 INFO wandb-upload_2:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt +2023-03-09 23:31:28,261 INFO wandb-upload_4:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:31:28,461 INFO Thread-12 :13248 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:31:28,461 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:28,461 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:31:28,461 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:28,461 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:31:28,461 INFO SenderThread:13248 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:31:28,461 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:31:28,462 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:28,462 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:31:28,462 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:28,462 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:31:28,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:31,144 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:31:31,146 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:31,146 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:31,147 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:31:31,148 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:31,148 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:31:31,148 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:31:31,148 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:31,148 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send():336] send: final +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send():336] send: footer +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:31,149 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:31:31,150 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:31,150 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:31,150 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:31:31,151 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:31:31,151 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:31:31,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:31:31,868 INFO MainThread:13248 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:31:31,869 INFO MainThread:13248 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:31:31,869 INFO MainThread:13248 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:31:31,870 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:31:31,870 INFO HandlerThread:13248 [handler.py:finish():842] shutting down handler +2023-03-09 23:31:32,151 INFO WriterThread:13248 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb +2023-03-09 23:31:32,591 WARNING StreamThr :13248 [internal.py:is_dead():416] Internal process exiting, parent pid 13116 disappeared +2023-03-09 23:31:32,591 ERROR StreamThr :13248 [internal.py:wandb_internal():153] Internal process shutdown. +2023-03-09 23:31:32,868 INFO SenderThread:13248 [sender.py:finish():1504] shutting down sender +2023-03-09 23:31:32,868 INFO SenderThread:13248 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:31:32,868 INFO SenderThread:13248 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/debug.log new file mode 100644 index 0000000..79c403d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Configure stats pid to 13116 +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug.log +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug-internal.log +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():602] starting backend +2023-03-09 23:27:14,833 INFO MainThread:13116 [wandb_init.py:init():606] setting up manager +2023-03-09 23:27:14,835 INFO MainThread:13116 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:27:14,837 INFO MainThread:13116 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:27:14,841 INFO MainThread:13116 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:27:14,842 INFO MainThread:13116 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:27:16,098 INFO MainThread:13116 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:27:21,163 INFO MainThread:13116 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:27:21,164 INFO MainThread:13116 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:27:22,043 INFO MainThread:13116 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:27:22,043 INFO MainThread:13116 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/conda-environment.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml new file mode 100644 index 0000000..63a5b53 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml @@ -0,0 +1,48 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678374292.648935 + t: + 1: + - 1 + - 41 + - 55 + 2: + - 1 + - 41 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +amp: + desc: null + value: false +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true +val_percent: + desc: null + value: 0.1 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log new file mode 100644 index 0000000..fbeee45 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log @@ -0,0 +1,21 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 4580 + Validation size: 508 + Checkpoints: True + Device: cuda + Images scaling: 0.5 + Mixed Precision: False +Traceback (most recent call last): + File "train.py", line 220, in + amp=args.amp + File "train.py", line 78, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 222, in + except torch.cuda.OutOfMemoryError: +AttributeError: module 'torch.cuda' has no attribute 'OutOfMemoryError' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-metadata.json new file mode 100644 index 0000000..44a4095 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:04:58.965045", + "startedAt": "2023-03-09T15:04:52.615795", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.9841749999996, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.386, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.381, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.797, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.409, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.928, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.392, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.39, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.043, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.01, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.991, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.238, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.396, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.371, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.606, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.822, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.571, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.39, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.141, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2508.62, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.213, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.531, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.237, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.597, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.581, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.775, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.383, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.589, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.712, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.415, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.406, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.426, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.793, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.439, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.182, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.762, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.396, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.205, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.467, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.745, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.795, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.911, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.331, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.74, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.364, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.055, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.069, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.34, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.686, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.732, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.303, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.008, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.33, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.063, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.562, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.548, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.72, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.286, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.68, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.347, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.377, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.344, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.607, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.654, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.335, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.849, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.763, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.978, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.376, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.926, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.29, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.08, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.227, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.319, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.65425872802734 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json new file mode 100644 index 0000000..fe9e22c --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 0}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug-internal.log new file mode 100644 index 0000000..6e5d800 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug-internal.log @@ -0,0 +1,186 @@ +2023-03-09 23:04:52,653 INFO StreamThr :5577 [internal.py:wandb_internal():90] W&B internal server running at pid: 5577, started at: 2023-03-09 23:04:52.651614 +2023-03-09 23:04:52,655 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:04:52,656 INFO WriterThread:5577 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/run-uhtrz7dz.wandb +2023-03-09 23:04:52,661 DEBUG SenderThread:5577 [sender.py:send():336] send: header +2023-03-09 23:04:52,661 DEBUG SenderThread:5577 [sender.py:send():336] send: run +2023-03-09 23:04:52,700 INFO SenderThread:5577 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/uhtrz7dz +2023-03-09 23:04:53,843 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:04:53,843 INFO SenderThread:5577 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files +2023-03-09 23:04:53,844 INFO SenderThread:5577 [sender.py:_start_run_threads():1081] run started: uhtrz7dz with start time 1678374292.648935 +2023-03-09 23:04:53,844 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:04:53,872 INFO SenderThread:5577 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:04:53,873 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:04:54,847 INFO Thread-13 :5577 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json +2023-03-09 23:04:58,844 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:04:58,874 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:04:58,924 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:04:58,929 DEBUG HandlerThread:5577 [system_info.py:__init__():31] System info init +2023-03-09 23:04:58,929 DEBUG HandlerThread:5577 [system_info.py:__init__():46] System info init done +2023-03-09 23:04:58,929 INFO HandlerThread:5577 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:04:58,930 INFO SystemMonitor:5577 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:04:58,930 INFO HandlerThread:5577 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:04:58,930 INFO SystemMonitor:5577 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:04:58,931 INFO SystemMonitor:5577 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:04:58,932 INFO SystemMonitor:5577 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:04:58,932 INFO SystemMonitor:5577 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:04:58,933 INFO SystemMonitor:5577 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:04:58,964 DEBUG HandlerThread:5577 [system_info.py:probe():195] Probing system +2023-03-09 23:04:58,983 DEBUG HandlerThread:5577 [git.py:repo():40] git repository is invalid +2023-03-09 23:04:58,983 DEBUG HandlerThread:5577 [system_info.py:probe():240] Probing system done +2023-03-09 23:04:58,983 DEBUG HandlerThread:5577 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:04:58.965045', 'startedAt': '2023-03-09T15:04:52.615795', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.9841749999996, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.386, 'min': 0.0, 'max': 0.0}, {'current': 2500.381, 'min': 0.0, 'max': 0.0}, {'current': 2500.797, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.409, 'min': 0.0, 'max': 0.0}, {'current': 2499.928, 'min': 0.0, 'max': 0.0}, {'current': 2500.392, 'min': 0.0, 'max': 0.0}, {'current': 2500.39, 'min': 0.0, 'max': 0.0}, {'current': 2500.043, 'min': 0.0, 'max': 0.0}, {'current': 2501.01, 'min': 0.0, 'max': 0.0}, {'current': 2500.991, 'min': 0.0, 'max': 0.0}, {'current': 2504.238, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2498.371, 'min': 0.0, 'max': 0.0}, {'current': 2499.606, 'min': 0.0, 'max': 0.0}, {'current': 2498.822, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.571, 'min': 0.0, 'max': 0.0}, {'current': 2501.39, 'min': 0.0, 'max': 0.0}, {'current': 2501.141, 'min': 0.0, 'max': 0.0}, {'current': 2508.62, 'min': 0.0, 'max': 0.0}, {'current': 2499.213, 'min': 0.0, 'max': 0.0}, {'current': 2503.531, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.237, 'min': 0.0, 'max': 0.0}, {'current': 2499.597, 'min': 0.0, 'max': 0.0}, {'current': 2499.581, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.775, 'min': 0.0, 'max': 0.0}, {'current': 2504.383, 'min': 0.0, 'max': 0.0}, {'current': 2500.589, 'min': 0.0, 'max': 0.0}, {'current': 2499.712, 'min': 0.0, 'max': 0.0}, {'current': 2500.415, 'min': 0.0, 'max': 0.0}, {'current': 2500.406, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.426, 'min': 0.0, 'max': 0.0}, {'current': 2498.793, 'min': 0.0, 'max': 0.0}, {'current': 2500.439, 'min': 0.0, 'max': 0.0}, {'current': 2499.182, 'min': 0.0, 'max': 0.0}, {'current': 2500.762, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2506.205, 'min': 0.0, 'max': 0.0}, {'current': 2505.467, 'min': 0.0, 'max': 0.0}, {'current': 2504.745, 'min': 0.0, 'max': 0.0}, {'current': 2500.795, 'min': 0.0, 'max': 0.0}, {'current': 2503.911, 'min': 0.0, 'max': 0.0}, {'current': 2504.331, 'min': 0.0, 'max': 0.0}, {'current': 2500.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.364, 'min': 0.0, 'max': 0.0}, {'current': 2500.055, 'min': 0.0, 'max': 0.0}, {'current': 2500.069, 'min': 0.0, 'max': 0.0}, {'current': 2506.34, 'min': 0.0, 'max': 0.0}, {'current': 2504.686, 'min': 0.0, 'max': 0.0}, {'current': 2503.732, 'min': 0.0, 'max': 0.0}, {'current': 2505.303, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2506.33, 'min': 0.0, 'max': 0.0}, {'current': 2504.063, 'min': 0.0, 'max': 0.0}, {'current': 2504.562, 'min': 0.0, 'max': 0.0}, {'current': 2500.548, 'min': 0.0, 'max': 0.0}, {'current': 2500.72, 'min': 0.0, 'max': 0.0}, {'current': 2504.286, 'min': 0.0, 'max': 0.0}, {'current': 2501.68, 'min': 0.0, 'max': 0.0}, {'current': 2500.347, 'min': 0.0, 'max': 0.0}, {'current': 2506.377, 'min': 0.0, 'max': 0.0}, {'current': 2503.344, 'min': 0.0, 'max': 0.0}, {'current': 2503.607, 'min': 0.0, 'max': 0.0}, {'current': 2503.654, 'min': 0.0, 'max': 0.0}, {'current': 2506.335, 'min': 0.0, 'max': 0.0}, {'current': 2501.849, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.763, 'min': 0.0, 'max': 0.0}, {'current': 2504.978, 'min': 0.0, 'max': 0.0}, {'current': 2505.376, 'min': 0.0, 'max': 0.0}, {'current': 2502.926, 'min': 0.0, 'max': 0.0}, {'current': 2506.29, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2507.08, 'min': 0.0, 'max': 0.0}, {'current': 2507.227, 'min': 0.0, 'max': 0.0}, {'current': 2502.319, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.65425872802734}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:04:58,983 INFO HandlerThread:5577 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:04:58,983 INFO HandlerThread:5577 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:04:58,983 DEBUG HandlerThread:5577 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:04:59,026 DEBUG HandlerThread:5577 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:04:59,026 DEBUG HandlerThread:5577 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:04:59,739 DEBUG HandlerThread:5577 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:04:59,777 INFO HandlerThread:5577 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:04:59,780 DEBUG SenderThread:5577 [sender.py:send():336] send: files +2023-03-09 23:04:59,780 INFO SenderThread:5577 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:04:59,794 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:04:59,800 DEBUG SenderThread:5577 [sender.py:send():336] send: telemetry +2023-03-09 23:04:59,810 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:05:00,021 INFO Thread-13 :5577 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-metadata.json +2023-03-09 23:05:00,021 INFO Thread-13 :5577 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/conda-environment.yaml +2023-03-09 23:05:00,022 INFO Thread-13 :5577 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt +2023-03-09 23:05:00,411 DEBUG SenderThread:5577 [sender.py:send():336] send: config +2023-03-09 23:05:00,440 DEBUG SenderThread:5577 [sender.py:send():336] send: exit +2023-03-09 23:05:00,441 INFO SenderThread:5577 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:05:00,441 INFO SenderThread:5577 [sender.py:send_exit():561] handling runtime: 0 +2023-03-09 23:05:00,492 INFO SenderThread:5577 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:05:00,502 INFO SenderThread:5577 [sender.py:send_exit():567] send defer +2023-03-09 23:05:00,518 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,523 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:05:00,544 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,549 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:05:00,549 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:05:00,554 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,554 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:05:00,555 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,555 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:05:00,555 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:05:00,555 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,555 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:05:00,555 INFO HandlerThread:5577 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:05:00,556 DEBUG SystemMonitor:5577 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:05:00,556 DEBUG SystemMonitor:5577 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:05:00,557 DEBUG SystemMonitor:5577 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:05:00,558 INFO HandlerThread:5577 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:05:00,558 INFO HandlerThread:5577 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:05:00,578 INFO HandlerThread:5577 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:05:00,578 INFO HandlerThread:5577 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:05:00,579 INFO HandlerThread:5577 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:05:00,579 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,579 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:05:00,579 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:05:00,579 DEBUG SenderThread:5577 [sender.py:send():336] send: stats +2023-03-09 23:05:00,580 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,580 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:05:00,580 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,580 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:05:00,580 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:05:00,580 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,580 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:05:00,581 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,581 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:05:00,581 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:05:00,581 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,581 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:05:00,581 DEBUG SenderThread:5577 [sender.py:send():336] send: summary +2023-03-09 23:05:00,609 INFO SenderThread:5577 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:05:00,610 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,610 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:05:00,610 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:05:00,610 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:00,610 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:05:00,610 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:00,610 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:05:00,615 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:05:00,868 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:01,024 INFO Thread-13 :5577 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json +2023-03-09 23:05:01,024 INFO Thread-13 :5577 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log +2023-03-09 23:05:01,080 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:05:01,080 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:01,081 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:01,081 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:05:01,081 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:01,081 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:05:01,140 INFO wandb-upload_0:5577 [upload_job.py:push():138] Uploaded file /tmp/tmpailb5z3lwandb/w9csxry5-wandb-metadata.json +2023-03-09 23:05:01,869 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:02,025 INFO Thread-13 :5577 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml +2023-03-09 23:05:02,477 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:05:02,483 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:02,488 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:02,493 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:05:02,514 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:02,519 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:05:02,550 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:05:02,560 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:02,560 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:05:02,560 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:02,560 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:05:02,560 INFO SenderThread:5577 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:05:02,915 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:03,027 INFO SenderThread:5577 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log +2023-03-09 23:05:03,027 INFO SenderThread:5577 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files +2023-03-09 23:05:03,028 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:05:03,028 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log output.log +2023-03-09 23:05:03,028 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt requirements.txt +2023-03-09 23:05:03,033 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml config.yaml +2023-03-09 23:05:03,049 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json wandb-summary.json +2023-03-09 23:05:03,080 INFO SenderThread:5577 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:05:03,080 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:05:03,080 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:03,095 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:03,095 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:05:03,151 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:03,156 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:05:03,156 INFO SenderThread:5577 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:05:03,971 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:03,992 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:04,383 INFO wandb-upload_4:5577 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/wandb-summary.json +2023-03-09 23:05:04,540 INFO wandb-upload_3:5577 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/config.yaml +2023-03-09 23:05:04,718 INFO wandb-upload_2:5577 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/requirements.txt +2023-03-09 23:05:04,823 INFO wandb-upload_1:5577 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/files/output.log +2023-03-09 23:05:04,976 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:04,977 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:05,024 INFO Thread-12 :5577 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:05:05,024 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:05,024 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:05:05,024 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:05,024 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:05:05,024 INFO SenderThread:5577 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:05:05,025 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:05:05,025 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:05,025 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:05:05,025 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:05,025 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:05:05,900 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:05:05,900 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:05,900 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:05:05,901 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:05,901 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:05:05,901 INFO SenderThread:5577 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:05:05,901 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:05:05,901 INFO HandlerThread:5577 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:05:05,901 DEBUG SenderThread:5577 [sender.py:send():336] send: final +2023-03-09 23:05:05,901 DEBUG SenderThread:5577 [sender.py:send():336] send: footer +2023-03-09 23:05:05,901 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: defer +2023-03-09 23:05:05,901 INFO SenderThread:5577 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:05:05,902 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:05,902 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:05,903 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:05:05,903 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:05:05,903 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:05:05,904 DEBUG SenderThread:5577 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:05:05,914 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:05:05,944 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:05:06,633 INFO MainThread:5577 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:05:06,634 INFO MainThread:5577 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:05:06,634 INFO MainThread:5577 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:05:06,634 DEBUG HandlerThread:5577 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:05:06,634 INFO HandlerThread:5577 [handler.py:finish():842] shutting down handler +2023-03-09 23:05:06,904 INFO WriterThread:5577 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/run-uhtrz7dz.wandb +2023-03-09 23:05:07,633 INFO SenderThread:5577 [sender.py:finish():1504] shutting down sender +2023-03-09 23:05:07,633 INFO SenderThread:5577 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:05:07,634 INFO SenderThread:5577 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug.log new file mode 100644 index 0000000..0655bd1 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:04:52,642 INFO MainThread:5442 [wandb_setup.py:_flush():76] Configure stats pid to 5442 +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug.log +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/logs/debug-internal.log +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:init():602] starting backend +2023-03-09 23:04:52,643 INFO MainThread:5442 [wandb_init.py:init():606] setting up manager +2023-03-09 23:04:52,645 INFO MainThread:5442 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:04:52,647 INFO MainThread:5442 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:04:52,651 INFO MainThread:5442 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:04:52,652 INFO MainThread:5442 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:04:53,843 INFO MainThread:5442 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:04:58,874 INFO MainThread:5442 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:04:58,875 INFO MainThread:5442 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:04:59,787 INFO MainThread:5442 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:04:59,788 INFO MainThread:5442 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:04:59,788 INFO MainThread:5442 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:04:59,788 INFO MainThread:5442 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:04:59,789 INFO MainThread:5442 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:04:59,789 INFO MainThread:5442 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:05:07,647 WARNING MsgRouterThr:5442 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/run-uhtrz7dz.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_230452-uhtrz7dz/run-uhtrz7dz.wandb new file mode 100644 index 0000000000000000000000000000000000000000..2cd13dd918b703a6c0c319590039367d8a934635 GIT binary patch literal 5298 zcmcJTe{38_7012jAGu7N#IY0HYygd1Q9IRnwLfpqCtN}5Mg%ntsA;JR)oE^b?A!C* z?P+(2(T0lsYf1s)Yh(8Ky{%BDvrAS4* zJ>Sl5W@t!H!hcwM^LF06@8`XFJGPa5HZ}uJi{&@AppX8pFZbSLJ^lDDJJSfS7mnCUx#87NS z({A6QDw+Wo2ZqL^8Rsstp0!C{Tgsur3++g3=U(x!n!ThW>{2>Q`2XlF=PXvL{fXSwZrv0*3*Iuz|f zTfOE|C3Hvwe89~XiBh>v$J*WniTK)y}Wj4 z&%&eNYHR(A%+-Zo{R=_|Vh6nj#Z%ofQQE6h1q>u9)^)Xjy%zp#(|0=!dPkQU6zx0- zmO=$C`cQQKxr{3@zS7`-g z(MDHPBbdK#pBL~k>Mhlxic1}`I-mVfz)imine8;^lZygg^aDzq+I(EK`}?5s>D8$% zF_qB_l~d~{Kj-RD!PyOy20Oii3t8IE$5cokES1QmDqDp0EKVI@9 zOj@|pqQ~3RiD0)Z)74{s;9^K~Czgt*d-|sPkhXqnA&RB#$VhwYm#6!Yw(-`Zzx^h- z+F8G!N!$3_;~?!ITId9Er>ntJF75)@U&K+3Vp8>2D5l39+KxL<0hIbM9;8FqcbrZT zP`8ZwP~{`757_y6fFXMc0?fjIQ>Z0Ik2 zr=QR#elNfLU2ru~CoD04`S5!X`jHzn|Rzpc0YW6}SRO$KMuQ->yJf>S`^(Be$GVkV{{XUc?O3#MW+tHhZ!#ZF9f zoGErFqw$!A#^X>GmD&Cqk7OPXBicO%}SGG ziD207v_vp0P>TnX=}guQz*sz(j0j5v!wSvf!DNzTiC|b!SUi~QJPu5DvlK-xY_k+a zE-cQX$b}`aC~`@Cid<5kA{VBwC~`@C@?0{Tx+ro< + amp=args.amp + File "train.py", line 78, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 222, in + except torch.cuda.OutOfMemoryError: +AttributeError: module 'torch.cuda' has no attribute 'OutOfMemoryError' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-metadata.json new file mode 100644 index 0000000..f0cad8c --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:15:01.932555", + "startedAt": "2023-03-09T15:14:54.388532", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2500.5411125, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.977, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.964, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.144, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.943, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2509.623, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.121, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2509.386, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.765, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.086, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.835, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.806, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.516, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.331, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.883, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.017, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.324, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.608, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.986, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.842, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.711, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.775, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.382, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.222, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.546, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.571, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.623, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2495.377, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.119, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.975, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.225, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.004, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.984, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.989, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.96, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.922, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.944, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.86, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.142, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.045, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.044, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.777, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.421, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.171, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2497.283, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.039, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.073, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.018, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.363, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.769, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.011, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.005, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.891, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.942, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.6, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.943, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.098, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.805, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.697, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.982, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.851, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.11, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.041, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.067, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.021, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.806, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.049, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.944, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.016, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.954, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.65583419799805 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug-internal.log new file mode 100644 index 0000000..190412d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug-internal.log @@ -0,0 +1,182 @@ +2023-03-09 23:14:54,423 INFO StreamThr :11705 [internal.py:wandb_internal():90] W&B internal server running at pid: 11705, started at: 2023-03-09 23:14:54.420943 +2023-03-09 23:14:54,424 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:14:54,425 INFO WriterThread:11705 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/run-vws83bqw.wandb +2023-03-09 23:14:54,430 DEBUG SenderThread:11705 [sender.py:send():336] send: header +2023-03-09 23:14:54,430 DEBUG SenderThread:11705 [sender.py:send():336] send: run +2023-03-09 23:14:54,461 INFO SenderThread:11705 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/vws83bqw +2023-03-09 23:14:56,828 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:14:56,829 INFO SenderThread:11705 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files +2023-03-09 23:14:56,829 INFO SenderThread:11705 [sender.py:_start_run_threads():1081] run started: vws83bqw with start time 1678374894.421132 +2023-03-09 23:14:56,829 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:14:56,854 INFO SenderThread:11705 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:14:56,855 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:14:57,833 INFO Thread-13 :11705 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json +2023-03-09 23:15:01,829 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:15:01,856 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:01,904 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:15:01,909 DEBUG HandlerThread:11705 [system_info.py:__init__():31] System info init +2023-03-09 23:15:01,910 DEBUG HandlerThread:11705 [system_info.py:__init__():46] System info init done +2023-03-09 23:15:01,910 INFO HandlerThread:11705 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:15:01,910 INFO SystemMonitor:11705 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:15:01,910 INFO HandlerThread:11705 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:15:01,911 INFO SystemMonitor:11705 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:15:01,911 INFO SystemMonitor:11705 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:15:01,912 INFO SystemMonitor:11705 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:15:01,912 INFO SystemMonitor:11705 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:15:01,913 INFO SystemMonitor:11705 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:15:01,932 DEBUG HandlerThread:11705 [system_info.py:probe():195] Probing system +2023-03-09 23:15:01,958 DEBUG HandlerThread:11705 [git.py:repo():40] git repository is invalid +2023-03-09 23:15:01,958 DEBUG HandlerThread:11705 [system_info.py:probe():240] Probing system done +2023-03-09 23:15:01,958 DEBUG HandlerThread:11705 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:15:01.932555', 'startedAt': '2023-03-09T15:14:54.388532', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2500.5411125, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.977, 'min': 0.0, 'max': 0.0}, {'current': 2506.964, 'min': 0.0, 'max': 0.0}, {'current': 2504.144, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2509.623, 'min': 0.0, 'max': 0.0}, {'current': 2506.121, 'min': 0.0, 'max': 0.0}, {'current': 2509.386, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.765, 'min': 0.0, 'max': 0.0}, {'current': 2500.086, 'min': 0.0, 'max': 0.0}, {'current': 2499.835, 'min': 0.0, 'max': 0.0}, {'current': 2499.806, 'min': 0.0, 'max': 0.0}, {'current': 2499.516, 'min': 0.0, 'max': 0.0}, {'current': 2500.331, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.883, 'min': 0.0, 'max': 0.0}, {'current': 2500.017, 'min': 0.0, 'max': 0.0}, {'current': 2500.324, 'min': 0.0, 'max': 0.0}, {'current': 2499.608, 'min': 0.0, 'max': 0.0}, {'current': 2499.986, 'min': 0.0, 'max': 0.0}, {'current': 2499.842, 'min': 0.0, 'max': 0.0}, {'current': 2500.711, 'min': 0.0, 'max': 0.0}, {'current': 2498.775, 'min': 0.0, 'max': 0.0}, {'current': 2500.382, 'min': 0.0, 'max': 0.0}, {'current': 2500.222, 'min': 0.0, 'max': 0.0}, {'current': 2500.546, 'min': 0.0, 'max': 0.0}, {'current': 2498.571, 'min': 0.0, 'max': 0.0}, {'current': 2499.623, 'min': 0.0, 'max': 0.0}, {'current': 2495.377, 'min': 0.0, 'max': 0.0}, {'current': 2504.119, 'min': 0.0, 'max': 0.0}, {'current': 2499.975, 'min': 0.0, 'max': 0.0}, {'current': 2498.225, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2499.989, 'min': 0.0, 'max': 0.0}, {'current': 2499.96, 'min': 0.0, 'max': 0.0}, {'current': 2499.922, 'min': 0.0, 'max': 0.0}, {'current': 2499.944, 'min': 0.0, 'max': 0.0}, {'current': 2499.86, 'min': 0.0, 'max': 0.0}, {'current': 2500.142, 'min': 0.0, 'max': 0.0}, {'current': 2500.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.044, 'min': 0.0, 'max': 0.0}, {'current': 2498.777, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.421, 'min': 0.0, 'max': 0.0}, {'current': 2500.171, 'min': 0.0, 'max': 0.0}, {'current': 2497.283, 'min': 0.0, 'max': 0.0}, {'current': 2500.039, 'min': 0.0, 'max': 0.0}, {'current': 2500.073, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.018, 'min': 0.0, 'max': 0.0}, {'current': 2505.363, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.769, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.005, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.891, 'min': 0.0, 'max': 0.0}, {'current': 2499.942, 'min': 0.0, 'max': 0.0}, {'current': 2505.6, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2504.098, 'min': 0.0, 'max': 0.0}, {'current': 2500.805, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.697, 'min': 0.0, 'max': 0.0}, {'current': 2499.982, 'min': 0.0, 'max': 0.0}, {'current': 2499.851, 'min': 0.0, 'max': 0.0}, {'current': 2500.11, 'min': 0.0, 'max': 0.0}, {'current': 2500.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.067, 'min': 0.0, 'max': 0.0}, {'current': 2500.021, 'min': 0.0, 'max': 0.0}, {'current': 2499.806, 'min': 0.0, 'max': 0.0}, {'current': 2500.049, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.944, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2499.954, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.65583419799805}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:15:01,958 INFO HandlerThread:11705 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:15:01,958 INFO HandlerThread:11705 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:15:01,958 DEBUG HandlerThread:11705 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:15:01,982 DEBUG HandlerThread:11705 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:15:01,983 DEBUG HandlerThread:11705 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:15:02,734 DEBUG HandlerThread:11705 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:15:02,794 INFO HandlerThread:11705 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:15:02,797 DEBUG SenderThread:11705 [sender.py:send():336] send: files +2023-03-09 23:15:02,798 INFO SenderThread:11705 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:15:02,817 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:15:02,842 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:15:02,997 INFO Thread-13 :11705 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt +2023-03-09 23:15:03,022 INFO Thread-13 :11705 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/conda-environment.yaml +2023-03-09 23:15:03,032 INFO Thread-13 :11705 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-metadata.json +2023-03-09 23:15:03,289 DEBUG SenderThread:11705 [sender.py:send():336] send: telemetry +2023-03-09 23:15:03,289 DEBUG SenderThread:11705 [sender.py:send():336] send: config +2023-03-09 23:15:03,291 DEBUG SenderThread:11705 [sender.py:send():336] send: exit +2023-03-09 23:15:03,292 INFO SenderThread:11705 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:15:03,292 INFO SenderThread:11705 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:15:03,316 INFO SenderThread:11705 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:15:03,317 INFO SenderThread:11705 [sender.py:send_exit():567] send defer +2023-03-09 23:15:03,317 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,317 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:15:03,317 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,317 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:15:03,317 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:15:03,317 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,317 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:15:03,318 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,318 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:15:03,318 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:15:03,318 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,318 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:15:03,318 INFO HandlerThread:11705 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:15:03,318 DEBUG SystemMonitor:11705 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:15:03,319 DEBUG SystemMonitor:11705 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:15:03,319 DEBUG SystemMonitor:11705 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:15:03,320 INFO HandlerThread:11705 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:15:03,320 INFO HandlerThread:11705 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:15:03,349 INFO HandlerThread:11705 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:15:03,349 INFO HandlerThread:11705 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:15:03,350 INFO HandlerThread:11705 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:15:03,350 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,350 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:15:03,350 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:15:03,350 DEBUG SenderThread:11705 [sender.py:send():336] send: stats +2023-03-09 23:15:03,351 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,351 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:15:03,351 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,351 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:15:03,351 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:15:03,351 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,351 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:15:03,351 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,351 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:15:03,351 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:15:03,352 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,352 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:15:03,352 DEBUG SenderThread:11705 [sender.py:send():336] send: summary +2023-03-09 23:15:03,375 INFO SenderThread:11705 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:15:03,376 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,376 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:15:03,376 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:15:03,376 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:03,376 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:15:03,376 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:03,376 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:15:03,382 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:15:03,920 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:03,983 INFO Thread-13 :11705 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json +2023-03-09 23:15:04,197 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:15:04,197 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:04,197 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:04,198 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:15:04,198 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:04,198 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:15:04,921 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:04,984 INFO Thread-13 :11705 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/config.yaml +2023-03-09 23:15:05,057 INFO wandb-upload_0:11705 [upload_job.py:push():138] Uploaded file /tmp/tmp4crk263cwandb/ri9jkoot-wandb-metadata.json +2023-03-09 23:15:05,314 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:15:05,314 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:05,314 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:05,314 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:15:05,315 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:05,315 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:15:05,315 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:15:05,315 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:05,315 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:15:05,316 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:05,316 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:15:05,316 INFO SenderThread:11705 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:15:05,921 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:05,986 INFO SenderThread:11705 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files +2023-03-09 23:15:05,987 INFO SenderThread:11705 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:15:05,987 INFO SenderThread:11705 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt requirements.txt +2023-03-09 23:15:05,987 INFO SenderThread:11705 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/config.yaml config.yaml +2023-03-09 23:15:05,992 INFO SenderThread:11705 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json wandb-summary.json +2023-03-09 23:15:06,048 INFO SenderThread:11705 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:15:06,054 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:15:06,054 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:06,090 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:06,120 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:15:06,160 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:06,166 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:15:06,166 INFO SenderThread:11705 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:15:07,008 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:07,042 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:07,187 INFO wandb-upload_0:11705 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/requirements.txt +2023-03-09 23:15:07,665 INFO wandb-upload_2:11705 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/config.yaml +2023-03-09 23:15:07,740 INFO wandb-upload_3:11705 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/files/wandb-summary.json +2023-03-09 23:15:07,940 INFO Thread-12 :11705 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:15:07,941 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:07,941 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:15:07,941 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:07,941 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:15:07,941 INFO SenderThread:11705 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:15:07,941 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:15:07,941 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:07,941 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:15:07,942 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:07,942 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:15:08,035 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:08,272 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:15:08,273 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:08,273 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:08,273 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:15:08,273 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:08,273 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:15:08,274 INFO SenderThread:11705 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:15:08,274 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:15:08,274 INFO HandlerThread:11705 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:15:08,274 DEBUG SenderThread:11705 [sender.py:send():336] send: final +2023-03-09 23:15:08,274 DEBUG SenderThread:11705 [sender.py:send():336] send: footer +2023-03-09 23:15:08,274 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: defer +2023-03-09 23:15:08,274 INFO SenderThread:11705 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:15:08,275 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:08,275 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:08,276 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:15:08,276 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:15:08,276 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:15:08,277 DEBUG SenderThread:11705 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:15:08,297 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:15:08,353 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:15:09,077 INFO MainThread:11705 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:15:09,078 INFO MainThread:11705 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:15:09,078 INFO MainThread:11705 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:15:09,078 DEBUG HandlerThread:11705 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:15:09,078 INFO HandlerThread:11705 [handler.py:finish():842] shutting down handler +2023-03-09 23:15:09,277 INFO WriterThread:11705 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/run-vws83bqw.wandb +2023-03-09 23:15:10,077 INFO SenderThread:11705 [sender.py:finish():1504] shutting down sender +2023-03-09 23:15:10,077 INFO SenderThread:11705 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:15:10,077 INFO SenderThread:11705 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug.log new file mode 100644 index 0000000..18aeda9 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:14:54,414 INFO MainThread:11497 [wandb_setup.py:_flush():76] Configure stats pid to 11497 +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug.log +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/logs/debug-internal.log +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:init():602] starting backend +2023-03-09 23:14:54,415 INFO MainThread:11497 [wandb_init.py:init():606] setting up manager +2023-03-09 23:14:54,417 INFO MainThread:11497 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:14:54,420 INFO MainThread:11497 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:14:54,424 INFO MainThread:11497 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:14:54,425 INFO MainThread:11497 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:14:56,828 INFO MainThread:11497 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:15:01,856 INFO MainThread:11497 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:15:01,857 INFO MainThread:11497 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:15:02,806 INFO MainThread:11497 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:15:02,807 INFO MainThread:11497 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:15:02,807 INFO MainThread:11497 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:15:02,807 INFO MainThread:11497 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:15:02,807 INFO MainThread:11497 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:15:02,808 INFO MainThread:11497 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:15:10,097 WARNING MsgRouterThr:11497 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/run-vws83bqw.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_231454-vws83bqw/run-vws83bqw.wandb new file mode 100644 index 0000000000000000000000000000000000000000..053076f85f67a58c0fa89826a997d218966beddb GIT binary patch literal 5304 zcmcJTZHya78OLwV&c!#mH20G9S|>%LtLR-K-)i62wzwj$$*HZHhNz*SN?@(m#P)^gSeN}&##OOv8$nT}I(RL!vUT+zx} zYSvJ+Y{^imYRmdXH1g{Sj8p3yepV}fdSm_C<>f0ddbES&|2=wcqF(VTnzgx>HCnX` z=rd>#DZS@oi;vP^?f~A8#|j&EvsCP$(WjoSEmVr-Qr1+98kG%Q$z^k;l94Mq@T=>N zNsEht?XSs;@&z=JdguBNULKp=`OBNz=p}R?PNmTHD7utD>BJ?3=EgM1k`(3GURhE! zxL8h3q6M_e&?nv3Q-{G&LY>8j@Ju(4#Rd)Rk{#Gm&3BvaYY$#7!xPCT7+o2v!S z4&de208+v~L8RH(Doa;r>8240)uDLh_ntQLI7 zu0srX*npZO>_!U$mhHL^H*cd=cKJB}r*C|!iBM{+_OsFHiPS&4WjgzvH_j!}-yePG z(Untq@>pPd0i;d>&#p2Lc})60C%Ufl;YUIYE-f3Y=((_uHAcFE@ILk)YC`h>w{ z9tmN1!=Wduo3une>QSfa!_;}Q!fU|(SKkKa_kwwc7HwyZ zEHqp{ATT3z27#pNq;C7c(fknebt=}o$6>xy?G8TQClTiUhG zGh~BSS1UoGL>+r;Df%Hp8g2tB+iom{7a3A^J!(6ZCDye^hoJJ-jpd&+DiaYZZ(V$M z7%DqgKAia?gxX(wj6>PEaki64KX7KNMNfF1>*YzI02@M3C@dT$t8PH-CTTb6W{Wxj ztd=#pwc&at0%dM9t7N|KZ+-}C`Un2~TW0N!h_#(}&Uc1r;q!O?*W=;NxwN?Q|7*%duU6xMdPlD6xy~Iwf?|p)I!aWWcEl_r1X_;yO;-^JpoLhj4dN z-TX6iHyLqv^-C9qarfrpYbM)UrXqXG7hZY>-2H#{mi4JWyuwnSj--C++;hXEt~IY* z{~CmvsoA_R+pj$jsh@tYg{Z1(d~fO1i3G0aR3x^?OQ+hwsq#tMaJ{Ww=MGV$dq?b> z%;>?0(d}P+br_?^P8`&qg;0lTpWsHX{_#aH`rt=`z^fM9LAb(tdnTD5n3;#}^hwht zcK9fn7yS*h!fw9({BM}QJ0t#fUis!Q{>HD@mybZG*$$eVoA5WXTVBK8GRn%63+O8- znL?B3RA%Yn?~S9>UHNa%0D~LdbH@)~|Iyb``h$43<++a3=+;ZJ*X?TEF4pPE>X z{c26}!=soSy$dhm_3oKVPVdLbQq^CBc{N><)BAA3X|?;>-FONIRN#CW4l{Ra2{?C$k|jt$D3r2-q)0|6 z3;7(Iw7LXn}q;e2r_ zI@jV@EUv}5sOiH!E~>zV6^JLJ#=)3jP*nt@c8lIMsNMpM8VAFRq>5n7NG@?OgFNG4 zSOfNQ5yAADv!{t*lmTo_0F#JnA{TX#rzUcV@YF;us>FH1Lf+k=@P#z=pov_f^NC!d zd5T=vTIqAq;#@dSP2>V!M8a{p$R)y4k8{a|jiw7+l1gsSFuDjPvT}71Or+8DI2gWK zbP-H6<}JYJaWK3=brFm**hP#u7(UJr!6*`6xrP7+8f~D(x$r_5B9}-Z43P`d`YYZL zxFlf<;FUK-FjmB19L!)`91QPlQv?$!pcx0l4VogDh(YrfU`!E=I#@}j2*!wL0+>WJ zPmzl + amp=args.amp + File "train.py", line 78, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 224, in + except torch.cuda.OutOfMemoryError: +AttributeError: module 'torch.cuda' has no attribute 'OutOfMemoryError' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-metadata.json new file mode 100644 index 0000000..dc89ced --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:21:17.729141", + "startedAt": "2023-03-09T15:21:11.158089", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.6207249999998, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.976, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.206, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.529, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.421, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.126, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.328, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.693, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.843, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.35, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.334, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.469, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.116, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.291, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.491, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.939, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.369, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.908, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.074, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.588, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.331, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.963, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.983, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.939, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.628, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.583, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.654, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.048, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.086, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.641, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.321, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.3, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2508.13, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.334, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.2, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.688, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.322, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.21, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.331, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.502, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.198, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.603, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.823, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.048, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.836, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.464, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.701, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.186, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.64, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.325, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.139, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.294, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.021, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.174, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.332, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.743, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.24, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.716, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.078, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.095, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.456, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.758, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.737, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.085, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.041, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.756, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.623, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.891, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.698, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.929, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.011, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.747, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.64722442626953 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json new file mode 100644 index 0000000..fe9e22c --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 0}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug-internal.log new file mode 100644 index 0000000..fc81416 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug-internal.log @@ -0,0 +1,182 @@ +2023-03-09 23:21:11,192 INFO StreamThr :12461 [internal.py:wandb_internal():90] W&B internal server running at pid: 12461, started at: 2023-03-09 23:21:11.190160 +2023-03-09 23:21:11,193 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:21:11,194 INFO WriterThread:12461 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/run-ehuqx5gi.wandb +2023-03-09 23:21:11,199 DEBUG SenderThread:12461 [sender.py:send():336] send: header +2023-03-09 23:21:11,199 DEBUG SenderThread:12461 [sender.py:send():336] send: run +2023-03-09 23:21:11,248 INFO SenderThread:12461 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/ehuqx5gi +2023-03-09 23:21:12,603 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:21:12,604 INFO SenderThread:12461 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files +2023-03-09 23:21:12,605 INFO SenderThread:12461 [sender.py:_start_run_threads():1081] run started: ehuqx5gi with start time 1678375271.188879 +2023-03-09 23:21:12,605 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:21:12,629 INFO SenderThread:12461 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:21:12,630 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:21:13,608 INFO Thread-13 :12461 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json +2023-03-09 23:21:17,604 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:21:17,631 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:21:17,684 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:21:17,689 DEBUG HandlerThread:12461 [system_info.py:__init__():31] System info init +2023-03-09 23:21:17,689 DEBUG HandlerThread:12461 [system_info.py:__init__():46] System info init done +2023-03-09 23:21:17,690 INFO HandlerThread:12461 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:21:17,690 INFO SystemMonitor:12461 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:21:17,690 INFO HandlerThread:12461 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:21:17,691 INFO SystemMonitor:12461 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:21:17,691 INFO SystemMonitor:12461 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:21:17,692 INFO SystemMonitor:12461 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:21:17,692 INFO SystemMonitor:12461 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:21:17,693 INFO SystemMonitor:12461 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:21:17,729 DEBUG HandlerThread:12461 [system_info.py:probe():195] Probing system +2023-03-09 23:21:17,747 DEBUG HandlerThread:12461 [git.py:repo():40] git repository is invalid +2023-03-09 23:21:17,747 DEBUG HandlerThread:12461 [system_info.py:probe():240] Probing system done +2023-03-09 23:21:17,747 DEBUG HandlerThread:12461 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:21:17.729141', 'startedAt': '2023-03-09T15:21:11.158089', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.6207249999998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.976, 'min': 0.0, 'max': 0.0}, {'current': 2498.206, 'min': 0.0, 'max': 0.0}, {'current': 2501.529, 'min': 0.0, 'max': 0.0}, {'current': 2500.421, 'min': 0.0, 'max': 0.0}, {'current': 2501.126, 'min': 0.0, 'max': 0.0}, {'current': 2499.328, 'min': 0.0, 'max': 0.0}, {'current': 2500.693, 'min': 0.0, 'max': 0.0}, {'current': 2498.843, 'min': 0.0, 'max': 0.0}, {'current': 2500.35, 'min': 0.0, 'max': 0.0}, {'current': 2500.334, 'min': 0.0, 'max': 0.0}, {'current': 2501.469, 'min': 0.0, 'max': 0.0}, {'current': 2501.116, 'min': 0.0, 'max': 0.0}, {'current': 2500.291, 'min': 0.0, 'max': 0.0}, {'current': 2498.491, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2500.369, 'min': 0.0, 'max': 0.0}, {'current': 2499.908, 'min': 0.0, 'max': 0.0}, {'current': 2500.074, 'min': 0.0, 'max': 0.0}, {'current': 2500.588, 'min': 0.0, 'max': 0.0}, {'current': 2500.331, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.963, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2499.628, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.583, 'min': 0.0, 'max': 0.0}, {'current': 2499.654, 'min': 0.0, 'max': 0.0}, {'current': 2501.048, 'min': 0.0, 'max': 0.0}, {'current': 2501.086, 'min': 0.0, 'max': 0.0}, {'current': 2498.641, 'min': 0.0, 'max': 0.0}, {'current': 2499.321, 'min': 0.0, 'max': 0.0}, {'current': 2499.3, 'min': 0.0, 'max': 0.0}, {'current': 2508.13, 'min': 0.0, 'max': 0.0}, {'current': 2499.334, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.2, 'min': 0.0, 'max': 0.0}, {'current': 2500.688, 'min': 0.0, 'max': 0.0}, {'current': 2499.322, 'min': 0.0, 'max': 0.0}, {'current': 2504.21, 'min': 0.0, 'max': 0.0}, {'current': 2505.331, 'min': 0.0, 'max': 0.0}, {'current': 2504.502, 'min': 0.0, 'max': 0.0}, {'current': 2505.198, 'min': 0.0, 'max': 0.0}, {'current': 2505.603, 'min': 0.0, 'max': 0.0}, {'current': 2503.823, 'min': 0.0, 'max': 0.0}, {'current': 2503.048, 'min': 0.0, 'max': 0.0}, {'current': 2500.836, 'min': 0.0, 'max': 0.0}, {'current': 2504.464, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.701, 'min': 0.0, 'max': 0.0}, {'current': 2504.186, 'min': 0.0, 'max': 0.0}, {'current': 2502.64, 'min': 0.0, 'max': 0.0}, {'current': 2506.325, 'min': 0.0, 'max': 0.0}, {'current': 2499.139, 'min': 0.0, 'max': 0.0}, {'current': 2500.294, 'min': 0.0, 'max': 0.0}, {'current': 2500.021, 'min': 0.0, 'max': 0.0}, {'current': 2504.174, 'min': 0.0, 'max': 0.0}, {'current': 2503.332, 'min': 0.0, 'max': 0.0}, {'current': 2501.743, 'min': 0.0, 'max': 0.0}, {'current': 2504.24, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.716, 'min': 0.0, 'max': 0.0}, {'current': 2500.078, 'min': 0.0, 'max': 0.0}, {'current': 2504.095, 'min': 0.0, 'max': 0.0}, {'current': 2505.456, 'min': 0.0, 'max': 0.0}, {'current': 2499.758, 'min': 0.0, 'max': 0.0}, {'current': 2505.737, 'min': 0.0, 'max': 0.0}, {'current': 2501.085, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.027, 'min': 0.0, 'max': 0.0}, {'current': 2502.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.756, 'min': 0.0, 'max': 0.0}, {'current': 2503.623, 'min': 0.0, 'max': 0.0}, {'current': 2503.891, 'min': 0.0, 'max': 0.0}, {'current': 2503.698, 'min': 0.0, 'max': 0.0}, {'current': 2502.929, 'min': 0.0, 'max': 0.0}, {'current': 2504.011, 'min': 0.0, 'max': 0.0}, {'current': 2503.747, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64722442626953}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:21:17,747 INFO HandlerThread:12461 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:21:17,747 INFO HandlerThread:12461 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:21:17,747 DEBUG HandlerThread:12461 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:21:17,761 DEBUG HandlerThread:12461 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:21:17,761 DEBUG HandlerThread:12461 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:21:18,482 DEBUG HandlerThread:12461 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:21:18,513 INFO HandlerThread:12461 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:21:18,516 DEBUG SenderThread:12461 [sender.py:send():336] send: files +2023-03-09 23:21:18,516 INFO SenderThread:12461 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:21:18,531 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:21:18,537 DEBUG SenderThread:12461 [sender.py:send():336] send: telemetry +2023-03-09 23:21:18,557 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:21:18,753 INFO Thread-13 :12461 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/conda-environment.yaml +2023-03-09 23:21:18,754 INFO Thread-13 :12461 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-metadata.json +2023-03-09 23:21:18,754 INFO Thread-13 :12461 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt +2023-03-09 23:21:19,002 DEBUG SenderThread:12461 [sender.py:send():336] send: config +2023-03-09 23:21:19,005 DEBUG SenderThread:12461 [sender.py:send():336] send: exit +2023-03-09 23:21:19,005 INFO SenderThread:12461 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:21:19,005 INFO SenderThread:12461 [sender.py:send_exit():561] handling runtime: 0 +2023-03-09 23:21:19,068 INFO SenderThread:12461 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:21:19,080 INFO SenderThread:12461 [sender.py:send_exit():567] send defer +2023-03-09 23:21:19,095 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,095 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:21:19,112 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,112 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:21:19,112 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:21:19,121 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,121 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:21:19,122 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,122 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:21:19,122 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:21:19,128 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,128 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:21:19,128 INFO HandlerThread:12461 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:21:19,189 DEBUG SystemMonitor:12461 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:21:19,237 DEBUG SystemMonitor:12461 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:21:19,238 DEBUG SystemMonitor:12461 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:21:19,239 INFO HandlerThread:12461 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:21:19,239 INFO HandlerThread:12461 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:21:19,273 INFO HandlerThread:12461 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:21:19,273 INFO HandlerThread:12461 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:21:19,273 INFO HandlerThread:12461 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:21:19,274 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,274 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:21:19,274 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:21:19,274 DEBUG SenderThread:12461 [sender.py:send():336] send: stats +2023-03-09 23:21:19,274 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,274 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:21:19,274 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,274 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:21:19,274 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:21:19,274 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,274 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:21:19,275 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,275 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:21:19,275 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:21:19,275 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,275 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:21:19,275 DEBUG SenderThread:12461 [sender.py:send():336] send: summary +2023-03-09 23:21:19,303 INFO SenderThread:12461 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:21:19,303 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,304 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:21:19,304 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:21:19,304 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,304 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:21:19,304 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,304 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:21:19,309 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:21:19,609 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:19,754 INFO Thread-13 :12461 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json +2023-03-09 23:21:19,901 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:21:19,901 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:19,901 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:19,901 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:21:19,902 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:19,902 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:21:19,997 INFO wandb-upload_0:12461 [upload_job.py:push():138] Uploaded file /tmp/tmpqs986ttwwandb/ttducsn2-wandb-metadata.json +2023-03-09 23:21:20,610 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:20,755 INFO Thread-13 :12461 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/config.yaml +2023-03-09 23:21:21,037 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:21:21,043 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:21,043 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:21,048 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:21:21,079 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:21,079 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:21:21,105 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:21:21,120 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:21,124 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:21:21,124 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:21,124 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:21:21,124 INFO SenderThread:12461 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:21:21,610 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:21,757 INFO SenderThread:12461 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files +2023-03-09 23:21:21,758 INFO SenderThread:12461 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:21:21,758 INFO SenderThread:12461 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt requirements.txt +2023-03-09 23:21:21,758 INFO SenderThread:12461 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/config.yaml config.yaml +2023-03-09 23:21:21,769 INFO SenderThread:12461 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json wandb-summary.json +2023-03-09 23:21:21,809 INFO SenderThread:12461 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:21:21,814 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:21:21,815 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:21,830 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:21,830 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:21:21,860 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:21,860 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:21:21,860 INFO SenderThread:12461 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:21:22,673 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:22,674 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:22,915 INFO wandb-upload_0:12461 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/requirements.txt +2023-03-09 23:21:23,243 INFO wandb-upload_3:12461 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/wandb-summary.json +2023-03-09 23:21:23,303 INFO wandb-upload_2:12461 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/files/config.yaml +2023-03-09 23:21:23,503 INFO Thread-12 :12461 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:21:23,503 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:23,504 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:21:23,504 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:23,504 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:21:23,504 INFO SenderThread:12461 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:21:23,504 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:21:23,504 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:23,504 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:21:23,504 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:23,504 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:21:23,674 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:23,835 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:21:23,835 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:23,835 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:23,836 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:21:23,836 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:23,836 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:21:23,836 INFO SenderThread:12461 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:21:23,836 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:21:23,836 INFO HandlerThread:12461 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:21:23,837 DEBUG SenderThread:12461 [sender.py:send():336] send: final +2023-03-09 23:21:23,837 DEBUG SenderThread:12461 [sender.py:send():336] send: footer +2023-03-09 23:21:23,837 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: defer +2023-03-09 23:21:23,837 INFO SenderThread:12461 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:21:23,837 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:23,838 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:23,838 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:21:23,838 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:21:23,838 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:21:23,839 DEBUG SenderThread:12461 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:21:23,869 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:21:23,905 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:21:24,593 INFO MainThread:12461 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:21:24,594 INFO MainThread:12461 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:21:24,594 INFO MainThread:12461 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:21:24,594 DEBUG HandlerThread:12461 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:21:24,594 INFO HandlerThread:12461 [handler.py:finish():842] shutting down handler +2023-03-09 23:21:24,839 INFO WriterThread:12461 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/run-ehuqx5gi.wandb +2023-03-09 23:21:25,593 INFO SenderThread:12461 [sender.py:finish():1504] shutting down sender +2023-03-09 23:21:25,593 INFO SenderThread:12461 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:21:25,593 INFO SenderThread:12461 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug.log new file mode 100644 index 0000000..14fb2a8 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Configure stats pid to 12325 +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug.log +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/logs/debug-internal.log +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:init():602] starting backend +2023-03-09 23:21:11,184 INFO MainThread:12325 [wandb_init.py:init():606] setting up manager +2023-03-09 23:21:11,187 INFO MainThread:12325 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:21:11,188 INFO MainThread:12325 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:21:11,190 INFO MainThread:12325 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:21:11,191 INFO MainThread:12325 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:21:12,603 INFO MainThread:12325 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:21:17,631 INFO MainThread:12325 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:21:17,631 INFO MainThread:12325 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:21:18,523 INFO MainThread:12325 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:21:18,524 INFO MainThread:12325 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:21:18,524 INFO MainThread:12325 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:21:18,524 INFO MainThread:12325 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:21:18,525 INFO MainThread:12325 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:21:18,525 INFO MainThread:12325 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:21:25,607 WARNING MsgRouterThr:12325 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/run-ehuqx5gi.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232111-ehuqx5gi/run-ehuqx5gi.wandb new file mode 100644 index 0000000000000000000000000000000000000000..48932072f08ad56509ba7df7bbcb3c0a7e2e8d26 GIT binary patch literal 3026 zcmcJRe{54#6vubFt(3bC$_5UKBsVp(jdU-sKi>Ky4&xY$r~?%QgTdzZz1#Nn$9vs- zuloVPK#dV5f(ikGQHK&1G0MaUGcjhOF<>;&V2po=@kgSO7$N*)kbgX{Ti-ejaW=92 z=k=a<&pDs(IrsENx2){`;e(oIeyaU0RD&F-h&*n>covW(>JvIcfnaAM7)khPNMHh` z{si+SI!~dR^G=K%ELG?q5|Z-n7k@l>5q8^(i2wK99%pVyNv5|85q{VxjG!mcj3;;f zBOU9R;aP|qu{}0O6)9dswjDduZK-%d3U~VB0mchKUnCrnBq0(P;TsH!oh;sAP(8zU z@FS?f@yqqGQM>Et@jY9Qp_SO-Ko@Liq87PpClK=316+vn`Bu&2IbQ%Cy6ar14b9@& zre1U`2cBAg3toboOKEiEm_a3KP%f>jiu#9Zbpqa~+se=QZ<UoVH~H1n2PE0b_Odx+=3gkjB1LkBx4#iV6ooEI(Pv$WR6d2pDZ zSsfPV&?&Px-hE@sG6f-rJ$=sBNX`HrUcVwpJjSrS_@)oVOpoeD0|s79md)m6vDi6^%X2f&LgPQ?sq%c;6SAR=@Ifuyn| zOLe0?T17nRVR20l#B-zgZj zPpM6~{IJJMKuJdYe!rK<3b`w%N`)+2T~ZC2wu@>>or8xeh)(Z3PfQ|QK{R>eU^PT< z^Pjms23HN~XOV?;bnFNq?YtFIC|n{ck;VXvWlK<5t!vgf_mV*-CsRgDVj>;xs{G+4 zIW-5BE#$h&4_=Z`HAcl$m)W)LRjAUw^s38L#aW@su9L^AQRVR9-Wz+tRbzUiCCd27 z%OJ{{O~ZM%R?}21N@6h>2u3W{)=rYDL8w9s3LDBZ(SXyEVZ(!}CJ`ue!9*pk(|oN} zpk)qxa@s^&P=PkS`*<~Ik7=Xd?*ms0)0-`{(XkVN){A=znmOrG@F!kjV1pZ95plM8s;wtFA-qG7>(zz<=CJv8PL$^X4yJF6kMHRE< z+KE#@_y5cmd(ZQ)n5j2aq`v3So7JQqd+jOt2)JraQ>!rJUyeiS58dh@`u)Lw&X%b< z5ku9)6lczOZbQM?kmzGMRU4k_+$w5>u6Uj>K^Iqmj$e4E8qi!wIuzh1v>Hb zSpdEF9>dV&c)=*2u&J3zT4!Kdp*wY=sDzeZC9QwwtLB7#dnV9l^4(U+clNz%_*UB< zzuOJ2T8hZ!arP@pE-VN8v;LIOhF(B62ilyAj+nQnrP23MQ-kAb>DDp%)}?;;owy~h zsUnvv6^o+-RL;_Pmi2WP(fmM&7Xo1}7;rDZ_Nh1E8Wi9n?iF}Z + amp=args.amp + File "train.py", line 79, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 223, in + except : +AttributeError: module 'torch.cuda' has no attribute 'OutOfMemoryError' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-metadata.json new file mode 100644 index 0000000..4abf8d6 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:22:32.313270", + "startedAt": "2023-03-09T15:22:25.825765", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2500.0203874999997, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.987, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.99, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.988, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.993, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.838, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.036, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.946, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.894, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.987, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.986, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.989, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.004, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.005, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.995, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.026, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.004, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.017, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.998, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.994, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.002, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.033, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6473693847656 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug-internal.log new file mode 100644 index 0000000..d247aae --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug-internal.log @@ -0,0 +1,186 @@ +2023-03-09 23:22:25,873 INFO StreamThr :12640 [internal.py:wandb_internal():90] W&B internal server running at pid: 12640, started at: 2023-03-09 23:22:25.871255 +2023-03-09 23:22:25,875 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:22:25,876 INFO WriterThread:12640 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/run-i4zmblxx.wandb +2023-03-09 23:22:25,881 DEBUG SenderThread:12640 [sender.py:send():336] send: header +2023-03-09 23:22:25,881 DEBUG SenderThread:12640 [sender.py:send():336] send: run +2023-03-09 23:22:25,950 INFO SenderThread:12640 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/i4zmblxx +2023-03-09 23:22:27,203 INFO SenderThread:12640 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files +2023-03-09 23:22:27,203 INFO SenderThread:12640 [sender.py:_start_run_threads():1081] run started: i4zmblxx with start time 1678375345.871626 +2023-03-09 23:22:27,203 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:22:27,204 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:22:27,237 INFO SenderThread:12640 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:22:27,238 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:22:28,207 INFO Thread-13 :12640 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json +2023-03-09 23:22:32,205 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:22:32,239 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:22:32,288 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:22:32,293 DEBUG HandlerThread:12640 [system_info.py:__init__():31] System info init +2023-03-09 23:22:32,293 DEBUG HandlerThread:12640 [system_info.py:__init__():46] System info init done +2023-03-09 23:22:32,293 INFO HandlerThread:12640 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:22:32,293 INFO SystemMonitor:12640 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:22:32,294 INFO HandlerThread:12640 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:22:32,294 INFO SystemMonitor:12640 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:22:32,295 INFO SystemMonitor:12640 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:22:32,295 INFO SystemMonitor:12640 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:22:32,296 INFO SystemMonitor:12640 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:22:32,296 INFO SystemMonitor:12640 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:22:32,313 DEBUG HandlerThread:12640 [system_info.py:probe():195] Probing system +2023-03-09 23:22:32,332 DEBUG HandlerThread:12640 [git.py:repo():40] git repository is invalid +2023-03-09 23:22:32,332 DEBUG HandlerThread:12640 [system_info.py:probe():240] Probing system done +2023-03-09 23:22:32,332 DEBUG HandlerThread:12640 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:22:32.313270', 'startedAt': '2023-03-09T15:22:25.825765', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2500.0203874999997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.838, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2502.036, 'min': 0.0, 'max': 0.0}, {'current': 2499.946, 'min': 0.0, 'max': 0.0}, {'current': 2499.894, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.986, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.989, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.005, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.026, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2500.017, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2500.033, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6473693847656}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:22:32,332 INFO HandlerThread:12640 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:22:32,332 INFO HandlerThread:12640 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:22:32,332 DEBUG HandlerThread:12640 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:22:32,367 DEBUG HandlerThread:12640 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:22:32,367 DEBUG HandlerThread:12640 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:22:33,080 DEBUG HandlerThread:12640 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:22:33,128 INFO HandlerThread:12640 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:22:33,131 DEBUG SenderThread:12640 [sender.py:send():336] send: files +2023-03-09 23:22:33,131 INFO SenderThread:12640 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:22:33,140 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:22:33,166 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:22:33,374 INFO Thread-13 :12640 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-metadata.json +2023-03-09 23:22:33,374 INFO Thread-13 :12640 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/conda-environment.yaml +2023-03-09 23:22:33,374 INFO Thread-13 :12640 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt +2023-03-09 23:22:33,628 DEBUG SenderThread:12640 [sender.py:send():336] send: telemetry +2023-03-09 23:22:33,628 DEBUG SenderThread:12640 [sender.py:send():336] send: config +2023-03-09 23:22:33,631 DEBUG SenderThread:12640 [sender.py:send():336] send: exit +2023-03-09 23:22:33,631 INFO SenderThread:12640 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:22:33,632 INFO SenderThread:12640 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:22:33,709 INFO SenderThread:12640 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:22:33,715 INFO SenderThread:12640 [sender.py:send_exit():567] send defer +2023-03-09 23:22:33,730 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,736 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:22:33,747 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,747 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:22:33,747 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:22:33,748 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,748 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:22:33,748 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,748 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:22:33,748 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:22:33,748 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,748 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:22:33,748 INFO HandlerThread:12640 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:22:33,749 DEBUG SystemMonitor:12640 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:22:33,749 DEBUG SystemMonitor:12640 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:22:33,749 DEBUG SystemMonitor:12640 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:22:33,751 INFO HandlerThread:12640 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:22:33,752 INFO HandlerThread:12640 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:22:33,766 INFO HandlerThread:12640 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:22:33,766 INFO HandlerThread:12640 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:22:33,767 INFO HandlerThread:12640 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:22:33,767 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,767 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:22:33,767 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:22:33,767 DEBUG SenderThread:12640 [sender.py:send():336] send: stats +2023-03-09 23:22:33,768 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,768 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:22:33,768 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,768 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:22:33,768 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:22:33,768 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,768 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:22:33,769 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,769 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:22:33,769 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:22:33,769 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,769 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:22:33,769 DEBUG SenderThread:12640 [sender.py:send():336] send: summary +2023-03-09 23:22:33,796 INFO SenderThread:12640 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:22:33,796 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,796 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:22:33,796 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:22:33,796 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:33,797 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:22:33,797 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:33,797 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:22:33,802 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:22:34,317 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:34,343 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:22:34,343 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:34,343 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:34,343 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:22:34,343 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:34,344 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:22:34,376 INFO Thread-13 :12640 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json +2023-03-09 23:22:34,376 INFO Thread-13 :12640 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/config.yaml +2023-03-09 23:22:34,376 INFO Thread-13 :12640 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/output.log +2023-03-09 23:22:34,938 INFO wandb-upload_0:12640 [upload_job.py:push():138] Uploaded file /tmp/tmpbbv1ppqiwandb/s5k8b9uw-wandb-metadata.json +2023-03-09 23:22:35,318 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:35,670 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:22:35,675 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:35,680 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:35,685 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:22:35,716 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:35,721 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:22:35,751 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:22:35,752 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:35,752 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:22:35,752 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:35,752 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:22:35,752 INFO SenderThread:12640 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:22:36,318 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:36,378 INFO SenderThread:12640 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/output.log +2023-03-09 23:22:36,378 INFO SenderThread:12640 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files +2023-03-09 23:22:36,379 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:22:36,379 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/output.log output.log +2023-03-09 23:22:36,379 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt requirements.txt +2023-03-09 23:22:36,394 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/config.yaml config.yaml +2023-03-09 23:22:36,404 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json wandb-summary.json +2023-03-09 23:22:36,460 INFO SenderThread:12640 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:22:36,460 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:22:36,461 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:36,491 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:36,541 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:22:36,673 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:36,678 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:22:36,678 INFO SenderThread:12640 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:22:37,471 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:37,509 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:37,716 INFO wandb-upload_2:12640 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/requirements.txt +2023-03-09 23:22:38,038 INFO wandb-upload_1:12640 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/output.log +2023-03-09 23:22:38,104 INFO wandb-upload_4:12640 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/wandb-summary.json +2023-03-09 23:22:38,247 INFO wandb-upload_3:12640 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/files/config.yaml +2023-03-09 23:22:38,448 INFO Thread-12 :12640 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:22:38,448 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:38,448 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:22:38,448 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:38,448 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:22:38,448 INFO SenderThread:12640 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:22:38,448 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:22:38,449 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:38,449 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:22:38,449 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:38,449 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:22:38,560 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:38,894 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:22:38,894 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:38,894 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:38,894 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:22:38,895 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:38,895 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:22:38,895 INFO SenderThread:12640 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:22:38,895 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:22:38,895 INFO HandlerThread:12640 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:22:38,895 DEBUG SenderThread:12640 [sender.py:send():336] send: final +2023-03-09 23:22:38,895 DEBUG SenderThread:12640 [sender.py:send():336] send: footer +2023-03-09 23:22:38,895 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: defer +2023-03-09 23:22:38,895 INFO SenderThread:12640 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:22:38,896 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:38,896 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:38,896 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:22:38,897 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:22:38,897 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:22:38,897 DEBUG SenderThread:12640 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:22:38,913 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:22:38,964 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:22:39,712 INFO MainThread:12640 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:22:39,712 INFO MainThread:12640 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:22:39,712 INFO MainThread:12640 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:22:39,713 DEBUG HandlerThread:12640 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:22:39,713 INFO HandlerThread:12640 [handler.py:finish():842] shutting down handler +2023-03-09 23:22:39,897 INFO WriterThread:12640 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/run-i4zmblxx.wandb +2023-03-09 23:22:40,712 INFO SenderThread:12640 [sender.py:finish():1504] shutting down sender +2023-03-09 23:22:40,712 INFO SenderThread:12640 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:22:40,712 INFO SenderThread:12640 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug.log new file mode 100644 index 0000000..b8e98d6 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Configure stats pid to 12503 +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug.log +2023-03-09 23:22:25,861 INFO MainThread:12503 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/logs/debug-internal.log +2023-03-09 23:22:25,862 INFO MainThread:12503 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:22:25,862 INFO MainThread:12503 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:22:25,862 INFO MainThread:12503 [wandb_init.py:init():602] starting backend +2023-03-09 23:22:25,862 INFO MainThread:12503 [wandb_init.py:init():606] setting up manager +2023-03-09 23:22:25,868 INFO MainThread:12503 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:22:25,871 INFO MainThread:12503 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:22:25,875 INFO MainThread:12503 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:22:25,876 INFO MainThread:12503 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:22:27,203 INFO MainThread:12503 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:22:32,239 INFO MainThread:12503 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:22:32,240 INFO MainThread:12503 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:22:33,138 INFO MainThread:12503 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:22:33,139 INFO MainThread:12503 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:22:33,139 INFO MainThread:12503 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:22:33,139 INFO MainThread:12503 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:22:33,140 INFO MainThread:12503 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:22:33,140 INFO MainThread:12503 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:22:40,737 WARNING MsgRouterThr:12503 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/run-i4zmblxx.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232225-i4zmblxx/run-i4zmblxx.wandb new file mode 100644 index 0000000000000000000000000000000000000000..1af49350da92696a89e9646dd34416ed1c7347a5 GIT binary patch literal 5124 zcmcJTeTWog9LMjuH`j5^^USwV@ac-~w7X$uc3yTDFUUQQgz};^DbQ_p_SxIr+ud1b zXYWN3^Rl?KA|g^UsiYSuC@U}x2`eb|kHo(cipU}mgRsP2D6D7pZl0NY9=vp+TR3K) z?>@iZ@B4Xv&$BZd-*o$$^B*^?xb^5Zy#a*4IOyOZQZiAb%dxbsA;pxCk(SMY979pP+M9nr?3rE+yFvIp!2ek8tS zW6yfzbj;)Ca;?b`qm)jKgU0PU@?F_fCLPt}R0N5tBE_PybXtw2O#D(5Q$wj9$EXa6 zJ>nh^4!w8z%~x96p1pYH<)#9ES~W`)qzJxP)7%ES zz%)afty7_8*ie(am|Ms#sMXO^M2?X*97D)gtkSVVzrB{hpJ;wq^y9Z*SElPiJguCA zpJyNX1Se}*wFhuvt`(InGg}!4K;_yvE@e1oHd)DSLwH2u7ISk8$goSf(qPgy96Z<} zp&n7?7I1ThjY6`FY!j87anOYQb4$2+6=N7B%`7s9%2v)5njO1}xcS`dTyZd2F^vL_ zfj_L`HEA-6WgIYL6e@V|I$C9#k0UVqdl7(8YyL=M`<&3R+A^K^;++KlM&r8m{R46M zfMeJWt{ghHk#jA?IXLyVwq`o+y@gOE(i)clbO}t!5|QqiLIGVAt~$xWi?U%V4VkQ{ zQISl3&?w~ais!7-l&*?VA|?p2%4F6JE;TQ39~#b?UODcdSxp;6T~{;)Q3bkeU5^mD zbiJwf=0;H(-e)5-SHY>{u-~O!lersbexNk#g)Hu!S$Ww<{?RuH{#NYWL@C1@f?Y+c z;y|1b_XWf?6=1=rI4k3Iq+7dX;OzuX7s}5i=}w;cK5^rPr@!BPU?+|=JHL)Zy7a__ z1dnm`=L!gR)OtiHk9BlIj1fUumbziC1XmZWbhUu)tRce<+iTc^6#-w=;pdh`%Qm@w zxPYI-$A7BB5BTZCpP%7KbMj9CVrb&x&k26VwG8R8nxC4_BsPT#HA@A%ciIl`h9f99 zn01nAWEx|=!8hHoXch6QR*SvfMK{b?HZsg?uiLOI>tM2F;KK_plW>5^#Q9(9U~>Py zclS-;NOSY|6C{_uxsu=$8^+3Lt!-O&943?aByf_+u9a}ma-dOy)e;&lBh$g#WC)Fo zSaurXC9b+gVds>)&N_^>ufAiaYwV_gv2$PlQHQbDPoJ~*;z;xI8;P;QPyU_Y`?C_s)jGT_%yBE+6J>tjMA`q(uD<{2M2QZqIO`VP9xVLZ-F-KHq8!-`qT6w#1^KmP zQ6^rR=;O6(Z9rL$uqVpo8o_!_$BW1Jhg~l>2fQ48Y;PT2c75>3@~3g6h51dy%ZXQ? z>ErLd%W>>ns_J;#XYw$FosMOjS;74p>cq`gfh7wX-bZ2QK-C$@+>45qJyzRx^=zNp zTHu$iyF~$a7tTLhhr4;nzwG5W(&BN@))9`D?4hkAvGmy1kzyBk4m5{A8@PPx(fsn+ zAhaaDuf07ybuV#p{Hq@u`8iyZS+3UZSwwy*Hy^(dL;?SwcJHHjl5}lJW~7KWD{^f~ zVI;|$rBaeC$eyH8k|>Z|IwO(dqm(2GF)v_@lH7ni*Y47gSR4}3SRPTLu|WKJED?pw zqmf8~T4*Mc;OAqZs3ay$MG{1!u~>YKMA#X9s@SBu|;*(u@J?WFia3 zPe_9aERxKEAtvM*z{oThQ6LK;vtayMW&jhJ0ZfDj6UY~#z~ILSQcHy8;%hT-iLhKq zV@Fsnq?sZ#mq5crXfAf z3C^Rr$bRo?G?ze3o#jI6)LAZMKjgPIDE + amp=args.amp + File "train.py", line 79, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 228, in + model.use_checkpointing() + File "/code/UNet-master/2dunet-pytorch/unet/unet_model.py", line 39, in use_checkpointing + self.inc = torch.utils.checkpoint(self.inc) +TypeError: 'module' object is not callable \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-metadata.json new file mode 100644 index 0000000..c49897c --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:23:35.946816", + "startedAt": "2023-03-09T15:23:27.559689", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.627674999999, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2499.975, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.583, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.019, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.267, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.194, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.565, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.066, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.593, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.071, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.977, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.041, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.214, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.981, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.76, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.217, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.742, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.327, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.319, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.34, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.678, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.151, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2509.236, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.306, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.153, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.932, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.971, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2508.963, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.408, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.441, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.477, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.027, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.597, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.728, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.169, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.879, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2494.744, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2495.449, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2496.412, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.356, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.629, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.307, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.96, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.249, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.085, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.04, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.146, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.102, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.387, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.667, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.815, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.406, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.796, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.49, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.983, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.725, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.315, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.968, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.437, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.768, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.557, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.368, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.559, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.22, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2494.099, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2496.522, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.379, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.474, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.207, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.6474838256836 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json new file mode 100644 index 0000000..971d479 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 1}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug-internal.log new file mode 100644 index 0000000..b882e5d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug-internal.log @@ -0,0 +1,203 @@ +2023-03-09 23:23:27,598 INFO StreamThr :12815 [internal.py:wandb_internal():90] W&B internal server running at pid: 12815, started at: 2023-03-09 23:23:27.596443 +2023-03-09 23:23:27,600 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:23:27,601 INFO WriterThread:12815 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/run-zuik1xnj.wandb +2023-03-09 23:23:27,605 DEBUG SenderThread:12815 [sender.py:send():336] send: header +2023-03-09 23:23:27,605 DEBUG SenderThread:12815 [sender.py:send():336] send: run +2023-03-09 23:23:27,633 INFO SenderThread:12815 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/zuik1xnj +2023-03-09 23:23:30,822 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:23:30,822 INFO SenderThread:12815 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files +2023-03-09 23:23:30,823 INFO SenderThread:12815 [sender.py:_start_run_threads():1081] run started: zuik1xnj with start time 1678375407.59513 +2023-03-09 23:23:30,823 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:23:30,848 INFO SenderThread:12815 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:23:30,848 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:23:31,826 INFO Thread-13 :12815 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json +2023-03-09 23:23:35,823 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:23:35,849 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:23:35,900 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:23:35,905 DEBUG HandlerThread:12815 [system_info.py:__init__():31] System info init +2023-03-09 23:23:35,905 DEBUG HandlerThread:12815 [system_info.py:__init__():46] System info init done +2023-03-09 23:23:35,905 INFO HandlerThread:12815 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:23:35,906 INFO SystemMonitor:12815 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:23:35,906 INFO HandlerThread:12815 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:23:35,906 INFO SystemMonitor:12815 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:23:35,907 INFO SystemMonitor:12815 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:23:35,908 INFO SystemMonitor:12815 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:23:35,908 INFO SystemMonitor:12815 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:23:35,909 INFO SystemMonitor:12815 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:23:35,946 DEBUG HandlerThread:12815 [system_info.py:probe():195] Probing system +2023-03-09 23:23:35,969 DEBUG HandlerThread:12815 [git.py:repo():40] git repository is invalid +2023-03-09 23:23:35,970 DEBUG HandlerThread:12815 [system_info.py:probe():240] Probing system done +2023-03-09 23:23:35,970 DEBUG HandlerThread:12815 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:23:35.946816', 'startedAt': '2023-03-09T15:23:27.559689', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.627674999999, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.975, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.583, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.019, 'min': 0.0, 'max': 0.0}, {'current': 2504.267, 'min': 0.0, 'max': 0.0}, {'current': 2500.194, 'min': 0.0, 'max': 0.0}, {'current': 2504.565, 'min': 0.0, 'max': 0.0}, {'current': 2504.066, 'min': 0.0, 'max': 0.0}, {'current': 2499.593, 'min': 0.0, 'max': 0.0}, {'current': 2504.001, 'min': 0.0, 'max': 0.0}, {'current': 2504.071, 'min': 0.0, 'max': 0.0}, {'current': 2504.977, 'min': 0.0, 'max': 0.0}, {'current': 2503.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.214, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2505.76, 'min': 0.0, 'max': 0.0}, {'current': 2504.217, 'min': 0.0, 'max': 0.0}, {'current': 2502.742, 'min': 0.0, 'max': 0.0}, {'current': 2505.327, 'min': 0.0, 'max': 0.0}, {'current': 2504.319, 'min': 0.0, 'max': 0.0}, {'current': 2504.34, 'min': 0.0, 'max': 0.0}, {'current': 2504.678, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.151, 'min': 0.0, 'max': 0.0}, {'current': 2509.236, 'min': 0.0, 'max': 0.0}, {'current': 2503.306, 'min': 0.0, 'max': 0.0}, {'current': 2505.153, 'min': 0.0, 'max': 0.0}, {'current': 2504.932, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.971, 'min': 0.0, 'max': 0.0}, {'current': 2508.963, 'min': 0.0, 'max': 0.0}, {'current': 2504.408, 'min': 0.0, 'max': 0.0}, {'current': 2505.441, 'min': 0.0, 'max': 0.0}, {'current': 2503.477, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2500.597, 'min': 0.0, 'max': 0.0}, {'current': 2504.728, 'min': 0.0, 'max': 0.0}, {'current': 2505.169, 'min': 0.0, 'max': 0.0}, {'current': 2503.879, 'min': 0.0, 'max': 0.0}, {'current': 2494.744, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2495.449, 'min': 0.0, 'max': 0.0}, {'current': 2496.412, 'min': 0.0, 'max': 0.0}, {'current': 2504.356, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.629, 'min': 0.0, 'max': 0.0}, {'current': 2499.307, 'min': 0.0, 'max': 0.0}, {'current': 2504.96, 'min': 0.0, 'max': 0.0}, {'current': 2499.249, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.085, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2501.146, 'min': 0.0, 'max': 0.0}, {'current': 2504.102, 'min': 0.0, 'max': 0.0}, {'current': 2500.387, 'min': 0.0, 'max': 0.0}, {'current': 2499.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.815, 'min': 0.0, 'max': 0.0}, {'current': 2500.406, 'min': 0.0, 'max': 0.0}, {'current': 2500.796, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.49, 'min': 0.0, 'max': 0.0}, {'current': 2500.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2500.315, 'min': 0.0, 'max': 0.0}, {'current': 2499.968, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.437, 'min': 0.0, 'max': 0.0}, {'current': 2498.768, 'min': 0.0, 'max': 0.0}, {'current': 2499.557, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2500.559, 'min': 0.0, 'max': 0.0}, {'current': 2500.22, 'min': 0.0, 'max': 0.0}, {'current': 2494.099, 'min': 0.0, 'max': 0.0}, {'current': 2496.522, 'min': 0.0, 'max': 0.0}, {'current': 2505.379, 'min': 0.0, 'max': 0.0}, {'current': 2500.474, 'min': 0.0, 'max': 0.0}, {'current': 2506.207, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6474838256836}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:23:35,970 INFO HandlerThread:12815 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:23:35,970 INFO HandlerThread:12815 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:23:35,970 DEBUG HandlerThread:12815 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:23:36,009 DEBUG HandlerThread:12815 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:23:36,009 DEBUG HandlerThread:12815 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:23:36,729 DEBUG HandlerThread:12815 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:23:36,782 INFO HandlerThread:12815 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:23:36,784 DEBUG SenderThread:12815 [sender.py:send():336] send: files +2023-03-09 23:23:36,785 INFO SenderThread:12815 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:23:36,804 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:23:36,822 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:23:37,010 INFO Thread-13 :12815 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/conda-environment.yaml +2023-03-09 23:23:37,015 INFO Thread-13 :12815 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-metadata.json +2023-03-09 23:23:37,015 INFO Thread-13 :12815 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt +2023-03-09 23:23:37,313 DEBUG SenderThread:12815 [sender.py:send():336] send: telemetry +2023-03-09 23:23:37,323 DEBUG SenderThread:12815 [sender.py:send():336] send: config +2023-03-09 23:23:37,356 DEBUG SenderThread:12815 [sender.py:send():336] send: exit +2023-03-09 23:23:37,361 INFO SenderThread:12815 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:23:37,361 INFO SenderThread:12815 [sender.py:send_exit():561] handling runtime: 1 +2023-03-09 23:23:37,410 INFO SenderThread:12815 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:23:37,416 INFO SenderThread:12815 [sender.py:send_exit():567] send defer +2023-03-09 23:23:37,447 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,452 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:23:37,469 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,474 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:23:37,474 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:23:37,484 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,490 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:23:37,510 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,510 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:23:37,510 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:23:37,511 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,511 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:23:37,511 INFO HandlerThread:12815 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:23:37,511 DEBUG SystemMonitor:12815 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:23:37,512 DEBUG SystemMonitor:12815 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:23:37,512 DEBUG SystemMonitor:12815 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:23:37,513 INFO HandlerThread:12815 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:23:37,513 INFO HandlerThread:12815 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:23:37,527 INFO HandlerThread:12815 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:23:37,528 INFO HandlerThread:12815 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:23:37,528 INFO HandlerThread:12815 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:23:37,528 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,528 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:23:37,528 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:23:37,528 DEBUG SenderThread:12815 [sender.py:send():336] send: stats +2023-03-09 23:23:37,529 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,529 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:23:37,529 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,529 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:23:37,529 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:23:37,529 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,529 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:23:37,529 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,529 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:23:37,529 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:23:37,529 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,529 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:23:37,530 DEBUG SenderThread:12815 [sender.py:send():336] send: summary +2023-03-09 23:23:37,551 INFO SenderThread:12815 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:23:37,552 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,552 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:23:37,552 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:23:37,552 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:37,552 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:23:37,552 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:37,552 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:23:37,557 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:23:37,909 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:38,024 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:23:38,024 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:38,024 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:38,024 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:23:38,025 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:38,025 INFO Thread-13 :12815 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json +2023-03-09 23:23:38,025 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:23:38,025 INFO Thread-13 :12815 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/config.yaml +2023-03-09 23:23:38,025 INFO Thread-13 :12815 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/output.log +2023-03-09 23:23:38,154 INFO wandb-upload_0:12815 [upload_job.py:push():138] Uploaded file /tmp/tmpsxjuifhcwandb/2bvlhtvx-wandb-metadata.json +2023-03-09 23:23:38,910 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:39,383 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:23:39,384 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:39,384 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:39,384 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:23:39,384 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:39,384 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:23:39,385 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:23:39,385 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:39,385 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:23:39,385 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:39,385 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:23:39,385 INFO SenderThread:12815 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:23:39,910 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:40,027 INFO SenderThread:12815 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/output.log +2023-03-09 23:23:40,027 INFO SenderThread:12815 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files +2023-03-09 23:23:40,028 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:23:40,028 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/output.log output.log +2023-03-09 23:23:40,028 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt requirements.txt +2023-03-09 23:23:40,028 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/config.yaml config.yaml +2023-03-09 23:23:40,034 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json wandb-summary.json +2023-03-09 23:23:40,039 INFO SenderThread:12815 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:23:40,039 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:23:40,040 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:40,100 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:40,212 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:23:40,311 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:40,316 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:23:40,316 INFO SenderThread:12815 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:23:41,125 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:41,149 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:41,419 INFO wandb-upload_0:12815 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/config.yaml +2023-03-09 23:23:42,126 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:42,126 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:43,127 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:43,127 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:43,761 INFO wandb-upload_4:12815 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/wandb-summary.json +2023-03-09 23:23:43,775 INFO wandb-upload_1:12815 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/output.log +2023-03-09 23:23:44,127 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:44,128 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:45,129 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:45,129 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:45,153 ERROR wandb-upload_2:12815 [internal_api.py:upload_file():1997] upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/anony-moose-510831/U-Net/zuik1xnj/requirements.txt?Expires=1678461912&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=WYC44eYQ8WRV2G%2FO0wXK7XBD6qGzaDSnQZEtZErQBIJx5mRGJDKvrtCpGqonht7Iv2fWpB3jKthANHeww9iPylt15Aq5FtCWstvUHTqgEcd3KjQOdWmgMyChT3M11cybKsfpGWxi9DA3OtOyGqCx61zSGX1CMgRsNfsuHTffkdCPLlz2Mo3n66JPd8fm33qP57l%2Bsoy6YkERAqaUNwqSZvOG%2BN3HmbgPyHvsnBK%2FbxDfkYUeWBeRT5p3uaHK%2BkQsHma5wBstJ%2B1t%2BNIDOewSlsv5ZT4u33U9S1i9RBfwfgjzRLeMFJLzFdu0Qhgtz2T2Y2nZ2iAA18Lm%2Bv3sTK6q0w%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-production.appspot.com/anony-moose-510831/U-Net/zuik1xnj/requirements.txt?Expires=1678461912&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=WYC44eYQ8WRV2G%2FO0wXK7XBD6qGzaDSnQZEtZErQBIJx5mRGJDKvrtCpGqonht7Iv2fWpB3jKthANHeww9iPylt15Aq5FtCWstvUHTqgEcd3KjQOdWmgMyChT3M11cybKsfpGWxi9DA3OtOyGqCx61zSGX1CMgRsNfsuHTffkdCPLlz2Mo3n66JPd8fm33qP57l%2Bsoy6YkERAqaUNwqSZvOG%2BN3HmbgPyHvsnBK%2FbxDfkYUeWBeRT5p3uaHK%2BkQsHma5wBstJ%2B1t%2BNIDOewSlsv5ZT4u33U9S1i9RBfwfgjzRLeMFJLzFdu0Qhgtz2T2Y2nZ2iAA18Lm%2Bv3sTK6q0w%3D%3D (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')) +2023-03-09 23:23:45,153 ERROR wandb-upload_2:12815 [internal_api.py:upload_file():1999] upload_file request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '2452'} +2023-03-09 23:23:45,153 ERROR wandb-upload_2:12815 [internal_api.py:upload_file():2001] upload_file response body: +2023-03-09 23:23:46,129 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:46,130 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:46,991 INFO wandb-upload_2:12815 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/files/requirements.txt +2023-03-09 23:23:47,130 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:47,130 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:47,131 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:23:47,191 INFO Thread-12 :12815 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:23:47,191 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:47,191 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:23:47,191 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:47,192 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:23:47,192 INFO SenderThread:12815 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:23:47,192 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:23:47,192 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:47,192 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:23:47,192 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:47,192 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:23:47,539 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:23:47,539 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:47,539 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:23:47,539 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:47,539 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:23:47,539 INFO SenderThread:12815 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:23:47,539 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:23:47,539 INFO HandlerThread:12815 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:23:47,540 DEBUG SenderThread:12815 [sender.py:send():336] send: final +2023-03-09 23:23:47,540 DEBUG SenderThread:12815 [sender.py:send():336] send: footer +2023-03-09 23:23:47,540 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: defer +2023-03-09 23:23:47,540 INFO SenderThread:12815 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:23:47,540 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:47,541 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:47,541 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:23:47,541 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:23:47,541 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:23:47,542 DEBUG SenderThread:12815 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:23:47,562 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:23:47,588 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:23:52,578 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:23:57,579 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:23:59,735 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:23:59,736 INFO MainThread:12815 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:23:59,736 INFO MainThread:12815 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:23:59,737 INFO MainThread:12815 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:23:59,737 DEBUG HandlerThread:12815 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:23:59,738 INFO HandlerThread:12815 [handler.py:finish():842] shutting down handler +2023-03-09 23:24:00,734 INFO SenderThread:12815 [sender.py:finish():1504] shutting down sender +2023-03-09 23:24:00,735 INFO SenderThread:12815 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:24:00,735 INFO SenderThread:12815 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:24:00,736 INFO WriterThread:12815 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/run-zuik1xnj.wandb diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug.log new file mode 100644 index 0000000..de9f730 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug.log @@ -0,0 +1,28 @@ +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Configure stats pid to 12683 +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug.log +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/logs/debug-internal.log +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:init():602] starting backend +2023-03-09 23:23:27,590 INFO MainThread:12683 [wandb_init.py:init():606] setting up manager +2023-03-09 23:23:27,593 INFO MainThread:12683 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:23:27,594 INFO MainThread:12683 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:23:27,597 INFO MainThread:12683 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:23:27,598 INFO MainThread:12683 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:23:30,821 INFO MainThread:12683 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:23:35,850 INFO MainThread:12683 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:23:35,850 INFO MainThread:12683 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:23:36,792 INFO MainThread:12683 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:23:36,792 INFO MainThread:12683 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:23:36,792 INFO MainThread:12683 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:23:36,792 INFO MainThread:12683 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:23:36,793 INFO MainThread:12683 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:23:36,793 INFO MainThread:12683 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:24:00,768 WARNING MsgRouterThr:12683 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/run-zuik1xnj.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232327-zuik1xnj/run-zuik1xnj.wandb new file mode 100644 index 0000000000000000000000000000000000000000..8a56c06a2f37d2a564cf7494ab7fdab0fe33c8e9 GIT binary patch literal 3544 zcmcIn4{Q@<6yMi%qdPVj8#pMSA2VzP*IV0b=~}JPjXyydBQQo{Vw&sS*Y$2~uijnv z2PEJ^L}7~y(MTW?0uhNuA%Kx+Km>`1s3ejAF)B!m6918C%#fM*zUzAHLWppQTe94J zf8Kk)-+S+OZ{am{%~y{X6$d`(2o@m+>PNLs91wj0uNaFFPdw!3{K2M>Hy#r?e<;WW zy#hzxL`5gd8OE_TD|LB$)AIMj>(Xc8wyhs|{`+=qxi=yCb&s6wRZ`p*v( zJzG$@wA&R7en^E2Bts!>+4Y!c1Yxq*P~A6Zj<9fWraE zNimFK!FtRXVTg z5IWXXUZNPn3?BViWS|b`Uoj4tTg{XLOeq-M_L^JEM!CS03%Z*lBX6aB)Qx%BA87KJ zmpk~R1Syx4^yq%T(`0V8n3QQbukDn!yy1nUTN3i&&_y9D^5*W+6yHfSOv$AfF{bq9 zueM6PM8qppB1jsz4&%j?ZeyFdchb*BdMZbtcqYiFkxBZM>oCv#L^rX#HvbaS_Ym!J*O!8JAZ(V_1Za0+GJRDb49OpG1L)+)gl6YJaB&b)@iUkUO z`D)6D1Qa-A&SYTs&|kM~&Icj?0*UcL7j~r-a1EqWXONd9PC_j6!WR5(owMt)h2XDD zo@DwPdVAy+=;OV%%s-bRdpCk9Mt;IHb>`jYMr=+fn#EF*;98?o*i2t-1BOHq?DKgW zu$0D+q!clmB##<}0|ZBSwNqo^!2+UVtG$~kqKO5Fc3;~xVgb=NH8+-h45rHE!LfQg zv-<^rwEj+zf*}Kufiw!RNYVh6R3qc0t^xNDsWYKRMI!Khk;0b-oKjMd*=#D3e`ufu zM|dF-p^ZDw0xN@wt#-;vMFA@t4{Wu<%D0F29(WNLEO2cM_fQdOWW!xwe}y8lO8rhTDYp`lk7 zgg*S)E-RszwXLZe1XEMxHD+Q??B4^SFS%1A`g~2dYm3p4Y*s6rA|R_SD=l?(M#}^W zZ(0Gop#!g5f%jxZSh@hF?vr0D5SnBA0p9@118fvPO&V9j%;`>n+!M9YCqhr@?o6Mq zs6wKVz8HUUx>KODTuzj3!)-{DMv`$>N((rG4LfXBmy#Mg_7QdEPgr4T!_vd&C`&a3 zES(s9%L+@&ZbnW&1E!|S51U1>f8ZS;>7fGQsn2~&>ah}&L2WR6;GrXp&m@co#6k<6 zdUZ!Xg)pN4!ufLttw7k&iv6>|l(QdI)|LkZm}KEAz^#>K7utsG4pfOQu6cIfNjNey ze2mt@O=I#rf9T+!$T^p($*79J=H?id@8+c>#njr|kIJ~9C%`RWxhCf%rZjf~W?_yE zIcGCd3a_b}MlVd@f_@(x@;hfS)rH&U&~5jyp`f#Zv5S({1*10SXFblTOgU5;h<6vO zfp8ns_PAA;O1g8agB?Hav;&Y?&Z4sE&X*Z@oDKgWxzHX|)NZ>zw*(gMKZJ_@1v@D; A8~^|S literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/conda-environment.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml new file mode 100644 index 0000000..9c9a6a7 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml @@ -0,0 +1,49 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678375513.665997 + t: + 1: + - 1 + - 41 + - 55 + 2: + - 1 + - 41 + - 55 + 3: + - 23 + - 24 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +amp: + desc: null + value: false +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true +val_percent: + desc: null + value: 0.1 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log new file mode 100644 index 0000000..f71d330 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log @@ -0,0 +1,50 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 4580 + Validation size: 508 + Checkpoints: True + Device: cuda + Images scaling: 0.5 + Mixed Precision: False +ERROR: Detected OutOfMemoryError! Enabling checkpointing to reduce memory usage, but this slows down training. Consider enabling AMP (--amp) for fast and memory efficient training +INFO: Creating dataset with 5088 examples +INFO: Scanning mask files to determine unique values + + + + + + + + + + + + +100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5088/5088 [00:25<00:00, 195.99it/s] +INFO: Unique mask values: [0, 1] +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 4580 + Validation size: 508 + Checkpoints: True + Device: cuda + Images scaling: 0.5 + Mixed Precision: False +Traceback (most recent call last): + File "train.py", line 221, in + amp=args.amp + File "train.py", line 79, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 237, in + amp=args.amp + File "train.py", line 79, in train_model + lr=learning_rate, weight_decay=weight_decay, momentum=momentum, foreach=True) +TypeError: __init__() got an unexpected keyword argument 'foreach' \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-metadata.json new file mode 100644 index 0000000..3fe9192 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:25:20.169296", + "startedAt": "2023-03-09T15:25:13.407685", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2501.960975, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.456, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.996, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.259, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.149, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.256, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.58, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.554, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.5, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.461, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.39, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.399, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.424, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.771, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.955, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.512, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.438, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.453, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.64, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.1, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.326, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.648, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.398, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.999, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.948, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.376, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.919, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.893, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.848, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.066, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.382, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.361, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.042, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.782, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.424, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.855, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.084, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.743, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.691, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.131, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.404, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.183, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.048, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.313, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.62, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.298, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.482, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.193, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.028, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.69, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.125, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.778, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.523, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.51, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.238, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.651, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.134, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.073, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.447, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.184, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.347, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.518, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.758, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.648, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.246, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.399, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.542, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.373, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.195, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.235, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.635, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.58, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.271, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.64766693115234 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json new file mode 100644 index 0000000..3892ef5 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json @@ -0,0 +1 @@ +{"_wandb": {"runtime": 31}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug-internal.log new file mode 100644 index 0000000..f6a2c64 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug-internal.log @@ -0,0 +1,204 @@ +2023-03-09 23:25:13,669 INFO StreamThr :12990 [internal.py:wandb_internal():90] W&B internal server running at pid: 12990, started at: 2023-03-09 23:25:13.666813 +2023-03-09 23:25:13,670 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:25:13,671 INFO WriterThread:12990 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/run-ba99nlnr.wandb +2023-03-09 23:25:13,676 DEBUG SenderThread:12990 [sender.py:send():336] send: header +2023-03-09 23:25:13,676 DEBUG SenderThread:12990 [sender.py:send():336] send: run +2023-03-09 23:25:13,718 INFO SenderThread:12990 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/ba99nlnr +2023-03-09 23:25:15,041 INFO SenderThread:12990 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files +2023-03-09 23:25:15,042 INFO SenderThread:12990 [sender.py:_start_run_threads():1081] run started: ba99nlnr with start time 1678375513.665997 +2023-03-09 23:25:15,042 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:25:15,042 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:25:15,073 INFO SenderThread:12990 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:25:15,074 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:25:16,045 INFO Thread-13 :12990 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json +2023-03-09 23:25:20,042 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:20,075 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:20,128 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:25:20,133 DEBUG HandlerThread:12990 [system_info.py:__init__():31] System info init +2023-03-09 23:25:20,133 DEBUG HandlerThread:12990 [system_info.py:__init__():46] System info init done +2023-03-09 23:25:20,133 INFO HandlerThread:12990 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:25:20,133 INFO SystemMonitor:12990 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:25:20,133 INFO HandlerThread:12990 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:25:20,134 INFO SystemMonitor:12990 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:25:20,134 INFO SystemMonitor:12990 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:25:20,135 INFO SystemMonitor:12990 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:25:20,135 INFO SystemMonitor:12990 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:25:20,136 INFO SystemMonitor:12990 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:25:20,169 DEBUG HandlerThread:12990 [system_info.py:probe():195] Probing system +2023-03-09 23:25:20,192 DEBUG HandlerThread:12990 [git.py:repo():40] git repository is invalid +2023-03-09 23:25:20,192 DEBUG HandlerThread:12990 [system_info.py:probe():240] Probing system done +2023-03-09 23:25:20,192 DEBUG HandlerThread:12990 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:25:20.169296', 'startedAt': '2023-03-09T15:25:13.407685', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.960975, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.456, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.259, 'min': 0.0, 'max': 0.0}, {'current': 2502.149, 'min': 0.0, 'max': 0.0}, {'current': 2498.256, 'min': 0.0, 'max': 0.0}, {'current': 2507.58, 'min': 0.0, 'max': 0.0}, {'current': 2499.554, 'min': 0.0, 'max': 0.0}, {'current': 2500.5, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2501.39, 'min': 0.0, 'max': 0.0}, {'current': 2501.399, 'min': 0.0, 'max': 0.0}, {'current': 2500.424, 'min': 0.0, 'max': 0.0}, {'current': 2501.771, 'min': 0.0, 'max': 0.0}, {'current': 2503.955, 'min': 0.0, 'max': 0.0}, {'current': 2499.512, 'min': 0.0, 'max': 0.0}, {'current': 2501.438, 'min': 0.0, 'max': 0.0}, {'current': 2500.453, 'min': 0.0, 'max': 0.0}, {'current': 2498.64, 'min': 0.0, 'max': 0.0}, {'current': 2500.1, 'min': 0.0, 'max': 0.0}, {'current': 2500.326, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2500.398, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.948, 'min': 0.0, 'max': 0.0}, {'current': 2500.376, 'min': 0.0, 'max': 0.0}, {'current': 2499.919, 'min': 0.0, 'max': 0.0}, {'current': 2498.893, 'min': 0.0, 'max': 0.0}, {'current': 2499.848, 'min': 0.0, 'max': 0.0}, {'current': 2500.066, 'min': 0.0, 'max': 0.0}, {'current': 2500.382, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.361, 'min': 0.0, 'max': 0.0}, {'current': 2500.042, 'min': 0.0, 'max': 0.0}, {'current': 2500.782, 'min': 0.0, 'max': 0.0}, {'current': 2501.424, 'min': 0.0, 'max': 0.0}, {'current': 2499.855, 'min': 0.0, 'max': 0.0}, {'current': 2500.084, 'min': 0.0, 'max': 0.0}, {'current': 2500.743, 'min': 0.0, 'max': 0.0}, {'current': 2506.691, 'min': 0.0, 'max': 0.0}, {'current': 2507.131, 'min': 0.0, 'max': 0.0}, {'current': 2507.404, 'min': 0.0, 'max': 0.0}, {'current': 2507.183, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.048, 'min': 0.0, 'max': 0.0}, {'current': 2506.313, 'min': 0.0, 'max': 0.0}, {'current': 2505.62, 'min': 0.0, 'max': 0.0}, {'current': 2501.298, 'min': 0.0, 'max': 0.0}, {'current': 2505.482, 'min': 0.0, 'max': 0.0}, {'current': 2506.193, 'min': 0.0, 'max': 0.0}, {'current': 2500.028, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.69, 'min': 0.0, 'max': 0.0}, {'current': 2503.125, 'min': 0.0, 'max': 0.0}, {'current': 2505.778, 'min': 0.0, 'max': 0.0}, {'current': 2499.523, 'min': 0.0, 'max': 0.0}, {'current': 2506.51, 'min': 0.0, 'max': 0.0}, {'current': 2504.238, 'min': 0.0, 'max': 0.0}, {'current': 2507.651, 'min': 0.0, 'max': 0.0}, {'current': 2505.134, 'min': 0.0, 'max': 0.0}, {'current': 2500.073, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.447, 'min': 0.0, 'max': 0.0}, {'current': 2499.184, 'min': 0.0, 'max': 0.0}, {'current': 2500.347, 'min': 0.0, 'max': 0.0}, {'current': 2500.518, 'min': 0.0, 'max': 0.0}, {'current': 2503.758, 'min': 0.0, 'max': 0.0}, {'current': 2506.648, 'min': 0.0, 'max': 0.0}, {'current': 2501.246, 'min': 0.0, 'max': 0.0}, {'current': 2500.399, 'min': 0.0, 'max': 0.0}, {'current': 2500.542, 'min': 0.0, 'max': 0.0}, {'current': 2503.373, 'min': 0.0, 'max': 0.0}, {'current': 2501.195, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.235, 'min': 0.0, 'max': 0.0}, {'current': 2503.635, 'min': 0.0, 'max': 0.0}, {'current': 2505.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.58, 'min': 0.0, 'max': 0.0}, {'current': 2500.271, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64766693115234}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:25:20,192 INFO HandlerThread:12990 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:25:20,192 INFO HandlerThread:12990 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:25:20,192 DEBUG HandlerThread:12990 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:25:20,250 DEBUG HandlerThread:12990 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:25:20,250 DEBUG HandlerThread:12990 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:25:20,972 DEBUG HandlerThread:12990 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:25:21,008 INFO HandlerThread:12990 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:25:21,011 DEBUG SenderThread:12990 [sender.py:send():336] send: files +2023-03-09 23:25:21,011 INFO SenderThread:12990 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:25:21,040 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:25:21,061 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:25:21,228 INFO Thread-13 :12990 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/conda-environment.yaml +2023-03-09 23:25:21,244 INFO Thread-13 :12990 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt +2023-03-09 23:25:21,244 INFO Thread-13 :12990 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-metadata.json +2023-03-09 23:25:21,561 DEBUG SenderThread:12990 [sender.py:send():336] send: telemetry +2023-03-09 23:25:21,566 DEBUG SenderThread:12990 [sender.py:send():336] send: config +2023-03-09 23:25:22,232 INFO Thread-13 :12990 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:22,300 INFO wandb-upload_0:12990 [upload_job.py:push():138] Uploaded file /tmp/tmpcwx89scywandb/frhvixk2-wandb-metadata.json +2023-03-09 23:25:24,254 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:25,410 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:26,529 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:28,610 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:30,425 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:30,709 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:32,722 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:33,923 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:35,513 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:36,118 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:36,128 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:36,320 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:38,521 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:40,519 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:40,609 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:41,024 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:42,808 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:44,913 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:45,615 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:46,108 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:46,724 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:48,719 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:48,720 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml +2023-03-09 23:25:50,823 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:51,110 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:25:51,421 DEBUG SenderThread:12990 [sender.py:send():336] send: telemetry +2023-03-09 23:25:51,422 DEBUG SenderThread:12990 [sender.py:send():336] send: config +2023-03-09 23:25:51,427 DEBUG SenderThread:12990 [sender.py:send():336] send: exit +2023-03-09 23:25:51,427 INFO SenderThread:12990 [sender.py:send_exit():559] handling exit code: 1 +2023-03-09 23:25:51,427 INFO SenderThread:12990 [sender.py:send_exit():561] handling runtime: 31 +2023-03-09 23:25:51,448 INFO SenderThread:12990 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:25:51,449 INFO SenderThread:12990 [sender.py:send_exit():567] send defer +2023-03-09 23:25:51,449 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,449 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:25:51,449 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,450 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:25:51,450 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:25:51,450 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,450 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:25:51,450 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,450 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:25:51,450 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:25:51,450 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,450 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:25:51,450 INFO HandlerThread:12990 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:25:51,451 DEBUG SystemMonitor:12990 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:25:51,451 DEBUG SystemMonitor:12990 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:25:51,451 DEBUG SystemMonitor:12990 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:25:51,453 INFO HandlerThread:12990 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:25:51,453 INFO HandlerThread:12990 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:25:51,469 INFO HandlerThread:12990 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:25:51,469 INFO HandlerThread:12990 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:25:51,469 INFO HandlerThread:12990 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:25:51,469 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,469 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:25:51,469 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:25:51,470 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,470 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:25:51,472 DEBUG SenderThread:12990 [sender.py:send():336] send: stats +2023-03-09 23:25:51,472 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,472 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:25:51,472 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:25:51,473 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,473 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:25:51,473 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,473 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:25:51,473 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:25:51,473 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,473 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:25:51,473 DEBUG SenderThread:12990 [sender.py:send():336] send: summary +2023-03-09 23:25:51,497 INFO SenderThread:12990 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:25:51,498 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,498 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:25:51,498 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:25:51,498 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,498 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:25:51,498 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,498 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:25:51,503 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:25:51,824 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json +2023-03-09 23:25:51,943 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:25:51,958 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:51,963 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:25:51,989 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:51,994 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:25:52,427 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:25:52,840 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:52,840 INFO Thread-13 :12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml +2023-03-09 23:25:53,958 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:25:53,958 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:25:53,958 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:53,958 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:25:53,959 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:53,959 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:25:53,959 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:25:53,959 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:53,959 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:25:53,960 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:53,960 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:25:53,960 INFO SenderThread:12990 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:25:54,842 INFO SenderThread:12990 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:54,843 INFO SenderThread:12990 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files +2023-03-09 23:25:54,843 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:25:54,843 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log output.log +2023-03-09 23:25:54,844 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt requirements.txt +2023-03-09 23:25:54,849 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml config.yaml +2023-03-09 23:25:54,884 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json wandb-summary.json +2023-03-09 23:25:54,955 INFO SenderThread:12990 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:25:54,955 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:25:54,975 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:54,981 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:25:55,047 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:55,047 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:25:55,047 INFO SenderThread:12990 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:25:56,060 INFO wandb-upload_0:12990 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/output.log +2023-03-09 23:25:56,440 INFO wandb-upload_2:12990 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/requirements.txt +2023-03-09 23:25:56,515 INFO wandb-upload_4:12990 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/wandb-summary.json +2023-03-09 23:25:56,565 INFO wandb-upload_1:12990 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/files/config.yaml +2023-03-09 23:25:56,765 INFO Thread-12 :12990 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:25:56,765 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:56,765 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:25:56,766 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:56,766 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:25:56,766 INFO SenderThread:12990 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:25:56,766 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:25:56,766 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:56,766 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:25:56,766 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:56,766 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:25:57,095 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:25:57,096 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:57,096 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:25:57,096 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:57,096 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:25:57,096 INFO SenderThread:12990 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:25:57,096 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:25:57,096 INFO HandlerThread:12990 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:25:57,097 DEBUG SenderThread:12990 [sender.py:send():336] send: final +2023-03-09 23:25:57,097 DEBUG SenderThread:12990 [sender.py:send():336] send: footer +2023-03-09 23:25:57,097 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: defer +2023-03-09 23:25:57,097 INFO SenderThread:12990 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:25:57,097 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:25:57,098 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:25:57,098 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:25:57,099 DEBUG SenderThread:12990 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:25:57,104 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:25:57,135 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:25:57,827 INFO MainThread:12990 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:25:57,827 INFO MainThread:12990 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:25:57,827 INFO MainThread:12990 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:25:57,828 DEBUG HandlerThread:12990 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:25:57,828 INFO HandlerThread:12990 [handler.py:finish():842] shutting down handler +2023-03-09 23:25:58,099 INFO WriterThread:12990 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/run-ba99nlnr.wandb +2023-03-09 23:25:58,826 INFO SenderThread:12990 [sender.py:finish():1504] shutting down sender +2023-03-09 23:25:58,827 INFO SenderThread:12990 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:25:58,827 INFO SenderThread:12990 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug.log new file mode 100644 index 0000000..b08800d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug.log @@ -0,0 +1,42 @@ +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Configure stats pid to 12858 +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug.log +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/logs/debug-internal.log +2023-03-09 23:25:13,660 INFO MainThread:12858 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:25:13,661 INFO MainThread:12858 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:25:13,661 INFO MainThread:12858 [wandb_init.py:init():602] starting backend +2023-03-09 23:25:13,661 INFO MainThread:12858 [wandb_init.py:init():606] setting up manager +2023-03-09 23:25:13,663 INFO MainThread:12858 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:25:13,665 INFO MainThread:12858 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:25:13,668 INFO MainThread:12858 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:25:13,669 INFO MainThread:12858 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:25:15,041 INFO MainThread:12858 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:25:20,075 INFO MainThread:12858 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:25:20,075 INFO MainThread:12858 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:25:21,019 INFO MainThread:12858 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:25:21,019 INFO MainThread:12858 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:25:21,020 INFO MainThread:12858 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:25:21,020 INFO MainThread:12858 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:25:21,020 INFO MainThread:12858 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:25:21,021 INFO MainThread:12858 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Configure stats pid to 12858 +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug.log +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug-internal.log +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():597] wandb.init() called when a run is still active +2023-03-09 23:25:51,422 INFO MainThread:12858 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:25:58,846 WARNING MsgRouterThr:12858 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/run-ba99nlnr.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232513-ba99nlnr/run-ba99nlnr.wandb new file mode 100644 index 0000000000000000000000000000000000000000..62fdddb2b996f2348345fd4b8c5f4776038245e9 GIT binary patch literal 67555 zcmeHQ349bq_NFHR;t0yI9^0;Kix()geasofT?KSe@jiC($WA65GDVUJGZPMtyAV(V z7!E}UD2Q?e6uCsRqAQmY5LZO~bLg%pXiz{SAfSl+-|J4gYck=g++p1P>11m9sPFsg z)vJ0{uafV-{Dv?8-D2Rq_bqm};F7p}t`jeK9DYx#E6tgfu4xWcQ&UyVmFn`Qrh6S; zmorVC$F*45#%@na^?AM7nc1Pe2YMY`2UpwjIqAe#cWR>soPmfc6HivZtWn$u&bkxd z^4@N@X^~E6+uPb(r3~_Ar={j|wnrZ$oikF?(>yK(UPW>`WUt4Ymge-P`r&^Lhu@{8 zc8mDJ{iSZwD6UP?mi=?4w`x~9b?w`>Dct4uq$IA?#+4*+d_oDwb!w%GZc&!6Y%Phh z3LkDrY{zxxqAK)IaBb2BP@x3nJo`EJj{19a%hDpgG+)Fgl5jA4Q~6zQro%51`%2dR zJDk*$P730v*V#8={Y@+2VJ)v5#o61~TWL8#e?~Z;NG7}yw=Xow)MX4#@myHyCc3(1m0YTgnR+4Lb6?`_@|d%>Ey~k1;J?+ z-GB%Ph(JZLY!=;G0fm52V5xyc3y`U zU}#9MP$(F>Sm>4QOT||dEPd$g1ci{6mh0DqEdAF)ZWt=mRY=W^2$75c)GRYNC@iD} z2W6W~ib9WIb~uoxg#@kguGjauL+ISC8;GilkRA*P>ArA8fJR?=TP;03;17UW&ATul zAP4*82LXX3;xz#{H1U;A{FR30dW1CK80v_PJFG>7L4iny091PfZ7{rArWR&+>Z_d+ zKYawAdIsrfK6Ue!rJZY)#0`4h+cE1Zc9k zOfT^JtN}%rKx4C$-CTz@Njvn$cIfBP&)fLVxV9j<%knA}*A)MoCrDT896caO$&z&K zpSlWv3Pi3BKXmF%q+83IdmvJ0(nu|Q_F!4lMk-H`qY#pnWYyDE5L}{LW<+Y;MGGE7 zq}tF(6&)y>lF+0zl_x0CNJ+`63`luI$;F7&*DIHfy9|(OOTNd}_qE>ohVERjisS!V z(fYocIJOBkygb1fjg=!=Qgu~H3L{n<*Prw2Re;r5WWR;g^i?~GZTxn`O1UiWtBD1b z{~y()K!Zjo(}qgE7R;5ZbQNT$LzKOYh)wwQyWhG2VrP>B7GmYE{pX1U{%b@`jY15d zYf_*AW~!GG8HJf5DsD#1v>t*e0%q;VL3D(^(Ocw(?Q4_xvT7j%)C!xB)BrT6Ck8YZ z2u*d0Zq}yRee#n76#%V0Ic$Me{`356`CSN_qpHr#Ryw-LKnKQOp*u=lZ3?tdMfQk_ z%BZ$Q>Asc@0ImZ$Vga{&&iG7z4+7^rCUAgYlfe$4bH+j^0d#5&jN9zT`TN}fT}Lw1 z23@DoP#3;&_!Ard17hbo9(GO0)&Y7R8a+kt8l2c-9ioHP@%UhoxF#9U`yCmMW9%}+ z?f-mwGQSVObDsb_qX9RK?jlc6Rodhk06Gvrm*`<_xhoyqu$}Y{#Xu=T@1b}k_Tj>H7dyt^1W=w!#$ih~v{=4s^RNW|C}1hW z5VX>(s{+^w{%quMf{(1wPM?Y`TR~xZR8e7V+5Nen_V)oa&m$8lG`EajoWMVB12m)E z-;?JpTo=Ck;a-j(U5mH^ zv`;l!(E_sWSY&bN=XHuM4b8$TYgJ+9Zz0B_^941jNxXRqTc zy8&N^DoU)b)shdt{##$b_YY(Sx>nyBL-6;vw&b6wBX|Mf#_4+jz#bQ6b`1gR#z7Ii ztn;&n2iZS*2mrp2%tiql053dH@(jm6iwK`+=)|Vs#tL$t;E^3M7=wsqC(MDfig>1f z?M*3w@gGUCg>lKKRr@%85@LLEBP=$g8)Ih&C_8A!%uq5;Jh^1i$?EL3TsnQQA5gxC zyh5QoDSw8Ie-2Uhunrxnb9}2OqtQ7Qd4e};SmDG@PamW?-0Uv!S>sxF=?6#)WTAz0 z>Fe(dPvBobq?rc1)gyXb>>Hb%2DF{Dg*MQ}0f*ugovbd#ZPNmyG63z1$s+6mztg+G zqP^Sf{EPJoHt^HPMtcBv)c8(^TS1Rba)=Tu+-G0@-MImPyAxTA;x^ze_;KbsPRCne zo`E>7l&e>Nqp~LuE|6q8l4s!U1+xv~X(wyE;D`6NCjdwez;5Ab?4ke(T`OgNzKV(l!Ktv+~e7*M~Iyk((YysO};1peg)LtW>j(HI#- z-a#Yp(mPMoyVoPitoG%MT}96f0^~0v%Pr)K)(>Bvz%M}LYjrfvp#4TNfCuyyiSA4d z^kGI+an&H!F{kZ69)A?jznrW_^mplkJ$;ipf7@z(LPO=?v&| z(l8kEu-f~nTg!uw0RvZ%_b3KN&i~lPFKs*w0BMbZ1mwU?M_JrvT)^pJpKE<~)ioQ2 z0|%YSdW(b79YY^a;Flo>bvYnqXbx~mgZB!+gzBbauR0U5ZjU5UVx60PrPqUBi~uIO zkPpzg->uuDg4G{q@^2#%^)O~Vz9Up^X#;ylK!ro54U<6yjI(5~L-eu+AV!~EboWT0 z;!3g+8-+nd@$jwFIDQ3kQD384h6d8GrV9CR(#s(YJ}?w4!$6Fcj|a1ci=%;$tH>u7 zA4R3pe&YC5jhqjlutDeDkddgNvB3yV#>=wkVrArp{~OS^5E$u7{?B5hc=@N$ZM+MN zsP#ECuGTP}=*xyA1vrTg@p~!*P_kPR9jtT9=Z=}aaXfI+jcm0zSy=Y3k_3J&aw6CL ztmlbPKN*A8aLgbtZhAIa=S7A^Rf;OJ2L^w;wfy3zfftc{h8_DJBXITecnAMJ($XNa z8cakr*upL##wDv%FvcQAc8f0729q7Uy?6o;b2a&|Ma-d{L$mk~niet8tQv|Nq{cz_ z_!cz^#3EVymo4e<@0|qHNMsw@GNa8D&sh7ijsFn2simbgCu5IlZ9ISnQN+e*zYheT?ih^u!e#Wevg+FyvO~K1XK=m!`=c zr|4mg<)1b6EoB-oq>wT+ZQtvC&Y>Uwoyc!Of*Qx%78^|s;2$7Ko|tB&I|h%$v{?p+dt&WMTcJuu5h{&S`~8-wd1uF|=W)Ru(QY}lC^iZYVvYevI8me*c`^Y;N*m9G-7F!#>d~TDC{}S11oU_wRqulzP zK?T0NGBtjx;0ss4dAy>+>Y-mWwEgd22EIIGzs1+)wUb6B@Y|aiU(m+tl`>?^K}|+f zFs9o>Rdld!Ci%j~_nrrgdC5L>27fTZKhr*MV-VJeLkuF3@Oc3Qjkvu=9hprA@ni_g|I(YuA!r zE!N7vnY*9kcL8gTCKA|Su#@%X>mzgYL}7)E#6$+LpPYGx&*c}+eP}5#*PZ-kF}HBU zGjnbHp61OQkX>KP=YTs|qGqRLJqd-3+`)b;mDPoO`>!b%ybj!5M~i{@4C7=uIH+Ph@z4Mc$0VE`A@9*VMu% zY!ueZggz4Jt-kdsu(5`9($T&>ck^37;GfCk7J;Qlze(qRLIRs+goS~|k@YZA019DS zjul?6XoHZ%O@TJZydFb#Ts3|LPh?-!+;iluxg`?|M|4y(4f8U2kFS8i*AZ~ zP8S(%DeQ5RS)1ba`S%TY7kIpZjInqeH?XivR%wn{Y|pHczC}9ZM&U3?ZL| zgVjepM|i35eW0@^nSf2xptHDaA@Z~A=Qcn8(u@y*(;G>V z#p$?NJ7A~PlW|TB8PuWqMOIyOz{2QCaehF?Qw$jM-JG-d!I2*Ut2dD;7OO>zHqEo~ zBazi+H~-DnUh9rnr5nZfvLQ5ZB_Wmb8Y;Q8vBDTC>nTsz(Vw>+Q zIs>s6|^WxkWeYa_PTM9sR>5p!Q}m9c{7EW)F=X`#i^wjdNQK|8+B~ z1njybDrrzOl9?92dL6%j|1mXC@qVeFZe% zPF}WXF4*^77XKX5+?4m$9Fv<9_P3|peQeG90W^-WmUV*t-vCzn8p)_T6af%SYxzBCnT`_jruzRQz^YB(>(l!#P}%M9+vD z4IX!TKyFp$+|6%%&wc$cXG^+|EVr01`SC~CGq3=dcg4e4tjrrK(D~=Z-PcTRky!JA zJBq*+WA&xV1DaTu2VJ;6|R4Cg> z@e84B8Mf$2tXWWoC+_+45#ax?`wfWBv_mP#gP{Km#&;BBQKp zK(_&qNs4(;?@8#D@*8O2Z)6R6en08%{_=SxPX3L!1n3^U=AUap1t@~3alRn}oSBzk z2O(=m;9a%P&Dpr5zGNL50z(7^W9Pib@&7^<#N#iuWC#oq)jahN>cC4o1V)E|!$3It z+{LPcQrDz#0_flYvfk3c=0kf9as0Bl4r*=zV((C6n*d6P-kf770mldK8jKINA9{Ow zYf!?2WP_!Ig~R7j${0dYGSY5-Z(tgiq)s1ltg!qS5bC;PJM;hRApcCv#W9wr~7 z(|AC48aMn}=;T+S9O8k$I+Ft^qDCeH1)@HkR_8iWw+n3#P76>ARY_bF~ zZs_Oz_%$eqc$}y%1z~g^H3%)CD5A2-7*+z4;42={&6<})P>*hG52E-x*^0eJg_T&k z9M1n<8&`$d<=6EGc42@nqK*%#@FwuM3Kh~&vId*pJN#nbj-U%4`P|aQ(2s`|+xYiU z7x8FQ-CK(q+6+{NgN{AxY0WVnCA;DLb=L9c{4UqMcn&BdmHf|AMnUgeX!=|L2X23{4lhkA3CJ4&r!W~@8zGK2iowHZI(8sfAz$Y1pdRgHtM<)iM>Cg zJU|^z`ba9-PzU-NQVkr?-5nk&J0H}MM!rM`^q@W>Sy;9-lm7_B;f{wh>(Ou+T49^n zLV3{VWf=01^~8oA(aSnuO?Yuc_X|KC8rgxy!;r_gm6PFI`H$oJh;7{U^bJd792MfB z7c(0ts-Qw38xeC7#VLOzf4LYGl1{#}6jJcPSja}S1r-vHh1RLWt0lQ3us^*M=%gRng&oZyy`veoW6>G>XK|U(j5XBfQ=T3}p?K�ftb} zUv)w*UDiFvIg7c!UkXCWAbTvKlsrAh%l{XJ5)Y|1s68>%d8$?gq~eZpUJa?}rw2eX zL)MA1n+~*mvD{t{!7$KJlNIAOGy-DHu& z==lDukB~Nx-j&Jkip$2NsS$oSY2OBf{-6Jy^8x9&`*$zu=G=0`szzI zejn;59zAUI`T;GSl)IcKI3#)*lcJyLA?XULK`Qh8174lc9Tb#9p0E_OdB!iWA?hbk zkS88SY6=Ph5uJ?tj2fcT;aeJl!&lkk)N@B{{rWo4&;aseBDg=l=opFX7Wc+MDbu$6R|_I z#!uuMfs|zW9L5TJ1iAGThs?gF>XxmSo^=yQDNM#;pJv#jp)2OYq0dKAOYuNN6Vwvu z>I88vP?I82Q_qx|V7sW3b&Phs`n%S>K}`|zw56t+zXPo)#eAj8z-sHb@7t0_wggmt{>NYs>< zO5|RlDeUnS@W5H~)V-em#vQkTrUsHp=sX=ZJn)ibt@$TWQt{|m6PMKSRRy9_6>4cZ zEh-QD=>V=LGA7>%q8db|prJBEHFVW-IHYAHsw%$ct?8=@gmvs8AL`0UM@x(j5T=~) z@Mb6LNN?x$U%YfT=xQ)|-qO{=cZxr@@uTCqGCY&!@$X}XS5R5>h6zPK^g?&7;9N#0 zt7|oQ)|d_tfU<^=nU=CPPkIv0Wqbye6`u}o78a{YMyIf@ zDtYI;tI8h+g*`&%priGxu@<)U-vNG7TwqmrX=Z;`Ggw5C(dpzVk->s%FKd=@CG}rh zd?2z%$viY;hR6y=jhxK!lR;(Pcub}_nlZDv#5!cCGkOCEr8A`l+3b&unEtFEbe2b6 zv2-?c$uP)>`5fvjo(!>B>kI@J({+Nfzlo$uUbmm zP`Z7#jh`A(E`oI~Bi;8{9|B_gCs|^NZRcz2o%{?G zTRhTudWg-CT$JMmf^$c$7`3chO|~`6PwPA}8w59$Bd=puYM8g>E8c_y?q{Lo;^Cvy zNpc`O+J{5Y(OJ{2E>-qA*+by3r|eA00nrWPNQou7@<~s?cD*@q(KTn} zC!9LSdb;;NCbsM`1mrh@BP(!J@tZz`J@oA^jbDW7i*G|bz10UYG=dSJzo?_p6jOh2 zWP&KO4r>2)>ExYxpufjCvKq~wp}&&1F^%bxxByKnaGHP6jUY|pg94WOVSTJ_%e>naPlwdrRf7;tXv-!aYI8Sg&3Y8+wvV?HpH+!0zKTJJ4 za|FonNsg?yWVqppHH9{QSz=pRiicw3GK@xGEJ`tI4#u>I3Rpzu!SJU&i#zse!6g-N z(ccHK#83F)Rmq2*w}U%aJ$hQk>Penjgiy(F*7*+;@-F2czd#NZ+2int|nyp!u^GT zzRX-LoGd(mwRni(ajkp$4-$CXQ(Rxwo|MFu+PIPgu7mlBZ&ku^%Q>!7E66S)%JP-1 zB~e!4!wre;xX#?E*YD|~ZUa}s0}Ht;M{)Kx_EuU>(4P^`=QyXmo!y@5i}*8A!huIL z_(ry$XK$OS`9j%&?0zXBUj)8vDQn#%r@f>7OsIQGju!H3*^zuM0ba{~zWwa5Z=jaq z&(Qq+bAkaZNQ{JXHG6ye8G)>RDPg}a6Yc>&cG}z86MR`YaKm(8W*ELaRTH$h%sw#) zhwzQz$Qw43sggr$A@#*OANn$SsCikLH` zQ&$1N)L>T)%moIrg=@2dX}Ou&bvke$-0r?mzpw}&80vJ(rE~gXot~!aRQ~fXEOols zT{!n$xYsirDden5J%1D`_1Lj0<%UBCpiX*SQ=np6re3YgQ1__T-c=Z+1^Q(~QqnZP zZ%B9gkFG*iFbmkt&FWtHp{tM{3~4@pMt78M7Y1Ck0e7}UTqZ=}@@4)y2Cmz66Bmwx zdp*mMyG>Y1pL%i~$M?8rNRHMk6bgotg_IPq3z3wR&Rv9l!HD3?7IL$-!8w{g0*$7> zHe^sRlqLYxxma1@ii#Ru!BDxuo{qh-aueyw6|5Y^Qn}PgnP(Tmy(V$wKC^Nqv&TT? zZnXEz4WY@)@MWi=IS_*BLL@^Ie5t{KnxGB#YdL6wxxuKAjmTYIJ zU&+;f9**@ZqU%?@emqP4?k|oMp_Q3TTbYva&qDp`#>$+N@#7!y<)_duzj;m(%gc}G z*!xMeGSAUgX55nJ;pLBQWmHH6Rm)ap&?mkktkd&!or+(W&QhoD4u9UeaIdLRR;K*$ z45(C7voe>Ro4EvWnMUJMzIQeYE?xh(_$XSL7icR}v~V84(lA!Wy+3mcR&F|7xzb7V zSt_?m3a4EN_j(b`Z@W%yhGZ#0??B1t8k{Y6=APyflel&lwaMJ_-OV;z(o0fC$Juxj1?hiho3Nt7?Z*P;WMZlo;7sI;Gc!n%PE~i&rZk4~xeqK%}=ohoJtYBz}=o{z@WcpGwwLUlGb7!b7mkOtOyPf=5_EyF(z`bBlBYuVnGa+*Qk{ zI(VvhWc~ts2gF5=0R|sfSr~&19I0s*#z5l`%G9a|0c6&dRS|+hrLvx2W2!t8WnY01*rqCl6lH1*LXt(98iNpB7=sY~wlN5S z@XZ&hijWMagH;wfsv-nd$SRCM2+oc&udD_k6I;je5OTyIMBla=gv_cqVi2Na0{b+p z08ihxDnh`g*&G~I5rXz%Dgh2Osd8J{(k%qls}zF0uSCchgAnXdHVb1Ag8V{eVKoT3 z&B7Rj=(mkQh>|8`!-{@e*g;{vPz*x!ZDSB}S_rvf5K=Ak=BkPim>RXRDzGFb`h}qJ zngVlGMTmY|R}4a`1w5Rh8hu+ZjAlJuF$ht7x?&KbY&=A;tK1g4ezPi75t6aXuCNL2 z7=$Pz<&Hr}wr&e~U8+2jvI*|02mySi`XI_wHivupMPJNp3C<$SleXx4qcrtQ c_&gIFM_Xs^C9cH<$$OJp!8fzG#axU32P6#})Bpeg literal 0 HcmV?d00001 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug.log new file mode 100644 index 0000000..a1c74e2 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug.log @@ -0,0 +1,15 @@ +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Configure stats pid to 12858 +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:25:51,420 INFO MainThread:12858 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug.log +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232551-j3ne0zj4/logs/debug-internal.log +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:25:51,421 INFO MainThread:12858 [wandb_init.py:init():597] wandb.init() called when a run is still active +2023-03-09 23:25:51,422 INFO MainThread:12858 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} +2023-03-09 23:25:58,846 WARNING MsgRouterThr:12858 [router.py:message_loop():77] message_loop has been closed diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml new file mode 100644 index 0000000..e69de29 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml new file mode 100644 index 0000000..2172dd0 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml @@ -0,0 +1,48 @@ +wandb_version: 1 + +_wandb: + desc: null + value: + cli_version: 0.13.11 + framework: torch + is_jupyter_run: false + is_kaggle_kernel: true + python_version: 3.7.11 + start_time: 1678375634.838134 + t: + 1: + - 1 + - 41 + - 55 + 2: + - 1 + - 41 + - 55 + 3: + - 23 + 4: 3.7.11 + 5: 0.13.11 + 8: + - 2 + - 5 +amp: + desc: null + value: false +batch_size: + desc: null + value: 1 +epochs: + desc: null + value: 5 +img_scale: + desc: null + value: 0.5 +learning_rate: + desc: null + value: 1.0e-05 +save_checkpoint: + desc: null + value: true +val_percent: + desc: null + value: 0.1 diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log new file mode 100644 index 0000000..8030533 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log @@ -0,0 +1,171 @@ +INFO: Starting training: + Epochs: 5 + Batch size: 1 + Learning rate: 1e-05 + Training size: 4580 + Validation size: 508 + Checkpoints: True + Device: cuda + Images scaling: 0.5 + Mixed Precision: False + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Epoch 1/5: 2%|██▌ | 81/4580 [03:29<3:07:14, 2.50s/img, loss (batch)=0.122]Exception ignored in: +Traceback (most recent call last): + File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1328, in __del__ + self._shutdown_workers() + File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1282, in _shutdown_workers + self._worker_result_queue.put((None, None)) + File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 87, in put + self._start_thread() + File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 165, in _start_thread + name='QueueFeederThread' + File "/opt/conda/lib/python3.7/threading.py", line 799, in __init__ + self._started = Event() + File "/opt/conda/lib/python3.7/threading.py", line 500, in __init__ + self._cond = Condition(Lock()) + File "/opt/conda/lib/python3.7/threading.py", line 228, in __init__ + except AttributeError: +KeyboardInterrupt: +Epoch 1/5: 2%|██▌ | 81/4580 [03:32<3:16:25, 2.62s/img, loss (batch)=0.122] +ERROR: Detected OutOfMemoryError! Enabling checkpointing to reduce memory usage, but this slows down training. Consider enabling AMP (--amp) for fast and memory efficient training +INFO: Creating dataset with 5088 examples +INFO: Scanning mask files to determine unique values + + + + + + + + + + + + 91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 4634/5088 [00:24<00:02, 193.05it/s] +Traceback (most recent call last): + File "train.py", line 221, in + amp=args.amp + File "train.py", line 122, in train_model + epoch_loss += loss.item() +KeyboardInterrupt +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 733, in next + item = self._items.popleft() +IndexError: pop from an empty deque +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "/code/UNet-master/2dunet-pytorch/utils/data_loading.py", line 55, in __init__ + total=len(self.ids) + File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1185, in __iter__ + for obj in iterable: + File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 737, in next + self._cond.wait(timeout) + File "/opt/conda/lib/python3.7/threading.py", line 296, in wait + waiter.acquire() +KeyboardInterrupt +During handling of the above exception, another exception occurred: +Traceback (most recent call last): + File "train.py", line 237, in + amp=args.amp + File "train.py", line 44, in train_model + dataset = CarvanaDataset(dir_img, dir_mask, img_scale) + File "/code/UNet-master/2dunet-pytorch/utils/data_loading.py", line 117, in __init__ + super().__init__(images_dir, mask_dir, scale, mask_suffix='_mask') + File "/code/UNet-master/2dunet-pytorch/utils/data_loading.py", line 55, in __init__ + total=len(self.ids) + File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 623, in __exit__ + self.terminate() + File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 548, in terminate + self._terminate() + File "/opt/conda/lib/python3.7/multiprocessing/util.py", line 224, in __call__ + res = self._callback(*self._args, **self._kwargs) + File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 617, in _terminate_pool + p.join() + File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 140, in join + res = self._popen.wait(timeout) + File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait + return self.poll(os.WNOHANG if timeout == 0.0 else 0) + File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll + pid, sts = os.waitpid(self.pid, flag) +KeyboardInterrupt \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt new file mode 100644 index 0000000..a70f071 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt @@ -0,0 +1,136 @@ +appdirs==1.4.4 +argon2-cffi-bindings==21.2.0 +argon2-cffi==21.3.0 +asttokens==2.0.8 +astunparse==1.6.3 +attrs==21.4.0 +backcall==0.2.0 +beautifulsoup4==4.10.0 +bleach==5.0.0 +brotlipy==0.7.0 +certifi==2021.10.8 +cffi==1.14.6 +chardet==4.0.0 +click==8.1.3 +conda-build==3.21.5 +conda-package-handling==1.7.3 +conda==4.10.3 +cryptography==35.0.0 +cycler==0.11.0 +debugpy==1.6.0 +decorator==5.1.0 +defusedxml==0.7.1 +dnspython==2.1.0 +docker-pycreds==0.4.0 +easydict==1.10 +entrypoints==0.4 +fastjsonschema==2.15.3 +filelock==3.3.1 +fonttools==4.38.0 +gitdb==4.0.10 +gitpython==3.1.31 +glob2==0.7 +idna==2.10 +imageio==2.25.0 +importlib-metadata==4.11.3 +importlib-resources==5.7.1 +ipykernel==6.13.0 +ipython-genutils==0.2.0 +ipython==7.29.0 +ipywidgets==8.0.4 +jedi==0.18.0 +jinja2==3.1.1 +json5==0.9.6 +jsonschema==4.4.0 +jupyter-client==7.3.0 +jupyter-core==4.10.0 +jupyterlab-pygments==0.2.2 +jupyterlab-server==1.2.0 +jupyterlab-widgets==3.0.5 +jupyterlab==2.2.5 +kiwisolver==1.4.4 +libarchive-c==2.9 +markupsafe==2.0.1 +matplotlib-inline==0.1.2 +matplotlib==3.5.3 +mindspore-cuda11-dev==2.0.0.dev20221108 +mindspore-dev==2.0.0.dev20230109 +minkowskiengine==0.5.4 +mistune==0.8.4 +mkl-fft==1.3.1 +mkl-random==1.2.2 +mkl-service==2.4.0 +ms-adapter==0.1.0 +msadapter==0.0.1a0 +nbclient==0.6.0 +nbconvert==6.5.0 +nbformat==5.3.0 +nest-asyncio==1.5.5 +networkx==2.6.3 +ninja==1.11.1 +notebook==6.4.11 +numpy==1.21.2 +olefile==0.46 +open3d-python==0.7.0.0 +opencv-python==4.6.0.66 +packaging==21.3 +pandas==1.3.5 +pandocfilters==1.5.0 +parso==0.8.2 +pathtools==0.1.2 +pexpect==4.8.0 +pickleshare==0.7.5 +pillow==8.4.0 +pip==21.0.1 +pkginfo==1.7.1 +prometheus-client==0.14.1 +prompt-toolkit==3.0.20 +protobuf==3.20.3 +psutil==5.8.0 +ptyprocess==0.7.0 +pycosat==0.6.3 +pycparser==2.20 +pygments==2.10.0 +pyopenssl==20.0.1 +pyparsing==3.0.8 +pypng==0.20220715.0 +pyrsistent==0.18.1 +pysocks==1.7.1 +python-dateutil==2.8.2 +python-etcd==0.4.5 +pytz==2021.3 +pywavelets==1.3.0 +pyyaml==6.0 +pyzmq==22.3.0 +requests==2.25.1 +ruamel-yaml-conda==0.15.100 +scikit-image==0.19.3 +scipy==1.7.3 +send2trash==1.8.0 +sentry-sdk==1.16.0 +setproctitle==1.3.2 +setuptools==58.0.4 +six==1.16.0 +smmap==5.0.0 +soupsieve==2.2.1 +tensorboardx==2.6 +terminado==0.13.3 +tifffile==2021.11.2 +tinycss2==1.1.1 +torch==1.10.0 +torchac==0.9.3 +torchelastic==0.2.0 +torchtext==0.11.0 +torchvision==0.11.1 +tornado==6.1 +tqdm==4.61.2 +traitlets==5.1.0 +typing-extensions==3.10.0.2 +urllib3==1.26.14 +wandb==0.13.11 +wcwidth==0.2.5 +webencodings==0.5.1 +wheel==0.36.2 +widgetsnbextension==4.0.5 +xlrd==1.2.0 +zipp==3.8.0 \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json new file mode 100644 index 0000000..0fded40 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json @@ -0,0 +1,443 @@ +{ + "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid", + "python": "3.7.11", + "heartbeatAt": "2023-03-09T15:27:21.219862", + "startedAt": "2023-03-09T15:27:14.806065", + "docker": null, + "cuda": null, + "args": [], + "state": "running", + "program": "train.py", + "codePath": "train.py", + "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0", + "username": "root", + "executable": "/opt/conda/bin/python", + "cpu_count": 40, + "cpu_count_logical": 80, + "cpu_freq": { + "current": 2502.0773624999993, + "min": 0.0, + "max": 0.0 + }, + "cpu_freq_per_core": [ + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.421, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.46, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.84, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.951, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.019, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.524, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.222, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.672, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.854, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.5, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.023, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.192, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.8, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.343, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.078, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.029, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.541, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.581, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.509, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.341, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.858, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.388, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.575, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.535, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.27, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.294, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.36, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.768, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.017, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.29, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.82, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.378, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.616, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.261, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.368, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.645, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.621, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.022, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.289, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.416, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.087, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.772, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.081, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.003, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.992, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.015, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.871, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2507.194, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.997, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.001, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.906, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2506.009, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.082, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.269, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2508.067, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.36, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.23, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.0, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.176, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2501.354, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.515, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2503.134, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.134, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.975, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.278, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2500.387, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.299, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.099, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2498.784, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.785, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2505.195, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.921, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.207, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.31, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2499.904, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2502.18, + "min": 0.0, + "max": 0.0 + }, + { + "current": 2504.924, + "min": 0.0, + "max": 0.0 + } + ], + "disk": { + "total": 878.6640281677246, + "used": 283.64785385131836 + }, + "gpu": "Tesla T4", + "gpu_count": 2, + "gpu_devices": [ + { + "name": "Tesla T4", + "memory_total": 15843721216 + }, + { + "name": "Tesla T4", + "memory_total": 15843721216 + } + ], + "memory": { + "total": 376.5794219970703 + } +} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json new file mode 100644 index 0000000..c2b81d3 --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json @@ -0,0 +1 @@ +{"train loss": 0.12218321859836578, "step": 80, "epoch": 1, "_timestamp": 1678375851.5881917, "_runtime": 216.75005769729614, "_step": 79, "_wandb": {"runtime": 243}} \ No newline at end of file diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug-internal.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug-internal.log new file mode 100644 index 0000000..4c52d3a --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug-internal.log @@ -0,0 +1,778 @@ +2023-03-09 23:27:14,841 INFO StreamThr :13248 [internal.py:wandb_internal():90] W&B internal server running at pid: 13248, started at: 2023-03-09 23:27:14.838951 +2023-03-09 23:27:14,842 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status +2023-03-09 23:27:14,843 INFO WriterThread:13248 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb +2023-03-09 23:27:14,848 DEBUG SenderThread:13248 [sender.py:send():336] send: header +2023-03-09 23:27:14,848 DEBUG SenderThread:13248 [sender.py:send():336] send: run +2023-03-09 23:27:14,863 INFO SenderThread:13248 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/76baujih +2023-03-09 23:27:16,098 INFO SenderThread:13248 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files +2023-03-09 23:27:16,098 INFO SenderThread:13248 [sender.py:_start_run_threads():1081] run started: 76baujih with start time 1678375634.838134 +2023-03-09 23:27:16,098 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:16,099 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: check_version +2023-03-09 23:27:16,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:16,162 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: check_version +2023-03-09 23:27:17,102 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:21,099 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:21,163 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:21,171 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: run_start +2023-03-09 23:27:21,178 DEBUG HandlerThread:13248 [system_info.py:__init__():31] System info init +2023-03-09 23:27:21,178 DEBUG HandlerThread:13248 [system_info.py:__init__():46] System info init done +2023-03-09 23:27:21,178 INFO HandlerThread:13248 [system_monitor.py:start():183] Starting system monitor +2023-03-09 23:27:21,179 INFO SystemMonitor:13248 [system_monitor.py:_start():147] Starting system asset monitoring threads +2023-03-09 23:27:21,179 INFO HandlerThread:13248 [system_monitor.py:probe():204] Collecting system info +2023-03-09 23:27:21,179 INFO SystemMonitor:13248 [interfaces.py:start():187] Started cpu monitoring +2023-03-09 23:27:21,180 INFO SystemMonitor:13248 [interfaces.py:start():187] Started disk monitoring +2023-03-09 23:27:21,181 INFO SystemMonitor:13248 [interfaces.py:start():187] Started gpu monitoring +2023-03-09 23:27:21,181 INFO SystemMonitor:13248 [interfaces.py:start():187] Started memory monitoring +2023-03-09 23:27:21,182 INFO SystemMonitor:13248 [interfaces.py:start():187] Started network monitoring +2023-03-09 23:27:21,219 DEBUG HandlerThread:13248 [system_info.py:probe():195] Probing system +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [git.py:repo():40] git repository is invalid +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_info.py:probe():240] Probing system done +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:27:21.219862', 'startedAt': '2023-03-09T15:27:14.806065', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2502.0773624999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.421, 'min': 0.0, 'max': 0.0}, {'current': 2500.46, 'min': 0.0, 'max': 0.0}, {'current': 2501.84, 'min': 0.0, 'max': 0.0}, {'current': 2500.951, 'min': 0.0, 'max': 0.0}, {'current': 2501.019, 'min': 0.0, 'max': 0.0}, {'current': 2499.524, 'min': 0.0, 'max': 0.0}, {'current': 2502.222, 'min': 0.0, 'max': 0.0}, {'current': 2500.672, 'min': 0.0, 'max': 0.0}, {'current': 2499.854, 'min': 0.0, 'max': 0.0}, {'current': 2506.5, 'min': 0.0, 'max': 0.0}, {'current': 2500.023, 'min': 0.0, 'max': 0.0}, {'current': 2499.192, 'min': 0.0, 'max': 0.0}, {'current': 2499.8, 'min': 0.0, 'max': 0.0}, {'current': 2500.343, 'min': 0.0, 'max': 0.0}, {'current': 2499.078, 'min': 0.0, 'max': 0.0}, {'current': 2500.029, 'min': 0.0, 'max': 0.0}, {'current': 2501.541, 'min': 0.0, 'max': 0.0}, {'current': 2498.581, 'min': 0.0, 'max': 0.0}, {'current': 2499.509, 'min': 0.0, 'max': 0.0}, {'current': 2500.341, 'min': 0.0, 'max': 0.0}, {'current': 2499.858, 'min': 0.0, 'max': 0.0}, {'current': 2500.388, 'min': 0.0, 'max': 0.0}, {'current': 2501.575, 'min': 0.0, 'max': 0.0}, {'current': 2500.535, 'min': 0.0, 'max': 0.0}, {'current': 2501.27, 'min': 0.0, 'max': 0.0}, {'current': 2499.294, 'min': 0.0, 'max': 0.0}, {'current': 2500.36, 'min': 0.0, 'max': 0.0}, {'current': 2500.768, 'min': 0.0, 'max': 0.0}, {'current': 2499.017, 'min': 0.0, 'max': 0.0}, {'current': 2505.29, 'min': 0.0, 'max': 0.0}, {'current': 2499.82, 'min': 0.0, 'max': 0.0}, {'current': 2500.378, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2499.261, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2507.621, 'min': 0.0, 'max': 0.0}, {'current': 2506.022, 'min': 0.0, 'max': 0.0}, {'current': 2505.289, 'min': 0.0, 'max': 0.0}, {'current': 2507.416, 'min': 0.0, 'max': 0.0}, {'current': 2506.087, 'min': 0.0, 'max': 0.0}, {'current': 2506.772, 'min': 0.0, 'max': 0.0}, {'current': 2506.081, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2506.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.015, 'min': 0.0, 'max': 0.0}, {'current': 2503.871, 'min': 0.0, 'max': 0.0}, {'current': 2507.194, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2506.906, 'min': 0.0, 'max': 0.0}, {'current': 2506.009, 'min': 0.0, 'max': 0.0}, {'current': 2501.082, 'min': 0.0, 'max': 0.0}, {'current': 2505.269, 'min': 0.0, 'max': 0.0}, {'current': 2508.067, 'min': 0.0, 'max': 0.0}, {'current': 2502.36, 'min': 0.0, 'max': 0.0}, {'current': 2505.23, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.176, 'min': 0.0, 'max': 0.0}, {'current': 2501.354, 'min': 0.0, 'max': 0.0}, {'current': 2503.515, 'min': 0.0, 'max': 0.0}, {'current': 2503.134, 'min': 0.0, 'max': 0.0}, {'current': 2500.134, 'min': 0.0, 'max': 0.0}, {'current': 2500.975, 'min': 0.0, 'max': 0.0}, {'current': 2504.278, 'min': 0.0, 'max': 0.0}, {'current': 2500.387, 'min': 0.0, 'max': 0.0}, {'current': 2504.299, 'min': 0.0, 'max': 0.0}, {'current': 2505.099, 'min': 0.0, 'max': 0.0}, {'current': 2498.784, 'min': 0.0, 'max': 0.0}, {'current': 2504.785, 'min': 0.0, 'max': 0.0}, {'current': 2505.195, 'min': 0.0, 'max': 0.0}, {'current': 2504.921, 'min': 0.0, 'max': 0.0}, {'current': 2499.207, 'min': 0.0, 'max': 0.0}, {'current': 2504.31, 'min': 0.0, 'max': 0.0}, {'current': 2499.904, 'min': 0.0, 'max': 0.0}, {'current': 2502.18, 'min': 0.0, 'max': 0.0}, {'current': 2504.924, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64785385131836}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}} +2023-03-09 23:27:21,254 INFO HandlerThread:13248 [system_monitor.py:probe():214] Finished collecting system info +2023-03-09 23:27:21,254 INFO HandlerThread:13248 [system_monitor.py:probe():217] Publishing system info +2023-03-09 23:27:21,254 DEBUG HandlerThread:13248 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment +2023-03-09 23:27:21,274 DEBUG HandlerThread:13248 [system_info.py:_save_pip():67] Saving pip packages done +2023-03-09 23:27:21,274 DEBUG HandlerThread:13248 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment +2023-03-09 23:27:21,996 DEBUG HandlerThread:13248 [system_info.py:_save_conda():86] Saving conda packages done +2023-03-09 23:27:22,031 INFO HandlerThread:13248 [system_monitor.py:probe():219] Finished publishing system info +2023-03-09 23:27:22,034 DEBUG SenderThread:13248 [sender.py:send():336] send: files +2023-03-09 23:27:22,034 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now +2023-03-09 23:27:22,079 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: stop_status +2023-03-09 23:27:22,085 DEBUG SenderThread:13248 [sender.py:send():336] send: telemetry +2023-03-09 23:27:22,095 DEBUG SenderThread:13248 [sender.py:send():336] send: config +2023-03-09 23:27:22,100 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: stop_status +2023-03-09 23:27:22,309 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json +2023-03-09 23:27:22,319 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt +2023-03-09 23:27:22,340 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml +2023-03-09 23:27:23,312 INFO Thread-13 :13248 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:23,673 INFO wandb-upload_0:13248 [upload_job.py:push():138] Uploaded file /tmp/tmpsqy0gtaqwandb/hoj8mklj-wandb-metadata.json +2023-03-09 23:27:25,318 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:27,115 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:32,116 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:35,417 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:35,424 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:35,426 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:35,426 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:35,446 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:36,417 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:37,043 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:37,043 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:37,420 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:37,570 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:37,859 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:37,860 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:37,860 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:37,943 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:38,421 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:39,423 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:40,294 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:40,295 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:40,295 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:40,309 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:40,424 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:41,427 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:42,046 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:42,740 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:42,741 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:42,741 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:42,742 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:42,759 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:43,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:43,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:45,177 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:45,178 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:45,179 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:45,252 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:45,432 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:47,049 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:47,436 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:47,614 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:47,615 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:47,615 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:47,653 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:48,437 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:48,659 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:49,440 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:49,440 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml +2023-03-09 23:27:50,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:50,052 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:50,053 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:50,103 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:50,441 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:51,443 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:52,049 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:52,491 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:52,492 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:52,492 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:52,519 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:53,453 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:53,453 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:54,526 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:54,935 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:54,937 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:54,937 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:55,020 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:55,454 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:57,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:27:57,388 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:57,389 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:57,389 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:57,403 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:27:57,458 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:57,458 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:27:59,461 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:27:59,837 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:27:59,838 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:27:59,838 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:27:59,839 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:27:59,879 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:00,462 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:01,465 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:02,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:02,294 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:02,295 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:02,295 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:02,331 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:02,465 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:03,468 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:04,751 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:04,752 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:04,753 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:04,788 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:05,471 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:05,471 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:05,789 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:07,051 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:07,208 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:07,209 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:07,209 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:07,244 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:07,473 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:09,477 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:09,669 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:09,670 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:09,671 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:09,689 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:10,478 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:11,480 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:11,704 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:12,052 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:12,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:12,132 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:12,132 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:12,168 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:12,481 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:13,484 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:14,597 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:14,598 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:14,598 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:14,664 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:15,487 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:15,487 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:17,052 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:17,062 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:17,063 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:17,063 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:17,063 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:17,097 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:17,488 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:19,492 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:19,533 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:19,534 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:19,534 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:19,597 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:20,493 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:21,182 DEBUG SystemMonitor:13248 [system_monitor.py:_start():161] Starting system metrics aggregation loop +2023-03-09 23:28:21,184 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:28:21,495 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:22,005 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:22,006 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:22,006 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:22,038 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:22,053 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:22,496 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:23,041 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:23,499 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:24,483 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:24,484 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:24,484 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:24,523 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:24,523 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:25,525 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:26,957 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:26,958 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:26,959 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:27,020 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:27,054 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:27,527 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:29,021 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:29,432 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:29,432 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:29,433 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:29,474 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:29,530 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:29,530 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:31,533 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:31,907 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:31,909 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:31,909 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:31,930 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:32,054 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:32,534 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:33,537 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:34,384 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:34,384 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:34,385 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:34,385 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:34,421 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:34,538 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:35,540 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:36,865 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:36,866 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:36,866 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:36,885 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:37,055 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:37,543 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:37,544 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:39,344 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:39,345 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:39,345 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:39,417 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:39,418 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:39,545 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:41,548 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:41,826 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:41,827 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:41,827 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:41,846 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:42,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:42,549 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:43,552 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:44,313 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:44,314 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:44,315 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:44,373 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:44,552 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:45,374 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:45,555 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:46,801 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:46,802 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:46,802 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:46,840 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:47,056 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:47,558 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:47,559 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:49,285 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:49,286 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:49,287 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:49,315 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:49,560 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:51,186 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:28:51,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:51,563 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:51,772 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:51,773 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:51,773 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:51,807 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:52,057 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:52,564 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:53,566 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:54,257 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:54,258 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:54,258 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:54,283 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:54,567 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:55,569 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:56,292 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:28:56,746 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:56,747 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:56,748 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:56,782 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:57,058 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:28:57,573 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:28:57,574 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:28:59,233 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:28:59,234 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:28:59,234 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:28:59,271 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:28:59,574 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:01,578 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:01,728 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:01,730 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:01,730 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:01,730 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:01,757 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:02,058 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:02,579 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:03,582 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:04,215 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:04,216 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:04,216 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:04,250 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:04,583 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:05,585 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:06,717 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:06,718 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:06,719 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:06,763 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:06,763 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:07,059 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:07,587 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:07,587 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:09,213 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:09,214 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:09,214 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:09,251 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:09,588 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:11,592 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:11,700 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:11,701 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:11,702 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:11,730 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:12,060 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:12,593 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:12,735 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:13,596 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:14,200 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:14,202 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:14,202 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:14,246 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:14,597 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:15,600 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:16,700 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:16,701 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:16,702 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:16,752 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:17,060 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:17,603 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:17,603 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:17,754 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:19,199 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:19,200 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:19,200 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:19,236 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:19,605 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:21,187 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:29:21,608 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:21,695 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:21,696 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:21,697 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:21,743 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:22,061 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:22,609 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:23,612 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:23,744 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:24,192 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:24,193 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:24,193 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:24,210 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:24,613 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:25,615 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:26,688 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:26,690 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:26,690 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:26,785 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:27,061 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:27,618 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:27,619 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:28,786 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:29,184 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:29,185 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:29,185 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:29,226 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:29,620 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:31,624 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:31,680 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:31,680 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:31,681 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:31,719 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:32,062 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:32,625 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:33,627 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:34,165 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:34,166 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:34,167 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:34,167 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:34,218 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:34,628 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:35,630 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:36,655 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:36,656 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:36,656 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:36,724 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:37,063 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:37,633 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:37,633 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:39,151 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:39,152 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:39,153 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:39,211 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:39,211 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:39,635 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:41,638 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:41,647 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:41,648 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:41,649 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:41,673 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:42,064 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:42,639 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:43,641 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:44,144 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:44,145 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:44,145 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:44,186 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:44,642 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:45,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:45,645 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:46,642 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:46,643 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:46,643 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:46,670 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:46,670 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:47,064 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:47,673 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:49,137 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:49,138 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:49,138 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:49,174 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:49,674 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:51,175 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:51,188 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:29:51,632 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:51,633 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:51,633 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:51,661 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:51,677 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:51,678 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:52,065 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:53,680 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:54,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:54,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:54,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:54,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:54,681 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:55,684 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:56,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:56,625 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:56,626 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:56,626 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:29:56,649 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:56,685 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:29:57,066 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:29:57,687 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:29:59,122 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:29:59,123 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:29:59,123 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:29:59,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:29:59,689 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:01,621 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:01,622 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:01,623 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:01,635 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:01,636 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:01,692 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:01,692 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:02,066 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:03,695 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:04,122 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:04,123 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:04,124 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:04,160 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:04,696 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:05,699 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:06,623 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:06,624 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:06,625 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:06,645 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:06,645 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:06,699 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:07,067 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:07,702 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:09,127 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:09,128 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:09,128 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:09,157 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:09,703 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:11,627 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:11,628 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:11,628 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:11,686 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:11,687 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:11,707 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:11,707 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:12,067 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:13,710 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:14,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:14,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:14,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:14,199 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:14,711 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:15,713 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:16,630 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:16,631 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:16,631 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:16,653 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:16,714 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:17,068 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:17,664 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:17,717 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:19,129 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:19,130 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:19,130 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:19,164 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:19,718 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:21,209 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:30:21,629 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:21,630 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:21,631 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:21,773 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:21,773 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:21,774 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:22,069 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:22,774 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:23,777 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:24,127 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:24,128 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:24,129 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:24,166 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:24,777 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:25,780 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:26,623 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:26,624 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:26,625 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:26,682 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:26,781 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:27,069 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:27,783 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:28,683 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:29,119 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:29,120 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:29,120 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:29,161 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:29,786 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:29,787 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:31,612 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:31,613 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:31,613 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:31,675 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:31,788 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:32,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:33,795 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:34,110 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:34,111 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:34,111 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:34,111 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:34,141 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:34,795 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:35,798 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:36,604 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:36,605 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:36,605 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:36,628 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:36,799 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:37,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:37,815 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:39,101 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:39,102 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:39,102 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:39,178 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:39,180 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:39,831 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:39,831 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:41,602 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:41,603 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:41,603 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:41,684 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:41,832 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:42,070 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:43,836 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:44,096 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:44,097 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:44,097 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:44,120 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:44,836 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:45,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:45,839 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:46,593 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:46,594 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:46,594 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:46,628 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:46,840 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:47,071 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:47,843 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:49,090 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:49,091 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:49,091 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:49,121 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:49,844 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:49,844 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:51,124 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:51,191 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:30:51,588 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: partial_history +2023-03-09 23:30:51,589 DEBUG SenderThread:13248 [sender.py:send():336] send: history +2023-03-09 23:30:51,590 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: summary_record +2023-03-09 23:30:51,642 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:30:51,845 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:30:52,072 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:53,849 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:55,891 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:30:56,254 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:30:57,130 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:30:57,895 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:01,310 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:01,913 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:02,208 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:04,119 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:06,117 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:06,339 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:07,212 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:08,208 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:10,212 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:11,425 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:12,214 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:12,216 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:14,513 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:16,426 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:16,617 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:17,217 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:18,430 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:20,512 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:21,210 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:31:21,814 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:22,218 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: keepalive +2023-03-09 23:31:22,516 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:24,576 DEBUG SenderThread:13248 [sender.py:send():336] send: exit +2023-03-09 23:31:24,577 INFO SenderThread:13248 [sender.py:send_exit():559] handling exit code: 255 +2023-03-09 23:31:24,577 INFO SenderThread:13248 [sender.py:send_exit():561] handling runtime: 243 +2023-03-09 23:31:24,608 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:24,708 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:31:24,913 INFO SenderThread:13248 [sender.py:send_exit():567] send defer +2023-03-09 23:31:25,019 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,108 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 0 +2023-03-09 23:31:25,118 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,118 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 0 +2023-03-09 23:31:25,118 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 1 +2023-03-09 23:31:25,119 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,119 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 1 +2023-03-09 23:31:25,119 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,119 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 1 +2023-03-09 23:31:25,119 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 2 +2023-03-09 23:31:25,120 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,120 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 2 +2023-03-09 23:31:25,120 INFO HandlerThread:13248 [system_monitor.py:finish():193] Stopping system monitor +2023-03-09 23:31:25,121 DEBUG SystemMonitor:13248 [system_monitor.py:_start():168] Finished system metrics aggregation loop +2023-03-09 23:31:25,121 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined cpu monitor +2023-03-09 23:31:25,121 DEBUG SystemMonitor:13248 [system_monitor.py:_start():172] Publishing last batch of metrics +2023-03-09 23:31:25,124 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined disk monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined gpu monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined memory monitor +2023-03-09 23:31:25,187 INFO HandlerThread:13248 [interfaces.py:finish():199] Joined network monitor +2023-03-09 23:31:25,188 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,188 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 2 +2023-03-09 23:31:25,188 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 3 +2023-03-09 23:31:25,188 DEBUG SenderThread:13248 [sender.py:send():336] send: stats +2023-03-09 23:31:25,189 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,189 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 3 +2023-03-09 23:31:25,189 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,189 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 3 +2023-03-09 23:31:25,189 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 4 +2023-03-09 23:31:25,190 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,190 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 4 +2023-03-09 23:31:25,190 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,190 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 4 +2023-03-09 23:31:25,190 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 5 +2023-03-09 23:31:25,190 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,190 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 5 +2023-03-09 23:31:25,190 DEBUG SenderThread:13248 [sender.py:send():336] send: summary +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end +2023-03-09 23:31:25,221 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 5 +2023-03-09 23:31:25,221 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 6 +2023-03-09 23:31:25,221 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,222 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 6 +2023-03-09 23:31:25,222 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 6 +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 7 +2023-03-09 23:31:25,222 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: status_report +2023-03-09 23:31:25,222 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,222 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 7 +2023-03-09 23:31:25,222 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,222 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 7 +2023-03-09 23:31:25,577 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:25,582 INFO Thread-13 :13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:31:25,930 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 8 +2023-03-09 23:31:25,930 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:25,931 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,931 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 8 +2023-03-09 23:31:25,931 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,932 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 8 +2023-03-09 23:31:25,932 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 9 +2023-03-09 23:31:25,932 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:25,932 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 9 +2023-03-09 23:31:25,933 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:25,933 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 9 +2023-03-09 23:31:25,933 INFO SenderThread:13248 [dir_watcher.py:finish():365] shutting down directory watcher +2023-03-09 23:31:26,578 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:26,584 INFO SenderThread:13248 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:26,584 INFO SenderThread:13248 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/conda-environment.yaml conda-environment.yaml +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log output.log +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt requirements.txt +2023-03-09 23:31:26,585 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml config.yaml +2023-03-09 23:31:26,743 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json wandb-summary.json +2023-03-09 23:31:26,800 INFO SenderThread:13248 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-metadata.json wandb-metadata.json +2023-03-09 23:31:26,806 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 10 +2023-03-09 23:31:26,806 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:26,831 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:26,836 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 10 +2023-03-09 23:31:26,853 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:26,858 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 10 +2023-03-09 23:31:26,899 INFO SenderThread:13248 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:31:27,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:27,640 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:27,835 INFO wandb-upload_1:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/output.log +2023-03-09 23:31:28,157 INFO wandb-upload_3:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/config.yaml +2023-03-09 23:31:28,167 INFO wandb-upload_2:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/requirements.txt +2023-03-09 23:31:28,261 INFO wandb-upload_4:13248 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/files/wandb-summary.json +2023-03-09 23:31:28,461 INFO Thread-12 :13248 [sender.py:transition_state():587] send defer: 11 +2023-03-09 23:31:28,461 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:28,461 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 11 +2023-03-09 23:31:28,461 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:28,461 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 11 +2023-03-09 23:31:28,461 INFO SenderThread:13248 [file_pusher.py:join():169] waiting for file pusher +2023-03-09 23:31:28,461 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 12 +2023-03-09 23:31:28,462 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:28,462 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 12 +2023-03-09 23:31:28,462 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:28,462 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 12 +2023-03-09 23:31:28,624 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:31,144 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 13 +2023-03-09 23:31:31,146 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:31,146 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:31,147 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 13 +2023-03-09 23:31:31,148 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:31,148 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 13 +2023-03-09 23:31:31,148 INFO SenderThread:13248 [sender.py:transition_state():587] send defer: 14 +2023-03-09 23:31:31,148 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: defer +2023-03-09 23:31:31,148 INFO HandlerThread:13248 [handler.py:handle_request_defer():170] handle defer: 14 +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send():336] send: final +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send():336] send: footer +2023-03-09 23:31:31,149 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: defer +2023-03-09 23:31:31,149 INFO SenderThread:13248 [sender.py:send_request_defer():583] handle sender defer: 14 +2023-03-09 23:31:31,150 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: poll_exit +2023-03-09 23:31:31,150 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: poll_exit +2023-03-09 23:31:31,150 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: server_info +2023-03-09 23:31:31,151 DEBUG SenderThread:13248 [sender.py:send_request():363] send_request: server_info +2023-03-09 23:31:31,151 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: get_summary +2023-03-09 23:31:31,187 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: sampled_history +2023-03-09 23:31:31,868 INFO MainThread:13248 [wandb_run.py:_footer_history_summary_info():3429] rendering history +2023-03-09 23:31:31,869 INFO MainThread:13248 [wandb_run.py:_footer_history_summary_info():3461] rendering summary +2023-03-09 23:31:31,869 INFO MainThread:13248 [wandb_run.py:_footer_sync_info():3387] logging synced files +2023-03-09 23:31:31,870 DEBUG HandlerThread:13248 [handler.py:handle_request():144] handle_request: shutdown +2023-03-09 23:31:31,870 INFO HandlerThread:13248 [handler.py:finish():842] shutting down handler +2023-03-09 23:31:32,151 INFO WriterThread:13248 [datastore.py:close():298] close: /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb +2023-03-09 23:31:32,591 WARNING StreamThr :13248 [internal.py:is_dead():416] Internal process exiting, parent pid 13116 disappeared +2023-03-09 23:31:32,591 ERROR StreamThr :13248 [internal.py:wandb_internal():153] Internal process shutdown. +2023-03-09 23:31:32,868 INFO SenderThread:13248 [sender.py:finish():1504] shutting down sender +2023-03-09 23:31:32,868 INFO SenderThread:13248 [file_pusher.py:finish():164] shutting down file pusher +2023-03-09 23:31:32,868 INFO SenderThread:13248 [file_pusher.py:join():169] waiting for file pusher diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug.log b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug.log new file mode 100644 index 0000000..79c403d --- /dev/null +++ b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug.log @@ -0,0 +1,27 @@ +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Configure stats pid to 13116 +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-pytorch/wandb/settings +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug.log +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-pytorch/wandb/run-20230309_232714-76baujih/logs/debug-internal.log +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():546] calling init triggers +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():553] wandb.init called with sweep_config: {} +config: {} +2023-03-09 23:27:14,832 INFO MainThread:13116 [wandb_init.py:init():602] starting backend +2023-03-09 23:27:14,833 INFO MainThread:13116 [wandb_init.py:init():606] setting up manager +2023-03-09 23:27:14,835 INFO MainThread:13116 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn +2023-03-09 23:27:14,837 INFO MainThread:13116 [wandb_init.py:init():613] backend started and connected +2023-03-09 23:27:14,841 INFO MainThread:13116 [wandb_init.py:init():701] updated telemetry +2023-03-09 23:27:14,842 INFO MainThread:13116 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout +2023-03-09 23:27:16,098 INFO MainThread:13116 [wandb_run.py:_on_init():2131] communicating current version +2023-03-09 23:27:21,163 INFO MainThread:13116 [wandb_run.py:_on_init():2140] got version response +2023-03-09 23:27:21,164 INFO MainThread:13116 [wandb_init.py:init():789] starting run threads in backend +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_console_start():2112] atexit reg +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():2032] Wrapping output streams. +2023-03-09 23:27:22,042 INFO MainThread:13116 [wandb_run.py:_redirect():2057] Redirects installed. +2023-03-09 23:27:22,043 INFO MainThread:13116 [wandb_init.py:init():831] run started, returning control to user process +2023-03-09 23:27:22,043 INFO MainThread:13116 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'val_percent': 0.1, 'save_checkpoint': True, 'img_scale': 0.5, 'amp': False} diff --git a/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb b/official/cv/unet2d/2dunet-pytorch/wandb/run-20230309_232714-76baujih/run-76baujih.wandb new file mode 100644 index 0000000000000000000000000000000000000000..dcbcda50bc343d0f6a3e65905a3cd9fd594fcd51 GIT binary patch literal 153296 zcmeEP34jw-ww)vmV#E#CiOv{tP~1XlPg>M*9d+DB9oO%)y#(mcbhq7US7rurLj*wq z0TIC+1r<~fQ2{q>7Z4Q{#RWG67X(Gc{hwP^se1J)=`>Hk#KHeZp{r{<_q<#8-FNPN z<)`g==zFvO(EP>He;I!WbP5a%bd5wStCH1Ki9|eEqo!0eWlGs?n+vnp)K> zad+SkPjn84I~jUaWlOrY&%Ud_z4$qt9UK^l7e2dd=jP_>hQ_`Z$~8(|-@$>)0`8R} z9sarh@u^JLt;1V}J64=uSzl8%Fc7@>-|6mss(RHVbh%1N#Wht*CXzKZTC%Dd|EH?e zda7#wOl5Q5`2O+1fzF*?{`$^|9k*Ha(ksJ4Hw5+!cj^>q4F+b00+G|baZq{L+!GUn+aK}_*Lv^3#fq{S)-XTBzJS2R^-@WT#LYX7(v-Y&ewd6jh)jj5*URDEV(AcXf1 z?;PH`x$?YJMRlK4b>GH@S~^gcX=+J@w+(Mz+pl*;b9H4Mu7O|HV3SZ~zeZfJS7lu@ ze%KDGc-~{tw#$C#7YKCfn4S^rvPGviO*0LjH+qN29l@iIKk|g~=*gMNrVO4unrW)6 zrB^5qN9})(FkiYn``2hxa}FI!2t)}1`>^C3I@X3FN>I>I3ybVkPO)#}oUCezxO4KU zm36hScx^+yeO!wtoP&osh&5YxJ%v!qUik3Td9~Hq$Dtoqx71WRN007T**n!7C2Z*( zqftUPd+Kqu7o=*UCpM+3YnyTF^5~I-ws|3s$_xc56e2}NODg}E0>jWMcXeb$uTM3+yhVv0CmO3mmM+^-e~hK!vf*m z!(nr%M(Y}y;gPyXFt z)2nombR(e{nyzUvRhE*vmI&__?o`p#Qcu5yi^~`$2_>OOM#7NdN&GHdu)@B2yvrX> z-rd+@57;oA{w&a?bEl83@4$;c&JUD^5xo{ua$0OZ6Y6Mp)9vqlAOP0|jJRUfaeWD? zEm))`&6_8 zJd77Ws^x{)I3oPkxd3O>kr0;z%sA=8H%D~?Qro6i z(znn;YU#tP?h8cT_D6p2Uq7tqg1-Jyl_7khr&=y2%2fk|S7KUD7cYuc$ESANhbZ4A zH{}yjQi~_`gcetlnjBXc%9}2^a7d%eF++{VRmtE(8iUN6j2n7FR;5HjHpfslq)|Zm z(6cXft0qHkXBl#(N%_UYZ+|aXDu#rUXDlhl+hxgwEI|66r)&EWR@<{!Ex&o{m!*W2 z=CU4GQB+eNmrT}6c~~!m)fq3}I&~CCzeAVd{=fz!$uML?Lc%9u7^)#98Pc0!w@dn{ zH;&&n8^`NfOfrnPu5g@J;xR=@Al^X)pOlcxF<3XRg7am&1@AlVFRCb7_y4vLY z@DU%KQYw`1A7!nmw9AS>T!8WyoU!Hh1l3L~RFgjYC{;R8=?visDm7j%QCuU(63JX~ zjY&=cs8+mRb}`Ldcg{6)HRqO?b?Wfbl8QO3$}rvxMB2f4Bc@A=ih1du*@^463f#Y; zDXJ7#l%%3*o_VVV!gm-w_Qn&*asAP9Twj@lA2V{^?Iq*5`bSwaYV9&3X7d7OY#ck_ zb;2sbVl`mYqYsr3Rt7_P!U~a%mNY4EsJSUGhE?>_r+%QwW|!QNO;Um*P(R>{32ovc`zV;+iDO1|Q5TT1=J{GfQLWi1(8o((537$1AS;VIsM%T`kx3jY;~2 z9`0XL;&DY!%DSw3 z8E=5`n_YTt_k+lF?Pj^IwI<`2PMz~fo=I7e+fkwLR4!;oKrTS}#9m+gjj)QcShbE? z`F#mtm2^38VkPP2iWS*ti9Bvz2&>UociEF7o85CqHYyVFYEn)bk}AjLM1qNIlI`TY z8l%7k+4u0D*wAJh75+bIs49Zjq=sZA8_Fj@`A(OeH*PaBV^_~mMj}j zZvHR1-F#9rRB&ENU{T3XCE}&M3>B2(Ue4#6zZynN!Q?m2yLnZPX)3NypAlU}-k;@s z5}e=s*|!5D$d27CJN{&He$u-mA1)Qw#o3V}RH>jH0f_+T*QTyGim=*~#cKKMw?A7# zSTXCYiiwp}F4>%ya=UpkthV2AzZb@U^Lup}kt<~b!Y7oZkyLd>qwEdK`F5_o8p#;e zU@7gxC2drE0Fp_B`U*ZJ29{nmC{eO&uNYFGeYkn-OW%2h5z+G~jfScwF^o>YnZEbZI52Ahy+Nh+y8-{Z~Ps0J-b8A+YS1mYaY z!+KhS-D3NRXAdDu?rT|c;5Q^boA}-=EJ@{DXen|_sua@~wj@9oVEq$6Ty-H~wI7Ss zfG6L+v4pT<7Vs>rkh-y)y_zqk7s4vEbJt^u_4^a+yLxpt+r?w5qH8J=E5v>z>J>^+ zIuQ=H5}Pl=NvJUc$r~lfvpyNa+#At+95S1Pt|%VX(?V?Lzr5D@aI$2~vgBnZ>*tQ2 zy(~`>t{7R*T9QK5ctJ~IQCfiY3r_xJDPh%v#cJWbFMLx%STPHDs*ROIOR@0vM(z+t z46Aph4>^TGoH&IzIan_z;;I71Qy`*ADy_ddtZ(OHY{H1i@dT~qaPO|cFS6a^gg*nxwISiighV>jkF1|>=W1*cDrbl$b(x#WYViL_S;3N3B95S*WW?-Lmeymp zzG&oA336VB<-CTNte-UZiGy-GuOc_3X2$eJc>zO0uO);@eJC|dVYD|MtEso@m zS39pI5$Y%K5ztAgjP#A_Aw8|fZnI$9V|$SMD!1I%H74no|MJ<5CFH(XQ<`xqgyC(Q z5^xK!{_-c!F;#g7uvoRuUOlCRuwqtZwJcWFifrDPUI?qH)w36p2%%?g5rP8MqX7XV zQt3Km?w(XVw#lQB(EBpr1l-khbb#-KgGIgPOpxMGZNSX0`8s$9U7Lf$L+m*%O2 z)gdfaOCR~>nmnHV>f#IPWoA)U%LeuFHUk_mRKP<7tM6LUV~aqiL(A{p5xU(`Rs_*{ z)oGC}!(}zK&3#c_tf;YgBogjipURxy(9}29Y*r(0F3AZ+O~zy+@<;a=ieS$hh#AiX7=24pld{;7X94Ch*|f;%~8d4F7g( zsmpZerL)H*?iiMlontb0Oh;6fJx}o;)3e7^?igNE${y3WW67AJW{>IIF+8Vpp5Z$t zJMWj^j-gD!dCsKon50H_33nk}BtI~c<{ada(PJ7RW1Y!C6VgGG>{b2HsP0vHY;-I? zIcM_Vp!DB8=toD9oHPCKsETut$H#Newg5s6tCu;5T%1>t{Q$+Cd&(R@Wq1__ciE2+ zbI^|v*lJ@d^9X4W^f(9o2&oQ2GLKLqrY4*-{RlCy!XrfLqU@O{5AY&{sxbGUA0d}t zkXiF*YRN3W6b_+~7IQFwOY)uw?&;3+_yI~d7E}BHxfD`(fZ%K#^b{VT1UV|(C<+gd zj?8rSAS74((;5yysvjW5F(;HO{Fw>&AO{e9nkqZ@^dsaNMdc9!d=3#*9wBPJ;arwS z2&2J0$Rh+0+^6Lc!a}VBf|dgz2R_Y@kn2pa1mHyoxgy6ZnjfKr!y%+ny=TI?xew$= zNHy@?F+HEgA=Cj*)Ez6xR`J)B9TYS_M6!ER9wHreqS@hqB23R^;cm(Kn-8RO2;rFH zXfUJloCh!D+FJJm1XpaoCc;wBnH?m2s1)ZYhfoJykLeCX_zQO~vf>W(Iu8+y<}g?H zL*#l4-478y4>k;twtH?G!ez6>H8_YcSTy$_j}gX$1E9fUgaE}cnZaR%=q2eG+u$I= zJrOP09s_}l=V5VX!o4gH5PZBdKn*`Y?iUGqAn#={diBW8;cd)-35Q~tK25308uWe$ zN#Nyf;T?LL;gJ=89a2}

s$jHE0?mW3gGy{(@wOaGqbFz7LV@!=2eHn5}2xksYm{ zW>3TA6e$)rBIr_M{Wkt>|LS(ukEEFSqYg-X0j2H2Ve9uYDe7QThA|h<{!of#e+XD$ z{B5k?Y-~6`)pS~ACWCkDAY(4HK?o_xUqbOQMLJ);XhE+$AUGkJUDPx{04I?hcP<;x)qRh_~$)37I7uV5n zBk9cwl9+M3KW^DPoo4-qTeJS_&6s}D%$G}a`jez3Po%;NOn-#BYD_r#jjjaM5iC?o zCr{okcY>rCXNrveP0K=6U{);z)jo?>JVXhSBl9FkkP(ESJRw2jg60=!GvMaJ<@RYA zjkt<<6stTszv>CEkNk_S$vPA+n(kfamr2UF{VxAhf1*kM->gah7?bp4zW;Dpo&-rz zn^G!|+v#0u{?o>)28zK1CjFC-`sI&=)ln=~Q*WO4T?t|3NEuzM3as>tVYTMeU3Vnf zADxHx2^slnEcBy33CTgTP}K4D?b9-%;2LvvL&JqQUyr&aR9h>M0mKzGp+UFKBODc)E=|&OS=J@^d{)$DHN4R;i9V0{)VTfv)W5l9THh%&E+ z);&+iv+k)IXe*;?s+vd`n#$_dWlqYNXWi3~B(#sgVm`kFgwmxr)bLRDN|-QcdeTFB zQZnuo{PFfbks(jC3^~Ch{n`gEo0i9Y6}utrU{!6~kcuu6;RGh1d=g>x4;HJbk6-e5 z31Q`E;6VB;R&6i#i(!>m*4URKoRe}#II0=l=<#@5Lz75^BWz&LsTuS6dPzlL9u}w! zm5*?+B#LqkNQ09SNJ49epTeOlgtp@p^mGGyd`0&*zzo*r!^i zyv?Nj)FF#+EFllZnlc-#wrxtFEU7{g$n|6IN%lSS^2N4H_u(#k=OzzZkJ{gm56ei52=q+DV)| zq!+{LfWMw|AtiDCl{<-}NKhL>e2%<&GC@@rOvb)_l12kdgbKbEi6lRtO(462lqM>% zWhjNw;lRUsk~Hpmb8Bi>vSfv2$tfo52Tgf`@@YjmmcHdjg_y`#%m0aFp;$XNGc6ZW(?`=`^sf9?^oWR+#fdrj6) zTyXi%{?*xH^=mMXBTSmV-Ho{TbxjYL=g`CikWa8JUP{RauJcUl00+Bqz zCx-S4EzkaW$(oUy6YW!$DW{pVU;F;OgG$JOv8FU*Rn%d(-A4kb3pg<2$l+#HS1%7% zUq3%44_1SUFRoXZH9a|tRoinQF|4+@A(Pn|`t{!BBS^pgfXS=Y)hpYfUq{YShL#qB zK)-$+RTvNnL9q6hHuUSHa!0qZSTeGeD_@67mBd3i)$4jZp+L0PhI(B~xaxJpV`^Mw^0VFUhOgu*U--Vcw1twc#{5dm1lkL&c?%J;p2A zB<2_|Xv32@Z@}x>*hk^yY$1rqvI;t%blsE~X4UB&EO-Y;(TClXyc#*&(2?F@i#h%v zo;0g#=a4~(f$pBkV?%AzoillGaL~Q1A01aHDEZ-G9*DyS4+QJm81T|{VBkEDPq_|* zxCi|JxwrHw*UdM{-jV|dw{KH)m`#BrXL!v}_TWV}Azy7`h>IlLbp7dw2icw@RI z@#)rC2%uKv&vf;xKHa+KOh18`dvf?#1hTgFrj4DhMdO+|4&wYrSz76(>rZQ->f6=P<7GcQ&LJctUiZ{jD9PVpkXljvNEZgz5bSGLde)=A{O6>QBROFH_t zd#8jXUEFQxn#$=kC9JikIacVt_^DZ|@?=7aeVSuua;3JXIdVc|g>$FKzwRfj(kxaB zKbSWlPbQ=ov0|i)au%z$R|v$g`mST^-jx6AnonmJaiMKud-gju zP<@C^Q?Y*OON1cnEhVswgt8AbtV6RqHW5P7#prFfUvNBC9MoAA2Q#ev-=GgK%DsG1 zbcT##Z!m>1rP(r^I?}amN|8LsZ z8dgE&0G}vrS05ARUtc z*^|10+9Yf-knp&C`U^w)d@dhJ9)m7}<$d1eV+Ayc2nuVcs-8fVhKKf~VBBNZS?=!xrE1J<)M{%O^B5D8Dgh7n!NQJP6I$o7Xbre=<)Ollraqz22*on9tZU+d#aKEftXhrC9K` z(+I2!S+G|9xc0nK0*lcxI$+t2r}Mc9A+Rpn|C@gh@h{3ldb zmJ#*;om=4|V`?fTRkRJ2&@vZAMJ)H*DQBd1cBsivA&>EjcrkH+(}#K&c_STjXpgW< zAvxoo^P6tmiQL$KEI0PFN&UGKUrv>(E$ukJkS&Em*TV)MaX5kXUly#zZ{45Y=NH?? zn*F0HBWSc5uP4gY0&YSGtnnwuhTZ_;4+!K=?$Czp)E$Fnp(+yktfEnEr=k(-l?D={ z$m00CJ{2)xyI>Wy+DREr-aTHQG>v;@KJQf49oD=gU^%j9OzsbV|K z*i%Z1^j|-6OiMR{YoG_N0Z%{pPD#PV$l=s%tlxG{Kn$*Q)s!Zp|E2lpuS*D6P-F*= z!aGZBfhxBDYS%;;OMZAb?8WDc_NkZ&H;7|s(L$CXv-DR<+t|H+!Nn`-Gk#gX@@CJO z^j|yhg^IkczX;jSnA7Slo+y_Km{S*#XW#p6hdzYX7- zW~kUpB)yx>sI-W?_t^0@-;gi6()ML9SP}o*V@k{Qv*ygkum$oKLesa4?wfHA!8OPO z*YX!i%=I(tf0_-KYVVRA&%L)8#Nh($ziNxokCN-ZDxd35DpY)AXxRD=xkyh5lQ|aJ zk>fsuYfsMKJwTKLhEV|T1$sLmEZLj+b)C%`i3;pPxIuaBp@sLE8*is!OMDyVaY0b7*xWXu%*1E6|)Bq9AC{ zC7-(ePdU}?n?c=<`fh!F$lK|=lhvYgI$Y?2ogV9cDV*EW-7lkrP}^41`&FipY}vhrHdDF8k!|tZEfm4$oB~S#dP7M^_GYyZe$nJkb6->Sdq09qlF^;`!9=5Cl8i%Asy|vE$Y4K)@9t`#Q3l2kke> zA#Zn;=zjXR#N!*X!xy+!FzcvqcMDw*8S?H&bDoD6xD!JhbMXpy(_C3uxc{r=GB$_8 z9dDUe;f`7}r*>l96z(WwaIER;#;%Zr*|rHUamTAT1oAawXPpzTa0fuH!F&>TQy6X| zltbd~erR$?+;Lg^VR?x=9@cr0913^$80VkO^RiFFeZ0uf&M3(taCeQ#3*2!g;*&hB z*f*}iU4an`cVb8Xa#Y*-6z&RnH2bGng?k6^#QePEk+|zJ0;L?i*vqKfXI8@=x7xJ3 z&DP;*v7gVMEd4vLaPNq<{a7OL+3b$r(6Z3bKyJM++IK++0L>%X^UC-Svn*t8M$JqS z^EGY6aK`>z3O3{ARz#@}fQF4v(K-gLudzVnNe+_mao@_PZ z`03Qvl5%3KDYHrO0;V)X#M{rc9y^fW8s&j&@ytu#EGf7gy&RaIO^RE4mgHr=5L`nJ zO}t7z|9aEs`*pt(-5_xenem|B_IaCl90E*k%TEe_V$3d5&;ji zzn*x9q>r&x7a#Kgd9l&97kk6<`QrwiUP@k!wPiLVUciZMf%#e0kR^o4%X}fUW{x?(nHEFG=2;At&~OeL5W>90n>)nR`njgR$_qQZvPP!pr3727UhGQg-GRZ>6pcy7lTXObhk!FFUzEbdR8{8WBN zR^;ZiV|}H7IW-aTPx*G4OmL0!z_s{=nTtvaE=T_e_Gfi4mi{p>`-R||+VH|OS`580 z-(o0s&43s{g=`(o+R;^-b^PrVKw>ry>SGHgp9~E3PY~EhkU+xyLulr8{iJ}5|LvCP zhmtcJZ#%PhO!`lnH}r&3a%PM4L=+W9Kfkc}NXeZGm zPvwIitOj5`mAczuqbGDpU>!dRAf@Qiwa<_zn`nEor6%)d&VFEKDS0xFG3^MpfH7sE ze(ha9`-|!LZ}z}7cfwO`%#VtY`;KA`+_&M1+sz;Ha9<3reb4x?nlhZXm>EugM^ngG zusayy5>$SmMm51I-!)DEnQyX-T(yJ^#(Yr^%Alxqq=avak%*(DgeAX5@<;h^zZ~C* z9NDe5Bm2N4f9ve~jw&Iyud&9oW7q=51ndGDn40E)eM4y7=0R)mg!%dBN5!@eS^uc! z=;c8DELsJ`qC#k`-u|8>#XPs?i+Rw&5j9D0%?a#SAbBeNw07zr6A1Z{Aw_J%hx}M| zO~esYp`i!1d%{kC9`cj^QCYszOA0PWFbDeEa7k@feu%*p-D00EG~u0OPIz;8{dQ#EkIChJKb9NWE%n~SOaVy`#F81a?SWn}ITiAub1+sedhf13J z{dxDG)M#N>yMFSV2g#L9v0d3GCimyQKkxq1ab@V`q_OtQ2K@zmnMi+wAs0P#KEZX5 z2d+U^4tO#@Tt&Epp~kERa{b)%XL|&~#NDg*?LmRhz4-zk&>k6nRDNI=0txG|Ec@HJ zDGASw2F=Va23pgV794yyg+J92(5kZp1kkw zP`*zKl#e6j`xhp+X0E;E3|{%ZOW0AjH>c_|0|Q~q1kip@BQcV{tvhz=#-=xr0vA%g z>-Ydbvb43^*S!o{yCkGYxf;9M(svT-ZP(b;P;Iv;jWwp4s?Dc@9&sHyyOS)<6o+#f zZ|KLMgR_tMRfMMe$~@1nVTNKYZ)6>YU%4(+?EO!aBs(dXAalhfu}XlF0WVS|IN2oT#m zdUc|ngMN4*#<%r~d~0_=o^XzF_?YIfAIGPCXC=10_8rDbN?Ckzh~HhoqOY?%)WVL` z5!F!Mw?v0+S8vbZgNJoW6(ExHoQN_82XvqI9T#=P%P3&+oCwe%Jj_DC!GnXYcsaNB z-9D4ozT-@{hkp+7yW0*thxnb5uygpB5M*!3!Gl}6sPo!)9CZ1dL;UXc%H}(}O-Q}WVx_M*J<9mgE~aSq|TJMwZgc6TMcIh60Noq6Ru&~bIvIi&9n zUww_;+2RqO^c_rh1ogc19dGG&S?7GRdCazQ4&l4|sJ!r^CTf^4%eO4&}T1CCQ(+BOchBnSxsR9UVg25jc^r0*&u?S-Z9=$`J1+~cuKs$b*4KnIL) z>?*-a-(hdF1R;;~9Ua{Bba%%F1#M{Gk%FL|gv{RWy!O3A(uf&}+FQRqj8Y``=hMTW zD%#M|)&ep{^!#N-kmeSrOok+%{1+SA>DbB+yL8JzD8@YOCnaR!$`|)|h35P->^c9J<}7y1OD`T+N^^e3nA!H_g^U>& zSr`3k(DJVdt%p2l%^dsaaixS7ql9$ODo`dPgw_$mryfh`!I}BegEEM3pe?UKyYFhc zCq39sDI_-Vz*eT%&cG)MK$K^ot^^yrqxb_K05ADT3Yk>aJoW(jvWIP7w#Fp?s-bJ| zDlK2e8q}-t$eSP(Y4h_;*F!@D*DNnwj|`cWAFd+Y*;-@7bJ>3H@wOYo2yOBESnvA| zAw|q1d20R?Esmy0Xxa!lm#jn3#d!X9Dj^X7N-}nWhY*-s7Nxy>5(+l>RUtStuz4yQ z_mB{>{OJWRpGm&#QQMb&Ym$HYwG+zo`m$m-W>)0UwlRgWy$iMvokD0m=0R)WO{0$~ zCA4%#37PHsZg1+I&r=AY^>&ZOu@v>p&KvbW#E2HXkny64uZm8kF86g#37IGALDy_} zekd0Fjvq090=<^d=|#bI2bxC%uaOk;fFYk>_X~NlIkqSJ&gA}p%buQ8I-ab5)ML$T z$X}py>0yim}?^FvoA`{_>`U}H$(f2b)$4HWD8brM7N z9C5|k{^ZP_u$|eDCjDprxTthozt^1EZsrBd3E&0H`Sg@|dlOnudeCZp{Pw?>jOzzl zj2O~EtL-{TF|;o3_UuFC`k%_@`XK|*pwUIP1}+?ONS6GZ7&5Qxr=TD62462o+o`ZCs>*^s|&b3%V1K>okJ z_tT*S*E|ng6Q6$b;gb4NGU~Z(`}YkZ3K(M?^9r&5nS9AkIAt|~HXNAwqDqPF^I=YZ z3nx2~B!%aP5)x_}F5QFtx{Ou|I?BCK1tMV+5hm!@Neg*U@7JF@oqSoV?aMCt_RfhN zw^{tr(w?Q{%UEk>WBvlEPE{m%_tqCp^@P^59<)|{JRpC%r`Yy4gXd?obJ-5?cEL>E zutx~3v7LLoOWc1hpXW!db^=9z3YyWtM`KRP1pV#X-Y)@J1@^N+n|EKs11u34)JFv2 zR055FJ==chq=h{AyTp+vlPi1Pc4e2E++Tj(@X7gIS&^GFE9)*0^b47DR#oj?1lJ25 zxE5YH{E?D^ixJOd8^9Ox{6cX3xIEm4g8un=f_}7Dl@f8}`jPNQKHgy0xsbkTj|v_- zVl&ZipldMi`mIbq6_qCtV4{BZE0!#Kjx$wz5uN2 z=Q`cwoAE-xrpwqR5Q%(_{HTpGu*8d0l4QuLjM2f8pB8x!`S0JaUPjLBCEJ+|vf`cx zADL28&WyEZHtH`B_(&pA|HQ~)w-8(}d*B*#-K%XZ@)n`9V3q~59pK~E4$yfcA0fCV zy?Osm6#2Z8H|j@qo|cfIg4ZBYU^D%u-k}{)Ke`N~ml-q?KG%=ZDD3Hq{h(xYIZL8} znuqu)FIkQE!GrQVk|Ef!WC@I&^S~DB<7qF%(($9a+StI_jH_&=DkX}vu zyIN??{C3rGk@xcK?>eaHws=L{&#Ve&TfoN)bVw1xYrCK4ckkRW*y*)E`DhZt4>j2} zX2Eyu5W>sI(`ZnXnXOzMmAu^pUP0j?*bPlqAtgMTeUMsgD_030(xIT*@IrVhOU#Pa zdC?_F`19)E-9GPr9Xwt*uMVD+*QPK%hv3qY^I+A=&EC-n#1abB?{bD&+B+qN* zNOF}`DDpyhh`C+KF|UL-#i&_nIaeUS>P>w*cwE%IBd22~wV-=2CpNa4GKUV{1R!@0 zct)$7Lk92Okw?cy)x0RLgU3rc+D@Mg9w#~?)*Lc;v$VpxD#sgK)omW{M~AtpA01b4 z%gf-&l8#!N*TJ*a${~YyJ09~b;Mr%*p@Mf$(!o3E<BW=^qfh`njJ)*%9iU_} zV*t~)p{c2-ngAHMI(q&ChRL5}XkHC(vB0JeFNeo_I#>0n;mu7d>=V~f4eypy@M?G# zbzTgQW6mxMIkfPuopWg6-D(PhM~5AGIr_l6pFX}1yot`n*TU}@?%W&vknHM2lC!Hc zlG`PQN3`PW18>$C_{8wo41>l-9x;5J#zu}l@EuT!f%8CrdPcqxd>P!aqm=h+;Zd3? zZ+7N?-=jRq>-q8|NRQJ$Xx$jsp@nDL1z=*=t|TEs3gk%??1|^glaLr5GetC>#PTT? zU_CQMQbZnh(?xsq-3N1@#ewvffi9ixSbq7Gdyyw8oykuWEdxB$pR72h!1PC8ee}vl zkJkyXH#~r?9X}qK5*{$QTmSd3$5gTSW{W0Zj4aY>2XB;X1(HhwfIT+vhOKA_yqRwZ z7-)`)ObL1&p{)hB3uaS=?UO~K`zewoxPX6&6x{@{MYG+zcD>+FQq^P zB-CJVz)@QdrBA>Ic|HMX0HeS;W5Xu2w9~Pu!}6cgNap(lAa#Z~3bP2GND{en1Kda6 zJ&EEFG!XL)01`?5?a_T#|41I~L))W`HV44eX(MiH)1wu)Ma@ZoQC`5J07M|FjlCH@ zhyeS@1K83Br#@I>fH~4S41p|QZO@v-0NeRYXY-xrv4|WJ} zO(XNkgpuf#;z`uk@)Kbso6sd5UzjBJA4aci?+75FmPkc262EfuU?)sSUTJFJXz{l?%URroLVoAkr6%V`$?5QGzSKoOP zKDr+KUlHgsJP?KxXZJ+wf`%mMXON^xu%c1(N46{+8~C*=5rowW9a5VFHa*)l+^M3e zr9M;JFEuc*HOg*d1`4oHhN7Sky~q8NQu63dsj1(RH~Yl)W;dGTU;6sci6!RESZmq= zZ6RwaB9*b1XFlmhfPLx#to8Z{Q%ejmM_vc|+knYdSA@KZ86m(zt0zt)_y1WQ_a8S< zxrC1HYTSV8F)p)NAagp;XbYH=fERH84}HI=mhf8X!E4g!iGOViuj1NA z5qLTJI?z9hSKAo_LA>T`U+zJ*;a}vb4M*k&qAs+vRuj;?E39G+Ew0<;{;?VXy%`Dx z4Ilk60YLC$#L-q2-xyDAxIuEszmK`=ngQg`R@wgSW|RJtCXJcUravn#dtwp5V^BMs zmD@HbkQd?hPUa*xrr%tfnb2R+&Zy!a178 z(;iMqRhD&C)3C#(Ck4dy#mP4XhK>O%2nqBU;U)kGtzt;9BEh7`6EfD@yg4A!Odk7< zG-C~Uw6AQBcAGf_7XGw$|I+bjGy)idT7Bbj7-)$DiHZCSXOjWfVLHvv6oNk2equ?H7$gr5kVVQQNQge$l#y z=H_Vks>)1tpS=&nWK{^Tl^rtY(FjAoU&ei~w2)KK{X-kKdj=+IO}`yUQE_Yi}94Z|Qsi zICI)DZ2@y)e>#z6upTE2eU|Y0-hE;vbqk%j!vyLL2)6Hi!d#1{r(}$BBISFl+dCV_ih5k3MaNg+z&nXYfbd=9zFUjc+ zJ-72K>mWp!DhHST4%JSNIdJ*@QJby@?n?=A6 zk9*4;;(7O~ImGi00y)IjzS;CmylNif(!q`w&0C|(1~!Lk-YuckIT9y;brqU*k%`b<{-x( zoQZFB8@hRv;AD?-4#?Kc+i%M2=5b-y*1k6KW}EQrnfwT2`Na$8t;^bG;g$0+nNx?v z3+Hj7TQrudkG$g!c;!6Kbo~=AoyS4fKl!xtrqtVhO`mq2ea#%&dAB5i*UsZLolQ`3 zNavN}Xd|!SJ5pFVkDe5WSGEg>EmgbKM!pP=L(j_TrEG6`SkxTJIfe6ym{~5EM>wBQ ziNv{t^Bv%cXAbiQ@-%WeEC9|I$z*_8am0-HrtF|Akj|MBpjZGbbAsl z^Sm7fWW+uOy0s!@m^2_w<5rAFHF?5~?@YLdrvE?L(|;?PA3px=F{L&A_soCn3~+(@ z5Asw33By6d@4J-%``H87n8gdxgR>3QWO2rFfTVEa>80Z>_R#qjet7vP5 z8a0JfeWI$WSFc`jRgG2!eQ{N<#1oyv;ZBBLRoRlR?UOj-^}bi(76XFmlbl})MaW<#1>+y=H}Y^-iKFaDvxcbtVuN;ok=ytDk^GHbrluS%1ktV zfzeB@!I-SBMB|%-*8O=7{PJJN{0+}@Nie;Q^E|CLe5*zR;dQ^-kC`u?sm0X>2Gg>0 zwN;CLP$PSs(o|WUs;aE+8|~h&p*a)9Od5NlMXM|8>Y{a(&6&N+1@3q0lJQqf!Tl}` zrdyo*t^MK`HPR~@J+ihg746!ip)u2=x}mJz1RIFhbE7GgJ^OEOOvH9->(bNWa^BAKbnE7T^UUO*Lmcbk3XkJ z8aIsD!JqizS*9A&cy|<3Z}~)G+J*NV?_3~ z3~1S{qAAr3w@`6zOR6OmYi!AM?|ytkeQLib{d;c#n|`zX={?A%gISvn8gaiNWYb;# zzNmZzu6A`W{eR9|EWhOiBXV3poAx7jM>{53?yiUVip|WpOCUQA^Wmih4EDq+H9wNU zhOh>kJO8NzL=86S+~~41alxU%^d8QuPJDXV0g>(P5tpfK%2Z_fG^HwQL`K{FGe5tR zEOQNOnTab0_7t+rEoB!yd=suVEST=?Jnq~%H};I2{_8D+{^It?V%}N6NLx2A`GSlz zoHf$&3A1~O8mW5B4X2P>yq0x~i+@_&GqQ7iWxv#cd;NpP(UGYXzN)8~mfCCG-Qrfa zK3YqLx{fu}%n_@_43&QQ`pt&mY9oT_e>#Smdh=BWMvg3KD2r@}6_`?$(p70qXpHXW*Ihk#uaDdf}?`zJ)L))d(Au5k)4>agYfFW=n>~3 zKoA)|JCFWyOR~!KtX1ZIzPws!_>8{u-Y>{1qgksge&)++L91Yv^J}eg+Ul=TWR)?j zRmMCus7BN(@{40<)Zl_+gX#Spt1KNfvL^Cp%PJH=!6b*_KeZI1c0abEx^H)pug))z z?Q$#`=LXg|W3IWOM#wlP4L^7x8D|`8oVl$x)d(6#cH%qXA#>-1cY|b{8(HJ58hw|r zan87-+p#n&86RY3B@=&~QWM!RWzI^Xhh#EMwN)*d)DcZh4Nc|i?#F(dGN=m~HSXnGPWuU0yCax>nmRsM=r8>E(@}xQgMlMX zI_ZRy%A<#;GO6kezI!LMWKQUHT&iC~(}m_}-aUFmePtD9pS`2ieNxqZ8yjj-Wq`kB z8Zd{hX{k;{`<_$l{^SZ@p6@NmjemOoo!AnANT#6!4ii`_CO^kEM1Tii^m0_n)T(gx9kb1rUcU~U8v?Ry#Ja| zPiof^0=L1i_l8PU*EDz`7@xUPE9_H+|LGkEf3aL_Q#} znD#p8urs(_&si+LgmD;8Fo;P#;jt7O0{!v-ecM0C-b_>0c+)7!$15GQQG+X~FcRK?l8=&)jxsPI2_6p}{9|#^!CO0Z%{B)C!TW>hD}wk)So|CCN7U`;6ap8-;b^dv7~TVa@(9!{t6Hu2&M2+jRmFBYIfwCoig> z0!GLl+?j&OMJkk)l92g`1}yU*UHQ+$0po+g^bi-ur9Zrl{f2%Z7?bVgqVs_%ep2?2 zs!EcFlIjgma2=CXLC07&tn8R00Oj;xdMG)@hpgD|yBGAxPhv;{(xFAg!5j|*EDitY zrl1fyNk!B}&e`d;503(tGlJ>iL`w_H<=_2uLm=`Cz?3lQ&pJVec%krOKOS&C6ii?5!nt(TyWa;Qmj?l-BnYPic~RrqKdL3zFwz9o z41^%vmqaHnA3grr?k57&nZfiJ7u3azJD>PXeqD2h%sWunu3e;EGV>DuUHFA6zFuonJ0uWC5(EGT$c?STpQI`GzXW z*Silnxz8y8c2+QbqYLb$5i@3oB3G|RV1aP)VoN=j+0T^=E#^dVQO>TI@$S@90qrBf z^i2fqO!L#f{JpEk1|mZVTJ_iZSBJLv!__%H!$t8UB-G;U$4jM;mrSt$blQq&q<`yP zjej~F;655mPjtbZ`O<681|q}OJGj8NSYs~0tFW_C3tp6cN+|PCbwPjId*|}y{{p;^ z1=Dx9@U|{{1+l>>g12BgL?{pg&P5qBkXD-Yy(Y*K6k2GV)Zj3$uVJ>_Q|K|B9kjQ#>Jdy)bc%-}NW-BArM2=j!?_fRVV^PFJ%E}DWp zZ2I9<aD>;dNln|-!?2F)juS}Y+tpXgI1Evp93bA##0F3xlBd{m8$ zBb*cK^K+Mz<;Aw!J#9{ql-Sh~i_w%TM6Nn1iu5kiUK*ML(vJtz_qa$eee{)7WIQ3g z?&+DFaJK!!q5~UPtL)4}GxM^jD?m?f)L{vFk8=n9G2I7PKM_n%B`0lpkJeA7eH(~O z*l1wG6vc)&;6{@&H}z=3O&MHrBARI6ar@aXzLW;sPX^QXyKpambOK%T zI~MR2 z2}SFQCiwN-ddt=7xj_AyVESPf^)aKb?G?F;P+uR}&^CCT?@;l4OMyFtGR$XGH`5~~ z+m)Ci8nxWfyi(2p_ts$gQF1XB+m|o8`Q|`mG6BDy@}>CzZsX@ggFQj-aWp3MW=1tG z#}c9u%(ZvlaqW44{@Gx9whR53YhFdbc+YPOJ*-kJ-&X?PU}r`a?@fP${RTzB|4(wq zcP;|>&jr&@xZqD5Jr6rTO(XEvYtFR|mbY=JIJ`a#fTA4U@ld}68NG*e(a-FPwGWNy z4+A_OOh4rsVCjeVTpEfzKn7@2gVqLW>-SJm*nmus!vWEZHEv+fO~G&O9Zw!~;D2C( z7lP?|t_fOST=QTkGM!AY!E(3jY@;GrpnwsSIGaGXz5G*b_9u{`Eia_g@%c zelYzE1wN0M;pUh-e^4S16_*W=sQ>NFKvwW9V^~%oZ-icXil`MH(f{5u5LQ?aOg~3M z!Lq{g_rCoy5P6u)upu+->pU8YA;&|8(Adv{VaB7ho`hW~3{ged*|+U5e-I4uVlchX zHN=1c-`o+5%ppT;_~fvFIdIBvoV|xRRFzHonC387o**_&ikf5IAvfP zvF8oNFnKAMewDr&kD6bNwbP&7Eb=tWqHM(ErbF&<+q!MkX<|)-GsSZMsY8iy(*$#mLmsmUOy6cSrkmaMPC-n zgAIEB71SU-Plox8GJXQ)`ORzt(_s5M#*vw(q0JAeIl6Go=A3i>JYXzL^J*~t9u1kt z%#X{Uo8H|b@)B9*w@V2M7{@y0xAALap9CuhvW5;y1tjdyoDf|W>it>yo;SiiuLaW| z($KN&v-F*{PX!{ckb#OIcO_t{-(+nGCNgAp2qn#F3=JW4V-}Vnzw+(7i*JI7UJs_1 zxh7gXYROf>$g8DmB3Sje$P1H^JbEw7NVN8$BL^(H_OaK*m-fC5Mp}$Ae%DCLAOC(9 z%J|7h#gqRk8IdGRl~CB_LDN(c72TquM|dT?Zp!d+x5HF#1k<0>Cuz3%Nm{sKt{Qop zELCLb83FHyQ#N)Afo$c8>?~W++!S5J4N=kAtev-Cb0=)|W-$GQYpVf+homF#{%&jq zb8Y1HNmwfxXX=ooxEY62!YwJs4AD$N?|=VPGX>UKf}RuP1g()Z_xUf!q2~mdt5_3m zVq=$u{Ot zBcmmmh|o3~1?>>HXrjT!UDrN-@KhM>?O=MXYqZv%N8cZcd<>(hMW3=07;qcWZfI8# zXS*%fW}}Z8_6*TQbpk)F=y>!4Fxxx9^bhngn_~tZQ?H(TK;%=hT8X5O1Z=m_JSADq zU_wU8>@-3G18j;YUSNCN$pg-v3Cq13Ob-a*!)94-(D2Jp1pWn?u5=Pi0@lMR8_9q$ zpCmJNd$#!$%BCyg1MkVTOHQ8!^DPag2fF54JN4P;0+H4K3+96bH%>-`49NCLVho4^ zci|7-+?J7_JqiQ97ffH~8gS<8b8ihsz9IvbU>Z?iAa0c8FqsgWv^cAbwh7JlF4!Si zG|~7~e96CO!-Vgn)evLC>mGS682S2t$Aqx@#;_uch)qp>Ml>DiuaKE`8?e=lb78~} z(CWxF;=&d8P76i8B_oz*wpd^&;*^cB4nSs9*m=4Y&X{J@Vyfso{qwu7TKxpf_+cv_u02D0&4VpJ4yLam_iByG;Uho5Z1I2B zm>W8OL&juPi>xuRW31@6^5-?%FK&e~m!toiYs{sOJhKM<-~MMuCI<5cP6?1XbwrIrZ^F1=1k<-L z#-0Ad;85gRGVbqvL6b9Id;Yp%1~3e)%WPO|85p8r>eePKay;zJW8YW;1Fs6EZ>7)e z6Xxf3?#g#ML`H0~Z(eh^RemRWj4bS#$ubs3BTvzpY}JWx9r!jZygHb^-L>%2aqrbc zMv;a8H6TCxpon5ZG!IFD?cU}r*B&qd6HAd8JDqTk>L&dJm*g^j;NvsQA`)+k;) zg1R}6-elyqjlTi>HOy>CY*NZKb5e@wqI$l^FS_XB_hII*g6X?mGf({Gv#$e@v1I1| zrDfH&jhgf5`q#FQp*;()mZ3=mp&Kz>v{3Qe=6|2^Aq>4Hn4aPqdg@be-yV#N+hpjR z#^rvv-fc~$_I#e*F%8!(i{>RiJoS-lm%-Fu2h;burd~B{JD!sA)dQFTY6{VXR!BLv@53JZP|O$j1RF*;;m%x|KUYL)|l=5^LkyqAd|B#!6mbY ztVEF-imrus#=Lj0%kNwTlYbXX&vZ?`YW&)HfykX?^8f8+#dcbn|Ft$|(Cc^zW>2c@ zgxfTG9P1fKr-}F7d9!)v``5ti-=l@CYxc#DT{SEenM`K?-Ic@fCTb&Z^6L(*x5X_o zJ-c(3J>n_Xg6b?)G}m(0N0&bMElmGIFg=Gp>Q9*;_2t*Ur9|$7<@HS~S-;Ny>nXIu z=Cg5?)n8fzpLREaj6^tJe%Ysk8oz_}e+;G{cdg$#@2ftMX=MH1Z*AU2a{%jSiHz*; znZcP&JL9JPe?=3#Y17wy{3GoD6Iuw92Z1LmIpuo>)VeC?%b--#Q=cW?{Hm!;Nx-Wo3qV+KWAQE^ov=#}d z*%C6eny8@VwfToDe+ULT{Sr*Kx+Iu+|4VNKBM)sNfn&n;@`v@n1EPV>h65g7AiP0Q z`lry&LC|18DE*vEgQ*MFBBA#%(O}c+>U#187Lk6v^cX~tn5(`fWey z2r3K=r56wto;D|lQ$Lx3EZ}3CsIXqX2>y%wS~d_FJe8MLD`3hNsEJrYv@EDgEU~H+ z$Z%;Wy^tZpHzTkW^c*6?rsci$?H4R={K^>w(ZMr)wCG^==!6bhR8QIQrOGv7(BZOB z`em06!@pjIt*am3LglOs(SpKCG^Y}7nkij(O2xLI#g(D-837IoSiD&K z))m(WBJ+tDo0cLs3Ne12-ym`%+4>!m9C6woMp4Dpg4@jI zh`h3i9~;$iutfBjzMX#|h9uYttQ&q{PhCy4RN=rGdp7J2h71X%SGWvWc=uyHBX1Bx zHZ6v-hTG_WUiWYnJkfQQC*}?faWvFHUl#G{%Yf#N+w2LR3=O4M(x=AqB*U+G93^UR z6Hzv;lx=i!`2E+qitj)c!?QtzMHUKMWnFxm`lpS?72QFWYeMN&E?K5dTaBWacQ=v6 zZ@1s-C+jHIBf2Qe2ELL-7qekLx)g{?GEUl>mR2*Y9X|cqEg~NfUp6hu z6%Aj&nstg{z?g(5`|5m=q$C(4x{mux)r{|ZfHBvG(%-s_nR@-32S+|8#%x-qDJI5P zu4WxRPQ)5^DMvOpx-x@a8LNAu#l1VfaM))uSaV${{XKn{EI+e&*^OfYkrkV0^IJtN z1@GH_<`8k%A`^x_&wf?iMgTYQCLg^n};tU^j51MLyO2pZ;23UOl#-dfb zr%<3yGS0*{Hg(Kadx@O=e)gMudZ%uGC`;C;DOJ@IRaL!u^^&VgZCWoZRt8x-YG>3#EXv`4XsdL5 zxrE#0?zQ9D^U6V=QK9remp%(yZ$od@)kL37tBS=Yj~x2lAGZi}x;|7sVno_Lq%s`*@*nnJ=m10K5t;46T*8%OUBelxA0DlO%2tteyM&9O&7*0 z&#SDhtE{R^9e3!!z~<-)t{O%xu0*y9ceH+EU?7A-@5rCR+p?!MH8;}^)S5L-fApX^Qt4PE)g|soq04EvaOWX})8sb4lHM43~6X!0;ZzC3WX16TV}x ziv8kA?wAD86OMHuP+&d}D!88lQ(MP4RB%IFG<(pGOVT~a0Rub0#Mv|b&}izD1LkZfsH!))q0-C zb0QB95O7cA;K7NGMc|A)7xh>K1p)j)*CGlJk9kX*m5Lu8#X&&f;4xn$`;vZi)GX#$ zvGYC-hMsd#?DOk6h>Iqi6aDDmY3xT;Idp)4Yf04)4=kBIlZOY-zEJUR@~bsim-t~!qne6wRLw9N3l2MFShEuDi0 zFKOU&Y675h=yX7ADmf7796V*or1^2OPUY~yWgXXp3Qqocn1|)y@r-Ak!v_ZetW6&y z2SCo~GB|+nJdP7Ja^T~h=*P!Bk;8|c)jg4e$GoG>0mF}uGkOsJc+kOljwR9N$a4%g zbm?kv;Fu@w9NxwZjBr{k)2As_S<{S&N>bvH-NHNcHp3t*&N`&7uA#azld2gQ=zuSd zp=N(UvO_iY7vKQlD7FuGX0L$jf>@Cqte<92L)3*qrbo67Z*KiE{%!x}cGi!inE9g) zpbuiA?ZaW~_cEz|jd&I~6NEL}SwEFx)=$e|vN#6KHrB5;Hk_YoI;}F3!MhPh4LN&E zig}LdiI^%8yP5M&%*Kix;DPb+w2#_nx<`{(8I>;nxL5r_m|%?#rLUj~mK7gQ8uKY+ zn_pvsmDnV`D*g$U`GoCB7Sbe(Z8#>I8#I!kxFv-C!p(Yq+DhK;XmIJqQ2JVzOKaEMY(y?8gIC%lJuNk_{#N%N13rxprLUvUoaLLQe)1E_W=ukD(i@g6q1s+F zBSLv}Lgr4E<~$DB0#$Ue?Dm8IbH<4v)J>uE7?)5>Uwr~{k1NSTZ4xAwE}^nd-bPdo zTA|&D+gw7nXl42-ERl*>Ca*n_8lRue+$8(mrrf9X|dK&~cQZBi{}jpKY!O7stT zzk@_AL-J9J_TI$`im*UU9&kv_si4-xQ2HjK*7Ig~zI^4Pae>HCqShvX+V6#07!mm< zfgl%j8VtG2E$iZDQ++j-6wOCXyt3u2{{y*h4yA8%$u(ffqeyFxAaZRIAN`KV<#^nj z&3HgDEzW)#EsBxM2Q@N|Shgvz{`+N_#WkOb z7fjPVa!c2rK@w(&hWg_Vn6v8eKLk467D{jFxU&HdOz$}`&@~dRtV&i_B@*#=DaX%v zX7D2^koNXa`W~0GlkQ#6GctY?X`Qdt@B0sYrj}^S_R5vb9kb%p9u;ME!Ua-2zpSYH z8)$n+D7}qC+eu?z?HLJ#*PXea$CIGx>7yT>U_mDQ=L%Idz+ z?)@5?Gf}9Cq3exSSJu@<>nfWwdzTB`@aL6p-FyUYI4P9g%DG|d7nffjSoeoN@zw<+ zFTu6$3Z;*6t~Gq}Na#VzqtPR4>r&CKnWoCx`dH(IUH6OD)z+sV@kc|_+WKhEehoD( zb*Y2Gr1-Av*LYxMQ}5;&{vhy9HP4^=8NJip>^sf9Vw}i3t=_ZW^lNdg$)WU_&O0r9 z;%25cU*eOog(kpGU~uj&%m|r38lL_@3?%#y?Dnie@|Ul)lk_~b98+s)zs9|m=So^ z`tT|Lq-VXCeb%X;JRtI{7ay?K2SahK`$Fkcoo8M9*epEjk>SHznriENNBdOP*MMBT zqYb^HnLa6eI~vYQMN=14ry4W04fXp)E9)EZ^CtI4(T3{kmZqjuO?g=0{hrzXupj9C zrn2w1c*tCl_uGBT%J;|tOk*9uq`~v>ejCOC9CFV|Thhbd&p!O}ho2XD_~*_})J?~= z9tfrX;~MAdm+AIp)u2=x}mu2L3PM>7s`}Ig8FMtz(1{)xZBEtoidE2GnxSuNtV{LE2o0!~efPebaCnf# zVd=`(L~z*r;9X|+#kHn~(yD{Q@cC~6hrb+MUz57vh^D57rt&DBH`=SIp&tktg~Bg$ zVYDW7Zc9plf|qUn%Z~K)GuWqJHST?prytie>MR;!53xh6^~sNa%OUplxqp6z-ft%R zev2>vOyvFg&UiXVL+oL8hz-AH^#&Va@01Pv4?X-W_TdLD{#xYWujr=SMnmioc8D#% z^+(STtHvj%$3KrxW%lofxFFTkL#}B-fztjMYzOR@wL0s#xQ2O7_5SuvuiqU~czk;r&8SY~LMnqDTy;IFSGUwLx>(QL4VK9)Sgo6U` zvr*{P&=hT`N)tM`KoeDJiG*C^jC1-C46|7n27EDC1co6O%xIj3Yt0FzFLGh{Wf)*c z7lfgp$3>G8*5g{PmV9^Y{L0!)_e^cSR6|QfgbF9`c}|!xn9E`?=Gl=V82t2f=(d}1 zt;a*@-VO#c-xv!FPAiCkr33el067U&D_)#l(xgBAA=MPCtUkA;wkaj_dHK_@VfWAr zKf%6m>$salUU;vIzwA!&+LLU&HvG$5@xtpRUc3FRe$Di(PqEKB=+Q|c&w5nfCu8(6 zd7Ay0tX*;sp7l42*Zy&l^Z>ozJkR^xbic^^z4yS_8|Y*54Er&ed&fg~zYP2y)KL3PGvAn|ioUlbdeBcv$r{~yrTKwrV zBJY%bG+B8MuJwEW91U9n1c(du>1ZyjB3C{h|bt z^(PY~`v=V}y?WJNaNu4hhI@%1clLRcA1CA%u*fYRutWs8*#j%@rqJp|Hnf^})lvZG z&R%{MZW}JN8r*%?^9YB9EDo(tekg)Nt=ez|ji8s<5j5b173>HSoyF<0HH)jLNL}DW zmu6gn4{X2MdK6s=eO_ms*1?RaUS{8a?$4i#ynpDs5x?Ar_kSgnzQ7qkYp?hU@88E6 zKZ3JZO+|AQay8kvHRB5_uyTGTgg}Q(_zhyfA{K!`&weX{z@RZZp1Ba$dKF?W*M|&$ z??*szrRzi5ivdkHnBZELEq7Z{L8(hSwr)x_JCk|(G37bC?_&W&slffB``CZ$dp`YL z#96#>_nTu0qSrhSO}=D|!2ILp6R+HQ3a<5fDBa?aWYQy7petu06_gAy zsnQrrH`LY(F+}S3z=`zwi`my-`1N3s*YDKx#2X*QwcZG&FLN0(=;vX0|IE6-zm@f2 zI3lUoR089KUeOG4@KE*vu# zjs+Y;)_uU3FIa);W?u#p+jBURlBQIqrK#S^e>FDL)pc)Zj-7V=2}d1r{NJLrNPJq8 zv*>{bMp20xP1QB0qH&Q?v!Z#6a|oxmSe#m~yGaD6&t}Y?_Y|)6HmYk}oR;5mD{$(+ z?zABu@G!}igC#_)0aIJEU$i+x-UR>=dX2R;RvH6;*e}|vuCliXEYEctb{4_%4hzfL z5API#Wya?#ccBU0yX=H+@dNh&mGv^AdsdlsFFos0_E{JHG)?4LUw(PnIlJLn?}Y{i zw&~hA-m7HSdogzd2`Hs2JdEMD|P zIo~@c;&ja9I{vTp#)8X`%oNAgfFqidL0fXx-j!l zIe>DH;`u$@3_3_u-I2XAF62FmU&jmC%^qM4&(Sg#MwvsZ48zCiYr?nFp)=#oO?^1A zw(~dC!2>%_#KVNQac|1OREC4@yKpF31hxU@JL+_h&=A2nm2as-FXOcDP&ka}RG2k; zVGbmlPBy;2wmNusr=^OotqvJ3i-WJN4ov0X%eU2`o5JMToAM}u15VcvY-7M#9KIua zrmwROuIrr1ch({BoyS!wswXZTSv6-rH&%PF^VZCOdT0b%=bI9l|)_ z+?MxD7KLrT`1Nh8J=Vaz4Ba)^l8luiXxw*l7@@b2yO;arp1+w}X2+2%{y67YIBqzGXJTzJ@#I z(6LeC8}5*QajD36+@UXn>$rS9cW`0HbOui+2^)Jm&t&i@;kIVm7Q6?n#Jvu^t7BKb z?+#g=Im(SY?6I7G$UU^&N^@u1x2t|em!d=5*GY@O( zV+IDcgCBuLD52ng2>ulGRZN7p2$x}56CH7ai8%h!!4%dE3