official/cv/unet2d/2dunet-adapter/README.md

@@ -0,0 +1,189 @@
+# U-Net: Semantic segmentation with PyTorch
+<a href="#"><img src="https://img.shields.io/github/actions/workflow/status/milesial/PyTorch-UNet/main.yml?logo=github&style=for-the-badge" /></a>
+<a href="https://hub.docker.com/r/milesial/unet"><img src="https://img.shields.io/badge/docker%20image-available-blue?logo=Docker&style=for-the-badge" /></a>
+<a href="https://pytorch.org/"><img src="https://img.shields.io/badge/PyTorch-v1.13+-red.svg?logo=PyTorch&style=for-the-badge" /></a>
+<a href="#"><img src="https://img.shields.io/badge/python-v3.6+-blue.svg?logo=python&style=for-the-badge" /></a>
+
+![input and output for a random image in the test dataset](https://i.imgur.com/GD8FcB7.png)
+
+
+Customized implementation of the [U-Net](https://arxiv.org/abs/1505.04597) in PyTorch for Kaggle's [Carvana Image Masking Challenge](https://www.kaggle.com/c/carvana-image-masking-challenge) from high definition images.
+
+- [Quick start](#quick-start)
+  - [Without Docker](#without-docker)
+  - [With Docker](#with-docker)
+- [Description](#description)
+- [Usage](#usage)
+  - [Docker](#docker)
+  - [Training](#training)
+  - [Prediction](#prediction)
+- [Weights & Biases](#weights--biases)
+- [Pretrained model](#pretrained-model)
+- [Data](#data)
+
+## Quick start
+
+### Without Docker
+
+1. [Install CUDA](https://developer.nvidia.com/cuda-downloads)
+
+2. [Install PyTorch 1.13 or later](https://pytorch.org/get-started/locally/)
+
+3. Install dependencies
+```bash
+pip install -r requirements.txt
+```
+
+4. Download the data and run training:
+```bash
+bash scripts/download_data.sh
+python train.py --amp
+```
+
+### With Docker
+
+1. [Install Docker 19.03 or later:](https://docs.docker.com/get-docker/)
+```bash
+curl https://get.docker.com | sh && sudo systemctl --now enable docker
+```
+2. [Install the NVIDIA container toolkit:](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)
+```bash
+distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
+   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
+   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
+sudo apt-get update
+sudo apt-get install -y nvidia-docker2
+sudo systemctl restart docker
+```
+3. [Download and run the image:](https://hub.docker.com/repository/docker/milesial/unet)
+```bash
+sudo docker run --rm --shm-size=8g --ulimit memlock=-1 --gpus all -it milesial/unet
+```
+
+4. Download the data and run training:
+```bash
+bash scripts/download_data.sh
+python train.py --amp
+```
+
+## Description
+This model was trained from scratch with 5k images and scored a [Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) of 0.988423 on over 100k test images.
+
+It can be easily used for multiclass segmentation, portrait segmentation, medical segmentation, ...
+
+
+## Usage
+**Note : Use Python 3.6 or newer**
+
+### Docker
+
+A docker image containing the code and the dependencies is available on [DockerHub](https://hub.docker.com/repository/docker/milesial/unet).
+You can download and jump in the container with ([docker >=19.03](https://docs.docker.com/get-docker/)):
+
+```console
+docker run -it --rm --shm-size=8g --ulimit memlock=-1 --gpus all milesial/unet
+```
+
+
+### Training
+
+```console
+> python train.py -h
+usage: train.py [-h] [--epochs E] [--batch-size B] [--learning-rate LR]
+                [--load LOAD] [--scale SCALE] [--validation VAL] [--amp]
+
+Train the UNet on images and target masks
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --epochs E, -e E      Number of epochs
+  --batch-size B, -b B  Batch size
+  --learning-rate LR, -l LR
+                        Learning rate
+  --load LOAD, -f LOAD  Load model from a .pth file
+  --scale SCALE, -s SCALE
+                        Downscaling factor of the images
+  --validation VAL, -v VAL
+                        Percent of the data that is used as validation (0-100)
+  --amp                 Use mixed precision
+```
+
+By default, the `scale` is 0.5, so if you wish to obtain better results (but use more memory), set it to 1.
+
+Automatic mixed precision is also available with the `--amp` flag. [Mixed precision](https://arxiv.org/abs/1710.03740) allows the model to use less memory and to be faster on recent GPUs by using FP16 arithmetic. Enabling AMP is recommended.
+
+
+### Prediction
+
+After training your model and saving it to `MODEL.pth`, you can easily test the output masks on your images via the CLI.
+
+To predict a single image and save it:
+
+`python predict.py -i image.jpg -o output.jpg`
+
+To predict a multiple images and show them without saving them:
+
+`python predict.py -i image1.jpg image2.jpg --viz --no-save`
+
+```console
+> python predict.py -h
+usage: predict.py [-h] [--model FILE] --input INPUT [INPUT ...] 
+                  [--output INPUT [INPUT ...]] [--viz] [--no-save]
+                  [--mask-threshold MASK_THRESHOLD] [--scale SCALE]
+
+Predict masks from input images
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --model FILE, -m FILE
+                        Specify the file in which the model is stored
+  --input INPUT [INPUT ...], -i INPUT [INPUT ...]
+                        Filenames of input images
+  --output INPUT [INPUT ...], -o INPUT [INPUT ...]
+                        Filenames of output images
+  --viz, -v             Visualize the images as they are processed
+  --no-save, -n         Do not save the output masks
+  --mask-threshold MASK_THRESHOLD, -t MASK_THRESHOLD
+                        Minimum probability value to consider a mask pixel white
+  --scale SCALE, -s SCALE
+                        Scale factor for the input images
+```
+You can specify which model file to use with `--model MODEL.pth`.
+
+## Weights & Biases
+
+The training progress can be visualized in real-time using [Weights & Biases](https://wandb.ai/).  Loss curves, validation curves, weights and gradient histograms, as well as predicted masks are logged to the platform.
+
+When launching a training, a link will be printed in the console. Click on it to go to your dashboard. If you have an existing W&B account, you can link it
+ by setting the `WANDB_API_KEY` environment variable. If not, it will create an anonymous run which is automatically deleted after 7 days.
+
+
+## Pretrained model
+A [pretrained model](https://github.com/milesial/Pytorch-UNet/releases/tag/v3.0) is available for the Carvana dataset. It can also be loaded from torch.hub:
+
+```python
+net = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana', pretrained=True, scale=0.5)
+```
+Available scales are 0.5 and 1.0.
+
+## Data
+The Carvana data is available on the [Kaggle website](https://www.kaggle.com/c/carvana-image-masking-challenge/data).
+
+You can also download it using the helper script:
+
+```
+bash scripts/download_data.sh
+```
+
+The input images and target masks should be in the `data/imgs` and `data/masks` folders respectively (note that the `imgs` and `masks` folder should not contain any sub-folder or any other files, due to the greedy data-loader). For Carvana, images are RGB and masks are black and white.
+
+You can use your own dataset as long as you make sure it is loaded properly in `utils/data_loading.py`.
+
+
+---
+
+Original paper by Olaf Ronneberger, Philipp Fischer, Thomas Brox:
+
+[U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597)
+
+![network architecture](https://i.imgur.com/jeDVpqF.png)

official/cv/unet2d/2dunet-adapter/requirements.txt

@@ -0,0 +1,5 @@
+matplotlib==3.6.2
+numpy==1.23.5
+Pillow==9.3.0
+tqdm==4.64.1
+wandb==0.13.5

official/cv/unet2d/2dunet-adapter/scripts/download_data.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+if [[ ! -f ~/.kaggle/kaggle.json ]]; then
+  echo -n "Kaggle username: "
+  read USERNAME
+  echo
+  echo -n "Kaggle API key: "
+  read APIKEY
+
+  mkdir -p ~/.kaggle
+  echo "{\"username\":\"$USERNAME\",\"key\":\"$APIKEY\"}" > ~/.kaggle/kaggle.json
+  chmod 600 ~/.kaggle/kaggle.json
+fi
+
+pip install kaggle --upgrade
+
+kaggle competitions download -c carvana-image-masking-challenge -f train_hq.zip
+unzip train_hq.zip
+mv train_hq/* data/imgs/
+rm -d train_hq
+rm train_hq.zip
+
+kaggle competitions download -c carvana-image-masking-challenge -f train_masks.zip
+unzip train_masks.zip
+mv train_masks/* data/masks/
+rm -d train_masks
+rm train_masks.zip

official/cv/unet2d/2dunet-adapter/train.py

@@ -0,0 +1,164 @@
+import argparse
+import logging
+import os
+os.environ['CUDA_VISIBLE_DEVICES'] = '1'
+import random
+import sys
+import ms_adapter.pytorch as torch
+import ms_adapter.pytorch.nn as nn
+import ms_adapter.pytorch.nn.functional as F
+import mindspore as ms
+
+import ms_adapter.torchvision.transforms as transforms
+import ms_adapter.torchvision.transforms.functional as TF
+from pathlib import Path
+from ms_adapter.pytorch import optim
+from ms_adapter.pytorch.utils.data import DataLoader, random_split
+from tqdm import tqdm
+
+from unet import UNet
+from utils.data_loading import BasicDataset, CarvanaDataset
+from utils.dice_score import dice_loss
+
+dir_img = Path('/code/UNet-master/2dunet-pytorch/data/imgs/')
+dir_mask = Path('/code/UNet-master/2dunet-pytorch/data/masks/')
+dir_checkpoint = Path('./checkpoints/')
+
+ms.set_context(device_target='GPU')
+
+
+def train_model(
+        model,
+        epochs: int = 5,
+        batch_size: int = 1,
+        learning_rate: float = 1e-5,
+        save_checkpoint: bool = True,
+        img_scale: float = 0.5,
+        weight_decay: float = 1e-8,
+        momentum: float = 0.999,
+):
+    # 1. Create dataset
+    try:
+        dataset = CarvanaDataset(dir_img, dir_mask, img_scale)
+        print('Load CarvanaDataset')
+    except (AssertionError, RuntimeError, IndexError):
+        dataset = BasicDataset(dir_img, dir_mask, img_scale)
+        print('Load BasicDataset')
+
+    n_train = len(dataset)
+
+    # 3. Create data loaders
+    loader_args = dict(batch_size=batch_size, num_workers=os.cpu_count())
+    train_loader = DataLoader(dataset, shuffle=True, **loader_args)
+    #val_loader = DataLoader(val_set, shuffle=False, drop_last=True, **loader_args)
+
+    # (Initialize logging)
+
+    logging.info(f'''Starting training:
+        Epochs:          {epochs}
+        Batch size:      {batch_size}
+        Learning rate:   {learning_rate}
+        Training size:   {n_train}
+        Checkpoints:     {save_checkpoint}
+        Images scaling:  {img_scale}
+    ''')
+
+    # 4. Set up the optimizer, the loss, the learning rate scheduler and the loss scaling for AMP
+    optimizer = ms.nn.RMSProp(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay)
+    criterion = nn.CrossEntropyLoss() if model.n_classes > 1 else nn.BCEWithLogitsLoss()
+    global_step = 0
+    
+    
+    def forward_fn(data, label):
+        masks_pred = model(data)
+        
+        if model.n_classes == 1:
+            loss = criterion(masks_pred.squeeze(1), true_masks.float())
+            loss += dice_loss(F.sigmoid(masks_pred.squeeze(1)), true_masks.float(), multiclass=False)
+        else:
+            loss = criterion(masks_pred, true_masks)
+            loss += dice_loss(F.softmax(masks_pred, dim=1).float(), 
+                              F.one_hot(true_masks, model.n_classes).permute(0, 3, 1, 2).float(), multiclass=True)
+            
+        return loss, masks_pred
+
+    grad_fn = ms.ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True)
+
+    def train_step(data, label):
+        (loss, _), grads = grad_fn(data, label)
+        loss = ms.ops.depend(loss, optimizer(grads))
+        return loss
+    
+    
+
+    # 5. Begin training
+    for epoch in range(1, epochs + 1):
+        model.train()
+        epoch_loss = 0
+        print('train_loader', len(train_loader), train_loader)
+        with tqdm(total=n_train, desc=f'Epoch {epoch}/{epochs}', unit='img') as pbar:   
+            for batch in train_loader:
+                images, true_masks = batch['image'], batch['mask']
+                assert images.shape[1] == model.n_channels, \
+                    f'Network has been defined with {model.n_channels} input channels, ' \
+                    f'but loaded images have {images.shape[1]} channels. Please check that ' \
+                    'the images are loaded correctly.'
+
+                images = images.to(dtype=torch.float32)
+                true_masks = true_masks.to(dtype=torch.long)
+
+                #masks_pred = model(images)
+                loss = train_step(images, true_masks)
+    
+                pbar.update(images.shape[0])
+                global_step += 1
+                epoch_loss += loss.item()
+                print(
+                    'train loss: %f' % loss.item(),
+                    'step: %d' % global_step,
+                    'epoch: %d' % epoch
+                )
+                pbar.set_postfix(**{'loss (batch)': loss.item()})
+
+                
+
+
+def get_args():
+    parser = argparse.ArgumentParser(description='Train the UNet on images and target masks')
+    parser.add_argument('--epochs', '-e', metavar='E', type=int, default=5, help='Number of epochs')
+    parser.add_argument('--batch-size', '-b', dest='batch_size', metavar='B', type=int, default=1, help='Batch size')
+    parser.add_argument('--learning-rate', '-l', metavar='LR', type=float, default=1e-5,
+                        help='Learning rate', dest='lr')
+    parser.add_argument('--load', '-f', type=str, default=False, help='Load model from a .pth file')
+    parser.add_argument('--scale', '-s', type=float, default=0.5, help='Downscaling factor of the images')
+    parser.add_argument('--validation', '-v', dest='val', type=float, default=10.0,
+                        help='Percent of the data that is used as validation (0-100)')
+    parser.add_argument('--bilinear', action='store_true', default=False, help='Use bilinear upsampling')
+    parser.add_argument('--classes', '-c', type=int, default=2, help='Number of classes')
+
+    return parser.parse_args()
+
+
+if __name__ == '__main__':
+    args = get_args()
+
+    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
+
+    # Change here to adapt to your data
+    # n_channels=3 for RGB images
+    # n_classes is the number of probabilities you want to get per pixel
+    model = UNet(n_channels=3, n_classes=args.classes, bilinear=args.bilinear)
+    
+    logging.info(f'Network:\n'
+                 f'\t{model.n_channels} input channels\n'
+                 f'\t{model.n_classes} output channels (classes)\n'
+                 f'\t{"Bilinear" if model.bilinear else "Transposed conv"} upscaling')
+
+    
+    train_model(
+        model=model,
+        epochs=args.epochs,
+        batch_size=args.batch_size,
+        learning_rate=args.lr,
+        img_scale=args.scale,
+        )

official/cv/unet2d/2dunet-adapter/unet/__init__.py

@@ -0,0 +1 @@
+from .unet_model import UNet

official/cv/unet2d/2dunet-adapter/unet/unet_model.py

@@ -0,0 +1,48 @@
+""" Full assembly of the parts to form the complete network """
+
+from .unet_parts import *
+
+
+class UNet(nn.Module):
+    def __init__(self, n_channels, n_classes, bilinear=False):
+        super(UNet, self).__init__()
+        self.n_channels = n_channels
+        self.n_classes = n_classes
+        self.bilinear = bilinear
+
+        self.inc = (DoubleConv(n_channels, 64))
+        self.down1 = (Down(64, 128))
+        self.down2 = (Down(128, 256))
+        self.down3 = (Down(256, 512))
+        factor = 2 if bilinear else 1
+        self.down4 = (Down(512, 1024 // factor))
+        self.up1 = (Up(1024, 512 // factor, bilinear))
+        self.up2 = (Up(512, 256 // factor, bilinear))
+        self.up3 = (Up(256, 128 // factor, bilinear))
+        self.up4 = (Up(128, 64, bilinear))
+        self.outc = (OutConv(64, n_classes))
+
+    def forward(self, x):
+        x1 = self.inc(x)
+        x2 = self.down1(x1)
+        x3 = self.down2(x2)
+        x4 = self.down3(x3)
+        x5 = self.down4(x4)
+        x = self.up1(x5, x4)
+        x = self.up2(x, x3)
+        x = self.up3(x, x2)
+        x = self.up4(x, x1)
+        logits = self.outc(x)
+        return logits
+
+    def use_checkpointing(self):
+        self.inc = torch.utils.checkpoint(self.inc)
+        self.down1 = torch.utils.checkpoint(self.down1)
+        self.down2 = torch.utils.checkpoint(self.down2)
+        self.down3 = torch.utils.checkpoint(self.down3)
+        self.down4 = torch.utils.checkpoint(self.down4)
+        self.up1 = torch.utils.checkpoint(self.up1)
+        self.up2 = torch.utils.checkpoint(self.up2)
+        self.up3 = torch.utils.checkpoint(self.up3)
+        self.up4 = torch.utils.checkpoint(self.up4)
+        self.outc = torch.utils.checkpoint(self.outc)

official/cv/unet2d/2dunet-adapter/unet/unet_parts.py

@@ -0,0 +1,77 @@
+""" Parts of the U-Net model """
+
+import ms_adapter.pytorch as torch
+import ms_adapter.pytorch.nn as nn
+import ms_adapter.pytorch.nn.functional as F
+
+
+class DoubleConv(nn.Module):
+    """(convolution => [BN] => ReLU) * 2"""
+
+    def __init__(self, in_channels, out_channels, mid_channels=None):
+        super().__init__()
+        if not mid_channels:
+            mid_channels = out_channels
+        self.double_conv = nn.Sequential(
+            nn.Conv2d(in_channels, mid_channels, kernel_size=3, padding=1, bias=False),
+            nn.BatchNorm2d(mid_channels),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(mid_channels, out_channels, kernel_size=3, padding=1, bias=False),
+            nn.BatchNorm2d(out_channels),
+            nn.ReLU(inplace=True)
+        )
+
+    def forward(self, x):
+        return self.double_conv(x)
+
+
+class Down(nn.Module):
+    """Downscaling with maxpool then double conv"""
+
+    def __init__(self, in_channels, out_channels):
+        super().__init__()
+        self.maxpool_conv = nn.Sequential(
+            nn.MaxPool2d(2),
+            DoubleConv(in_channels, out_channels)
+        )
+
+    def forward(self, x):
+        return self.maxpool_conv(x)
+
+
+class Up(nn.Module):
+    """Upscaling then double conv"""
+
+    def __init__(self, in_channels, out_channels, bilinear=True):
+        super().__init__()
+
+        # if bilinear, use the normal convolutions to reduce the number of channels
+        if bilinear:
+            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
+            self.conv = DoubleConv(in_channels, out_channels, in_channels // 2)
+        else:
+            self.up = nn.ConvTranspose2d(in_channels, in_channels // 2, kernel_size=2, stride=2)
+            self.conv = DoubleConv(in_channels, out_channels)
+
+    def forward(self, x1, x2):
+        x1 = self.up(x1)
+        # input is CHW
+        diffY = x2.size()[2] - x1.size()[2]
+        diffX = x2.size()[3] - x1.size()[3]
+
+        x1 = F.pad(x1, [diffX // 2, diffX - diffX // 2,
+                        diffY // 2, diffY - diffY // 2])
+        # if you have padding issues, see
+        # https://github.com/HaiyongJiang/U-Net-Pytorch-Unstructured-Buggy/commit/0e854509c2cea854e247a9c615f175f76fbb2e3a
+        # https://github.com/xiaopeng-liao/Pytorch-UNet/commit/8ebac70e633bac59fc22bb5195e513d5832fb3bd
+        x = torch.cat([x2, x1], dim=1)
+        return self.conv(x)
+
+
+class OutConv(nn.Module):
+    def __init__(self, in_channels, out_channels):
+        super(OutConv, self).__init__()
+        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1)
+
+    def forward(self, x):
+        return self.conv(x)

official/cv/unet2d/2dunet-adapter/utils/data_loading.py

@@ -0,0 +1,117 @@
+import logging
+import numpy as np
+import ms_adapter.pytorch as torch
+from PIL import Image
+from functools import lru_cache
+from functools import partial
+from itertools import repeat
+from multiprocessing import Pool
+from os import listdir
+from os.path import splitext, isfile, join
+from pathlib import Path
+from ms_adapter.pytorch.utils.data import Dataset
+from tqdm import tqdm
+
+
+def load_image(filename):
+    ext = splitext(filename)[1]
+    if ext == '.npy':
+        return Image.fromarray(np.load(filename))
+    elif ext in ['.pt', '.pth']:
+        return Image.fromarray(torch.load(filename).numpy())
+    else:
+        return Image.open(filename)
+
+
+def unique_mask_values(idx, mask_dir, mask_suffix):
+    mask_file = list(mask_dir.glob(idx + mask_suffix + '.*'))[0]
+    mask = np.asarray(load_image(mask_file))
+    if mask.ndim == 2:
+        return np.unique(mask)
+    elif mask.ndim == 3:
+        mask = mask.reshape(-1, mask.shape[-1])
+        return np.unique(mask, axis=0)
+    else:
+        raise ValueError(f'Loaded masks should have 2 or 3 dimensions, found {mask.ndim}')
+
+
+class BasicDataset(Dataset):
+    def __init__(self, images_dir: str, mask_dir: str, scale: float = 1.0, mask_suffix: str = ''):
+        self.images_dir = Path(images_dir)
+        self.mask_dir = Path(mask_dir)
+        assert 0 < scale <= 1, 'Scale must be between 0 and 1'
+        self.scale = scale
+        self.mask_suffix = mask_suffix
+
+        self.ids = [splitext(file)[0] for file in listdir(images_dir) if isfile(join(images_dir, file)) and not file.startswith('.')]
+        if not self.ids:
+            raise RuntimeError(f'No input file found in {images_dir}, make sure you put your images there')
+
+        logging.info(f'Creating dataset with {len(self.ids)} examples')
+        logging.info('Scanning mask files to determine unique values')
+        with Pool() as p:
+            unique = list(tqdm(
+                p.imap(partial(unique_mask_values, mask_dir=self.mask_dir, mask_suffix=self.mask_suffix), self.ids),
+                total=len(self.ids)
+            ))
+
+        self.mask_values = list(sorted(np.unique(np.concatenate(unique), axis=0).tolist()))
+        logging.info(f'Unique mask values: {self.mask_values}')
+
+    def __len__(self):
+        return len(self.ids)
+
+    @staticmethod
+    def preprocess(mask_values, pil_img, scale, is_mask):
+        w, h = pil_img.size
+        newW, newH = int(scale * w), int(scale * h)
+        assert newW > 0 and newH > 0, 'Scale is too small, resized images would have no pixel'
+        pil_img = pil_img.resize((newW, newH), resample=Image.NEAREST if is_mask else Image.BICUBIC)
+        img = np.asarray(pil_img)
+
+        if is_mask:
+            mask = np.zeros((newH, newW), dtype=np.int64)
+            for i, v in enumerate(mask_values):
+                if img.ndim == 2:
+                    mask[img == v] = i
+                else:
+                    mask[(img == v).all(-1)] = i
+
+            return mask
+
+        else:
+            if img.ndim == 2:
+                img = img[np.newaxis, ...]
+            else:
+                img = img.transpose((2, 0, 1))
+
+            if (img > 1).any():
+                img = img / 255.0
+
+            return img
+
+    def __getitem__(self, idx):
+        name = self.ids[idx]
+        mask_file = list(self.mask_dir.glob(name + self.mask_suffix + '.*'))
+        img_file = list(self.images_dir.glob(name + '.*'))
+
+        assert len(img_file) == 1, f'Either no image or multiple images found for the ID {name}: {img_file}'
+        assert len(mask_file) == 1, f'Either no mask or multiple masks found for the ID {name}: {mask_file}'
+        mask = load_image(mask_file[0])
+        img = load_image(img_file[0])
+
+        assert img.size == mask.size, \
+            f'Image and mask {name} should be the same size, but are {img.size} and {mask.size}'
+
+        img = self.preprocess(self.mask_values, img, self.scale, is_mask=False)
+        mask = self.preprocess(self.mask_values, mask, self.scale, is_mask=True)
+
+        return {
+            'image': torch.as_tensor(img.copy()).float().contiguous(),
+            'mask': torch.as_tensor(mask.copy()).long().contiguous()
+        }
+
+
+class CarvanaDataset(BasicDataset):
+    def __init__(self, images_dir, mask_dir, scale=1):
+        super().__init__(images_dir, mask_dir, scale, mask_suffix='_mask')

official/cv/unet2d/2dunet-adapter/utils/dice_score.py

@@ -0,0 +1,28 @@
+import ms_adapter.pytorch as torch
+from ms_adapter.pytorch import Tensor
+
+
+def dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6):
+    # Average of Dice coefficient for all batches, or for a single mask
+    assert input.size() == target.size()
+    assert input.dim() == 3 or not reduce_batch_first
+
+    sum_dim = (-1, -2) if input.dim() == 2 or not reduce_batch_first else (-1, -2, -3)
+
+    inter = 2 * (input * target).sum(dim=sum_dim)
+    sets_sum = input.sum(dim=sum_dim) + target.sum(dim=sum_dim)
+    sets_sum = torch.where(sets_sum == 0, inter, sets_sum)
+
+    dice = (inter + epsilon) / (sets_sum + epsilon)
+    return dice.mean()
+
+
+def multiclass_dice_coeff(input: Tensor, target: Tensor, reduce_batch_first: bool = False, epsilon: float = 1e-6):
+    # Average of Dice coefficient for all classes
+    return dice_coeff(input.flatten(0, 1), target.flatten(0, 1), reduce_batch_first, epsilon)
+
+
+def dice_loss(input: Tensor, target: Tensor, multiclass: bool = False):
+    # Dice loss (objective to minimize) between 0 and 1
+    fn = multiclass_dice_coeff if multiclass else dice_coeff
+    return 1 - fn(input, target, reduce_batch_first=True)

official/cv/unet2d/2dunet-adapter/utils/utils.py

@@ -0,0 +1,13 @@
+import matplotlib.pyplot as plt
+
+
+def plot_img_and_mask(img, mask):
+    classes = mask.max() + 1
+    fig, ax = plt.subplots(1, classes + 1)
+    ax[0].set_title('Input image')
+    ax[0].imshow(img)
+    for i in range(classes):
+        ax[i + 1].set_title(f'Mask (class {i + 1})')
+        ax[i + 1].imshow(mask == i)
+    plt.xticks([]), plt.yticks([])
+    plt.show()

official/cv/unet2d/2dunet-adapter/wandb/debug-internal.log

@@ -0,0 +1,312 @@
+2023-03-10 00:10:05,030 INFO    StreamThr :17139 [internal.py:wandb_internal():90] W&B internal server running at pid: 17139, started at: 2023-03-10 00:10:05.028238
+2023-03-10 00:10:05,031 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status
+2023-03-10 00:10:05,033 INFO    WriterThread:17139 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb
+2023-03-10 00:10:05,038 DEBUG   SenderThread:17139 [sender.py:send():336] send: header
+2023-03-10 00:10:05,038 DEBUG   SenderThread:17139 [sender.py:send():336] send: run
+2023-03-10 00:10:05,174 INFO    SenderThread:17139 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/215re05m
+2023-03-10 00:10:06,363 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: check_version
+2023-03-10 00:10:06,364 INFO    SenderThread:17139 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files
+2023-03-10 00:10:06,364 INFO    SenderThread:17139 [sender.py:_start_run_threads():1081] run started: 215re05m with start time 1678378205.024339
+2023-03-10 00:10:06,364 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: summary_record
+2023-03-10 00:10:06,422 INFO    SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-10 00:10:06,422 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: check_version
+2023-03-10 00:10:07,368 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json
+2023-03-10 00:10:11,365 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:11,423 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:11,444 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: run_start
+2023-03-10 00:10:11,449 DEBUG   HandlerThread:17139 [system_info.py:__init__():31] System info init
+2023-03-10 00:10:11,449 DEBUG   HandlerThread:17139 [system_info.py:__init__():46] System info init done
+2023-03-10 00:10:11,449 INFO    HandlerThread:17139 [system_monitor.py:start():183] Starting system monitor
+2023-03-10 00:10:11,449 INFO    SystemMonitor:17139 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-10 00:10:11,449 INFO    HandlerThread:17139 [system_monitor.py:probe():204] Collecting system info
+2023-03-10 00:10:11,450 INFO    SystemMonitor:17139 [interfaces.py:start():187] Started cpu monitoring
+2023-03-10 00:10:11,451 INFO    SystemMonitor:17139 [interfaces.py:start():187] Started disk monitoring
+2023-03-10 00:10:11,451 INFO    SystemMonitor:17139 [interfaces.py:start():187] Started gpu monitoring
+2023-03-10 00:10:11,452 INFO    SystemMonitor:17139 [interfaces.py:start():187] Started memory monitoring
+2023-03-10 00:10:11,452 INFO    SystemMonitor:17139 [interfaces.py:start():187] Started network monitoring
+2023-03-10 00:10:11,489 DEBUG   HandlerThread:17139 [system_info.py:probe():195] Probing system
+2023-03-10 00:10:11,516 DEBUG   HandlerThread:17139 [git.py:repo():40] git repository is invalid
+2023-03-10 00:10:11,516 DEBUG   HandlerThread:17139 [system_info.py:probe():240] Probing system done
+2023-03-10 00:10:11,516 DEBUG   HandlerThread:17139 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T16:10:11.489682', 'startedAt': '2023-03-09T16:10:04.972918', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.301262500001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.047, 'min': 0.0, 'max': 0.0}, {'current': 2505.411, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2497.435, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.275, 'min': 0.0, 'max': 0.0}, {'current': 2497.673, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2501.929, 'min': 0.0, 'max': 0.0}, {'current': 2500.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.824, 'min': 0.0, 'max': 0.0}, {'current': 2498.743, 'min': 0.0, 'max': 0.0}, {'current': 2498.804, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2501.118, 'min': 0.0, 'max': 0.0}, {'current': 2502.038, 'min': 0.0, 'max': 0.0}, {'current': 2499.891, 'min': 0.0, 'max': 0.0}, {'current': 2500.327, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2499.951, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.433, 'min': 0.0, 'max': 0.0}, {'current': 2501.418, 'min': 0.0, 'max': 0.0}, {'current': 2501.066, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.969, 'min': 0.0, 'max': 0.0}, {'current': 2499.963, 'min': 0.0, 'max': 0.0}, {'current': 2499.319, 'min': 0.0, 'max': 0.0}, {'current': 2499.62, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2502.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2504.876, 'min': 0.0, 'max': 0.0}, {'current': 2499.546, 'min': 0.0, 'max': 0.0}, {'current': 2504.283, 'min': 0.0, 'max': 0.0}, {'current': 2506.421, 'min': 0.0, 'max': 0.0}, {'current': 2499.741, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2498.673, 'min': 0.0, 'max': 0.0}, {'current': 2505.326, 'min': 0.0, 'max': 0.0}, {'current': 2504.107, 'min': 0.0, 'max': 0.0}, {'current': 2505.212, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2509.28, 'min': 0.0, 'max': 0.0}, {'current': 2500.431, 'min': 0.0, 'max': 0.0}, {'current': 2499.538, 'min': 0.0, 'max': 0.0}, {'current': 2500.876, 'min': 0.0, 'max': 0.0}, {'current': 2503.961, 'min': 0.0, 'max': 0.0}, {'current': 2505.357, 'min': 0.0, 'max': 0.0}, {'current': 2500.3, 'min': 0.0, 'max': 0.0}, {'current': 2500.458, 'min': 0.0, 'max': 0.0}, {'current': 2499.894, 'min': 0.0, 'max': 0.0}, {'current': 2502.906, 'min': 0.0, 'max': 0.0}, {'current': 2499.885, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2504.513, 'min': 0.0, 'max': 0.0}, {'current': 2504.644, 'min': 0.0, 'max': 0.0}, {'current': 2503.685, 'min': 0.0, 'max': 0.0}, {'current': 2500.655, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.238, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.672, 'min': 0.0, 'max': 0.0}, {'current': 2504.909, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.59, 'min': 0.0, 'max': 0.0}, {'current': 2500.478, 'min': 0.0, 'max': 0.0}, {'current': 2500.85, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6511535644531}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-10 00:10:11,517 INFO    HandlerThread:17139 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-10 00:10:11,517 INFO    HandlerThread:17139 [system_monitor.py:probe():217] Publishing system info
+2023-03-10 00:10:11,517 DEBUG   HandlerThread:17139 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-10 00:10:11,546 DEBUG   HandlerThread:17139 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-10 00:10:11,546 DEBUG   HandlerThread:17139 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-10 00:10:12,268 DEBUG   HandlerThread:17139 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-10 00:10:12,302 INFO    HandlerThread:17139 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-10 00:10:12,305 DEBUG   SenderThread:17139 [sender.py:send():336] send: files
+2023-03-10 00:10:12,306 INFO    SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-10 00:10:12,320 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-10 00:10:12,333 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: stop_status
+2023-03-10 00:10:12,546 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml
+2023-03-10 00:10:12,546 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt
+2023-03-10 00:10:12,546 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json
+2023-03-10 00:10:12,800 DEBUG   SenderThread:17139 [sender.py:send():336] send: telemetry
+2023-03-10 00:10:12,800 DEBUG   SenderThread:17139 [sender.py:send():336] send: config
+2023-03-10 00:10:13,506 INFO    wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /tmp/tmpf905ug1_wandb/11iue9v6-wandb-metadata.json
+2023-03-10 00:10:13,547 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log
+2023-03-10 00:10:15,549 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log
+2023-03-10 00:10:16,804 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:21,805 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:26,806 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:27,314 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:27,314 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:31,806 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:32,316 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:36,812 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:37,317 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:37,569 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml
+2023-03-10 00:10:42,303 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:42,319 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:47,304 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:47,320 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:52,305 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:52,322 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:10:57,305 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:10:57,323 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:02,306 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:02,324 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:07,307 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:07,325 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:11,452 DEBUG   SystemMonitor:17139 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-10 00:11:11,458 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:11:12,326 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:12,459 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:17,327 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:17,460 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:22,329 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:22,461 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:27,330 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:27,461 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:32,331 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:32,462 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:37,332 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:37,463 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:41,460 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:11:42,333 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:43,462 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:47,334 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:48,463 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:52,336 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:53,464 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:11:57,337 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:11:58,464 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:02,338 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:03,465 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:07,339 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:08,466 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:11,463 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:12:12,340 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:14,465 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:17,342 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:19,466 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:22,343 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:24,467 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:27,344 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:29,467 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:32,345 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:34,468 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:37,346 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:39,469 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:41,466 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:12:42,348 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:45,468 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:47,349 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:50,469 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:52,350 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:12:55,469 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:12:57,351 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:00,470 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:02,352 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:05,471 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:07,353 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:10,471 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:11,469 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:13:12,355 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:16,470 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:17,356 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:21,471 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:22,357 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:26,472 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:27,358 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:31,472 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:32,359 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:36,473 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:37,360 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:41,548 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:41,550 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:13:42,362 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:46,552 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:47,373 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:51,552 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:52,375 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:13:56,553 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:13:57,376 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:01,554 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:02,377 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:06,554 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:07,378 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:11,553 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:14:12,379 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:12,554 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:17,381 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:17,555 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:22,382 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:22,556 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:27,383 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:27,557 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:32,384 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:32,557 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:37,386 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:37,558 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:41,555 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:14:42,387 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:43,557 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:47,388 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:48,557 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:52,389 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:53,558 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:14:57,390 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:14:58,559 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:02,391 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:15:03,559 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:07,393 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:15:08,560 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:11,558 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:15:12,394 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:15:14,560 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:17,409 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:15:18,419 DEBUG   SenderThread:17139 [sender.py:send():336] send: exit
+2023-03-10 00:15:18,419 INFO    SenderThread:17139 [sender.py:send_exit():559] handling exit code: 1
+2023-03-10 00:15:18,419 INFO    SenderThread:17139 [sender.py:send_exit():561] handling runtime: 306
+2023-03-10 00:15:18,723 INFO    SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-10 00:15:18,724 INFO    SenderThread:17139 [sender.py:send_exit():567] send defer
+2023-03-10 00:15:18,724 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:18,725 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 0
+2023-03-10 00:15:18,725 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:18,725 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 0
+2023-03-10 00:15:18,725 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 1
+2023-03-10 00:15:18,726 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:18,726 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 1
+2023-03-10 00:15:18,726 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:18,726 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 1
+2023-03-10 00:15:18,726 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 2
+2023-03-10 00:15:18,808 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:18,808 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 2
+2023-03-10 00:15:18,808 INFO    HandlerThread:17139 [system_monitor.py:finish():193] Stopping system monitor
+2023-03-10 00:15:19,020 DEBUG   SystemMonitor:17139 [system_monitor.py:_start():168] Finished system metrics aggregation loop
+2023-03-10 00:15:19,021 DEBUG   SystemMonitor:17139 [system_monitor.py:_start():172] Publishing last batch of metrics
+2023-03-10 00:15:19,109 INFO    HandlerThread:17139 [interfaces.py:finish():199] Joined cpu monitor
+2023-03-10 00:15:19,109 INFO    HandlerThread:17139 [interfaces.py:finish():199] Joined disk monitor
+2023-03-10 00:15:19,110 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json
+2023-03-10 00:15:20,110 INFO    HandlerThread:17139 [interfaces.py:finish():199] Joined gpu monitor
+2023-03-10 00:15:20,111 INFO    HandlerThread:17139 [interfaces.py:finish():199] Joined memory monitor
+2023-03-10 00:15:20,111 INFO    HandlerThread:17139 [interfaces.py:finish():199] Joined network monitor
+2023-03-10 00:15:20,111 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-10 00:15:20,111 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:20,113 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,114 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 2
+2023-03-10 00:15:20,114 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 3
+2023-03-10 00:15:20,114 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:20,114 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 3
+2023-03-10 00:15:20,208 DEBUG   SenderThread:17139 [sender.py:send():336] send: stats
+2023-03-10 00:15:20,209 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit
+2023-03-10 00:15:20,210 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,210 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 3
+2023-03-10 00:15:20,210 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 4
+2023-03-10 00:15:20,211 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:20,211 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log
+2023-03-10 00:15:20,211 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 4
+2023-03-10 00:15:20,211 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,211 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 4
+2023-03-10 00:15:20,212 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 5
+2023-03-10 00:15:20,212 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:20,212 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 5
+2023-03-10 00:15:20,212 DEBUG   SenderThread:17139 [sender.py:send():336] send: summary
+2023-03-10 00:15:20,233 INFO    SenderThread:17139 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-10 00:15:20,233 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,233 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 5
+2023-03-10 00:15:20,233 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 6
+2023-03-10 00:15:20,234 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:20,234 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 6
+2023-03-10 00:15:20,234 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,234 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 6
+2023-03-10 00:15:20,234 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 7
+2023-03-10 00:15:20,234 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:20,234 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:20,234 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 7
+2023-03-10 00:15:20,235 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:20,235 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 7
+2023-03-10 00:15:20,511 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-10 00:15:21,211 INFO    Thread-13 :17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json
+2023-03-10 00:15:23,339 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 8
+2023-03-10 00:15:23,339 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit
+2023-03-10 00:15:23,339 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:23,340 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 8
+2023-03-10 00:15:23,340 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:23,340 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 8
+2023-03-10 00:15:23,341 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 9
+2023-03-10 00:15:23,341 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:23,341 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 9
+2023-03-10 00:15:23,341 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:23,341 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 9
+2023-03-10 00:15:23,341 INFO    SenderThread:17139 [dir_watcher.py:finish():365] shutting down directory watcher
+2023-03-10 00:15:24,213 INFO    SenderThread:17139 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log
+2023-03-10 00:15:24,214 INFO    SenderThread:17139 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files
+2023-03-10 00:15:24,214 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/conda-environment.yaml conda-environment.yaml
+2023-03-10 00:15:24,214 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log output.log
+2023-03-10 00:15:24,215 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt requirements.txt
+2023-03-10 00:15:24,220 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml config.yaml
+2023-03-10 00:15:24,241 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json wandb-summary.json
+2023-03-10 00:15:24,256 INFO    SenderThread:17139 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-metadata.json wandb-metadata.json
+2023-03-10 00:15:24,261 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 10
+2023-03-10 00:15:24,277 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:24,287 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 10
+2023-03-10 00:15:24,419 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:24,424 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 10
+2023-03-10 00:15:24,424 INFO    SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-10 00:15:25,468 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: status_report
+2023-03-10 00:15:25,536 INFO    wandb-upload_4:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/wandb-summary.json
+2023-03-10 00:15:25,612 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-10 00:15:25,889 INFO    wandb-upload_0:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/output.log
+2023-03-10 00:15:25,909 INFO    wandb-upload_2:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/requirements.txt
+2023-03-10 00:15:26,005 INFO    wandb-upload_3:17139 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/files/config.yaml
+2023-03-10 00:15:26,205 INFO    Thread-12 :17139 [sender.py:transition_state():587] send defer: 11
+2023-03-10 00:15:26,206 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:26,206 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 11
+2023-03-10 00:15:26,206 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:26,206 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 11
+2023-03-10 00:15:26,206 INFO    SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher
+2023-03-10 00:15:26,206 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 12
+2023-03-10 00:15:26,206 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:26,206 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 12
+2023-03-10 00:15:26,207 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:26,207 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 12
+2023-03-10 00:15:28,352 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 13
+2023-03-10 00:15:28,353 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:28,353 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 13
+2023-03-10 00:15:28,353 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:28,353 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 13
+2023-03-10 00:15:28,353 INFO    SenderThread:17139 [sender.py:transition_state():587] send defer: 14
+2023-03-10 00:15:28,354 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: defer
+2023-03-10 00:15:28,354 INFO    HandlerThread:17139 [handler.py:handle_request_defer():170] handle defer: 14
+2023-03-10 00:15:28,354 DEBUG   SenderThread:17139 [sender.py:send():336] send: final
+2023-03-10 00:15:28,354 DEBUG   SenderThread:17139 [sender.py:send():336] send: footer
+2023-03-10 00:15:28,354 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: defer
+2023-03-10 00:15:28,354 INFO    SenderThread:17139 [sender.py:send_request_defer():583] handle sender defer: 14
+2023-03-10 00:15:28,355 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-10 00:15:28,355 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: poll_exit
+2023-03-10 00:15:28,356 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: server_info
+2023-03-10 00:15:28,356 DEBUG   SenderThread:17139 [sender.py:send_request():363] send_request: server_info
+2023-03-10 00:15:28,392 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: get_summary
+2023-03-10 00:15:28,438 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: sampled_history
+2023-03-10 00:15:29,089 INFO    MainThread:17139 [wandb_run.py:_footer_history_summary_info():3429] rendering history
+2023-03-10 00:15:29,089 INFO    MainThread:17139 [wandb_run.py:_footer_history_summary_info():3461] rendering summary
+2023-03-10 00:15:29,089 INFO    MainThread:17139 [wandb_run.py:_footer_sync_info():3387] logging synced files
+2023-03-10 00:15:29,089 DEBUG   HandlerThread:17139 [handler.py:handle_request():144] handle_request: shutdown
+2023-03-10 00:15:29,090 INFO    HandlerThread:17139 [handler.py:finish():842] shutting down handler
+2023-03-10 00:15:29,356 INFO    WriterThread:17139 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/run-215re05m.wandb
+2023-03-10 00:15:30,089 INFO    SenderThread:17139 [sender.py:finish():1504] shutting down sender
+2023-03-10 00:15:30,089 INFO    SenderThread:17139 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-10 00:15:30,089 INFO    SenderThread:17139 [file_pusher.py:join():169] waiting for file pusher

official/cv/unet2d/2dunet-adapter/wandb/debug.log

@@ -0,0 +1,28 @@
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Configure stats pid to 15417
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug.log
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230310_001004-215re05m/logs/debug-internal.log
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:init():546] calling init triggers
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:init():602] starting backend
+2023-03-10 00:10:05,010 INFO    MainThread:15417 [wandb_init.py:init():606] setting up manager
+2023-03-10 00:10:05,022 INFO    MainThread:15417 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-10 00:10:05,024 INFO    MainThread:15417 [wandb_init.py:init():613] backend started and connected
+2023-03-10 00:10:05,028 INFO    MainThread:15417 [wandb_init.py:init():701] updated telemetry
+2023-03-10 00:10:05,029 INFO    MainThread:15417 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-10 00:10:06,363 INFO    MainThread:15417 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-10 00:10:11,424 INFO    MainThread:15417 [wandb_run.py:_on_init():2140] got version response 
+2023-03-10 00:10:11,424 INFO    MainThread:15417 [wandb_init.py:init():789] starting run threads in backend
+2023-03-10 00:10:12,313 INFO    MainThread:15417 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-10 00:10:12,313 INFO    MainThread:15417 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-10 00:10:12,313 INFO    MainThread:15417 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-10 00:10:12,313 INFO    MainThread:15417 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-10 00:10:12,314 INFO    MainThread:15417 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-10 00:10:12,315 INFO    MainThread:15417 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}
+2023-03-10 00:15:30,109 WARNING MsgRouterThr:15417 [router.py:message_loop():77] message_loop has been closed

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml

@@ -0,0 +1,40 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678374242.245928
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/output.log

@@ -0,0 +1,6 @@
+Traceback (most recent call last):
+  File "train.py", line 170, in <module>
+    img_scale=args.scale,
+  File "train.py", line 68, in train_model
+    ''')
+NameError: name 'n_val' is not defined

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:04:08.672479",
+    "startedAt": "2023-03-09T15:04:02.165764",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.5035375,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2501.774,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.632,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.844,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.227,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.25,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.66,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.269,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.584,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.412,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.957,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.437,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.395,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.827,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.381,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.589,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.571,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.823,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.653,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2494.369,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.905,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.341,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.224,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.123,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.567,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.922,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.487,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.288,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.937,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.938,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.099,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.616,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.366,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.679,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.537,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.027,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.211,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.78,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.362,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.002,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.673,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.016,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.354,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.742,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.466,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.896,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.131,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.496,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.341,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.252,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.01,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.34,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.367,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.053,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.115,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.91,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.027,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.852,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.801,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.094,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.488,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.156,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2497.755,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.669,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.714,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.401,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.973,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.132,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.65414810180664
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json

@@ -0,0 +1 @@
+{"_wandb": {"runtime": 1}}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log

@@ -0,0 +1,209 @@
+2023-03-09 23:04:02,252 INFO    StreamThr :5402 [internal.py:wandb_internal():90] W&B internal server running at pid: 5402, started at: 2023-03-09 23:04:02.250393
+2023-03-09 23:04:02,254 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:04:02,255 INFO    WriterThread:5402 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb
+2023-03-09 23:04:02,261 DEBUG   SenderThread:5402 [sender.py:send():336] send: header
+2023-03-09 23:04:02,261 DEBUG   SenderThread:5402 [sender.py:send():336] send: run
+2023-03-09 23:04:02,318 INFO    SenderThread:5402 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/tfw61kps
+2023-03-09 23:04:03,555 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:04:03,556 INFO    SenderThread:5402 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files
+2023-03-09 23:04:03,556 INFO    SenderThread:5402 [sender.py:_start_run_threads():1081] run started: tfw61kps with start time 1678374242.245928
+2023-03-09 23:04:03,556 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:04:03,606 INFO    SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:04:03,606 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:04:04,560 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json
+2023-03-09 23:04:08,556 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:04:08,608 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:04:08,626 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:04:08,633 DEBUG   HandlerThread:5402 [system_info.py:__init__():31] System info init
+2023-03-09 23:04:08,633 DEBUG   HandlerThread:5402 [system_info.py:__init__():46] System info init done
+2023-03-09 23:04:08,633 INFO    HandlerThread:5402 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:04:08,634 INFO    SystemMonitor:5402 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:04:08,634 INFO    HandlerThread:5402 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:04:08,634 INFO    SystemMonitor:5402 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:04:08,635 INFO    SystemMonitor:5402 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:04:08,635 INFO    SystemMonitor:5402 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:04:08,636 INFO    SystemMonitor:5402 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:04:08,636 INFO    SystemMonitor:5402 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:04:08,672 DEBUG   HandlerThread:5402 [system_info.py:probe():195] Probing system
+2023-03-09 23:04:08,697 DEBUG   HandlerThread:5402 [git.py:repo():40] git repository is invalid
+2023-03-09 23:04:08,697 DEBUG   HandlerThread:5402 [system_info.py:probe():240] Probing system done
+2023-03-09 23:04:08,697 DEBUG   HandlerThread:5402 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:04:08.672479', 'startedAt': '2023-03-09T15:04:02.165764', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5035375, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2501.774, 'min': 0.0, 'max': 0.0}, {'current': 2499.632, 'min': 0.0, 'max': 0.0}, {'current': 2500.844, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.227, 'min': 0.0, 'max': 0.0}, {'current': 2498.25, 'min': 0.0, 'max': 0.0}, {'current': 2498.66, 'min': 0.0, 'max': 0.0}, {'current': 2501.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.584, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.412, 'min': 0.0, 'max': 0.0}, {'current': 2500.957, 'min': 0.0, 'max': 0.0}, {'current': 2500.437, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.381, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2499.571, 'min': 0.0, 'max': 0.0}, {'current': 2500.823, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2494.369, 'min': 0.0, 'max': 0.0}, {'current': 2499.905, 'min': 0.0, 'max': 0.0}, {'current': 2500.341, 'min': 0.0, 'max': 0.0}, {'current': 2500.224, 'min': 0.0, 'max': 0.0}, {'current': 2500.123, 'min': 0.0, 'max': 0.0}, {'current': 2499.567, 'min': 0.0, 'max': 0.0}, {'current': 2499.922, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.487, 'min': 0.0, 'max': 0.0}, {'current': 2500.288, 'min': 0.0, 'max': 0.0}, {'current': 2499.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.938, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2501.366, 'min': 0.0, 'max': 0.0}, {'current': 2499.679, 'min': 0.0, 'max': 0.0}, {'current': 2505.537, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2505.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.78, 'min': 0.0, 'max': 0.0}, {'current': 2504.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2505.673, 'min': 0.0, 'max': 0.0}, {'current': 2499.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.354, 'min': 0.0, 'max': 0.0}, {'current': 2503.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.466, 'min': 0.0, 'max': 0.0}, {'current': 2500.896, 'min': 0.0, 'max': 0.0}, {'current': 2500.131, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.496, 'min': 0.0, 'max': 0.0}, {'current': 2505.341, 'min': 0.0, 'max': 0.0}, {'current': 2505.252, 'min': 0.0, 'max': 0.0}, {'current': 2505.01, 'min': 0.0, 'max': 0.0}, {'current': 2506.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2503.996, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.115, 'min': 0.0, 'max': 0.0}, {'current': 2505.91, 'min': 0.0, 'max': 0.0}, {'current': 2504.027, 'min': 0.0, 'max': 0.0}, {'current': 2501.852, 'min': 0.0, 'max': 0.0}, {'current': 2504.801, 'min': 0.0, 'max': 0.0}, {'current': 2499.094, 'min': 0.0, 'max': 0.0}, {'current': 2504.488, 'min': 0.0, 'max': 0.0}, {'current': 2504.156, 'min': 0.0, 'max': 0.0}, {'current': 2497.755, 'min': 0.0, 'max': 0.0}, {'current': 2504.669, 'min': 0.0, 'max': 0.0}, {'current': 2504.714, 'min': 0.0, 'max': 0.0}, {'current': 2504.401, 'min': 0.0, 'max': 0.0}, {'current': 2504.973, 'min': 0.0, 'max': 0.0}, {'current': 2501.132, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.65414810180664}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:04:08,697 INFO    HandlerThread:5402 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:04:08,697 INFO    HandlerThread:5402 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:04:08,697 DEBUG   HandlerThread:5402 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:04:08,730 DEBUG   HandlerThread:5402 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:04:08,731 DEBUG   HandlerThread:5402 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:04:09,530 DEBUG   HandlerThread:5402 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:04:09,551 INFO    HandlerThread:5402 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:04:09,555 DEBUG   SenderThread:5402 [sender.py:send():336] send: files
+2023-03-09 23:04:09,556 INFO    SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:04:09,574 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:04:09,607 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:04:09,801 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml
+2023-03-09 23:04:09,802 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt
+2023-03-09 23:04:09,802 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json
+2023-03-09 23:04:10,151 DEBUG   SenderThread:5402 [sender.py:send():336] send: telemetry
+2023-03-09 23:04:10,156 DEBUG   SenderThread:5402 [sender.py:send():336] send: config
+2023-03-09 23:04:10,168 DEBUG   SenderThread:5402 [sender.py:send():336] send: exit
+2023-03-09 23:04:10,168 INFO    SenderThread:5402 [sender.py:send_exit():559] handling exit code: 1
+2023-03-09 23:04:10,168 INFO    SenderThread:5402 [sender.py:send_exit():561] handling runtime: 1
+2023-03-09 23:04:10,227 INFO    SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:04:10,237 INFO    SenderThread:5402 [sender.py:send_exit():567] send defer
+2023-03-09 23:04:10,253 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,258 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 0
+2023-03-09 23:04:10,278 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,283 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 0
+2023-03-09 23:04:10,283 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 1
+2023-03-09 23:04:10,283 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,283 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 1
+2023-03-09 23:04:10,284 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,284 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 1
+2023-03-09 23:04:10,284 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 2
+2023-03-09 23:04:10,284 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,284 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 2
+2023-03-09 23:04:10,284 INFO    HandlerThread:5402 [system_monitor.py:finish():193] Stopping system monitor
+2023-03-09 23:04:10,285 DEBUG   SystemMonitor:5402 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:04:10,286 DEBUG   SystemMonitor:5402 [system_monitor.py:_start():168] Finished system metrics aggregation loop
+2023-03-09 23:04:10,286 DEBUG   SystemMonitor:5402 [system_monitor.py:_start():172] Publishing last batch of metrics
+2023-03-09 23:04:10,288 INFO    HandlerThread:5402 [interfaces.py:finish():199] Joined cpu monitor
+2023-03-09 23:04:10,288 INFO    HandlerThread:5402 [interfaces.py:finish():199] Joined disk monitor
+2023-03-09 23:04:10,302 INFO    HandlerThread:5402 [interfaces.py:finish():199] Joined gpu monitor
+2023-03-09 23:04:10,303 INFO    HandlerThread:5402 [interfaces.py:finish():199] Joined memory monitor
+2023-03-09 23:04:10,303 INFO    HandlerThread:5402 [interfaces.py:finish():199] Joined network monitor
+2023-03-09 23:04:10,304 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,304 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 2
+2023-03-09 23:04:10,304 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 3
+2023-03-09 23:04:10,305 DEBUG   SenderThread:5402 [sender.py:send():336] send: stats
+2023-03-09 23:04:10,305 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,305 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 3
+2023-03-09 23:04:10,306 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,306 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 3
+2023-03-09 23:04:10,306 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 4
+2023-03-09 23:04:10,306 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,306 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 4
+2023-03-09 23:04:10,306 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,306 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 4
+2023-03-09 23:04:10,306 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 5
+2023-03-09 23:04:10,307 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,307 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 5
+2023-03-09 23:04:10,307 DEBUG   SenderThread:5402 [sender.py:send():336] send: summary
+2023-03-09 23:04:10,334 INFO    SenderThread:5402 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:04:10,335 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,335 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 5
+2023-03-09 23:04:10,335 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 6
+2023-03-09 23:04:10,335 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,335 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 6
+2023-03-09 23:04:10,335 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,335 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 6
+2023-03-09 23:04:10,341 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:04:10,671 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:10,767 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 7
+2023-03-09 23:04:10,767 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:10,767 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:10,767 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 7
+2023-03-09 23:04:10,768 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:10,768 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 7
+2023-03-09 23:04:10,784 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml
+2023-03-09 23:04:10,784 INFO    Thread-13 :5402 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json
+2023-03-09 23:04:10,863 INFO    wandb-upload_0:5402 [upload_job.py:push():138] Uploaded file /tmp/tmpbm6fg7powandb/0dy5vwa2-wandb-metadata.json
+2023-03-09 23:04:11,672 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:12,197 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 8
+2023-03-09 23:04:12,202 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:12,203 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:12,208 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 8
+2023-03-09 23:04:12,229 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:12,229 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 8
+2023-03-09 23:04:12,265 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 9
+2023-03-09 23:04:12,280 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:12,280 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 9
+2023-03-09 23:04:12,286 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:12,286 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 9
+2023-03-09 23:04:12,286 INFO    SenderThread:5402 [dir_watcher.py:finish():365] shutting down directory watcher
+2023-03-09 23:04:12,684 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:12,786 INFO    SenderThread:5402 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files
+2023-03-09 23:04:12,787 INFO    SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/conda-environment.yaml conda-environment.yaml
+2023-03-09 23:04:12,787 INFO    SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt requirements.txt
+2023-03-09 23:04:12,787 INFO    SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml config.yaml
+2023-03-09 23:04:12,787 INFO    SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json wandb-summary.json
+2023-03-09 23:04:12,792 INFO    SenderThread:5402 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-metadata.json wandb-metadata.json
+2023-03-09 23:04:12,793 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 10
+2023-03-09 23:04:12,793 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:12,818 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:12,818 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 10
+2023-03-09 23:04:12,940 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:12,946 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 10
+2023-03-09 23:04:12,946 INFO    SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:04:13,726 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:13,747 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:13,947 INFO    wandb-upload_3:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/wandb-summary.json
+2023-03-09 23:04:14,378 INFO    wandb-upload_2:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/config.yaml
+2023-03-09 23:04:14,732 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:14,733 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:15,734 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:15,734 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:16,734 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:16,735 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:17,736 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:17,736 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:18,736 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:18,737 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:19,244 ERROR   wandb-upload_1:5402 [internal_api.py:execute():245] 502 response executing GraphQL.
+2023-03-09 23:04:19,245 ERROR   wandb-upload_1:5402 [internal_api.py:execute():246] 
+<html><head>
+<meta http-equiv="content-type" content="text/html;charset=utf-8">
+<title>502 Server Error</title>
+</head>
+<body text=#000000 bgcolor=#ffffff>
+<h1>Error: Server Error</h1>
+<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
+<h2></h2>
+</body></html>
+
+2023-03-09 23:04:19,738 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:04:19,738 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:19,739 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:20,739 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:20,740 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:21,423 INFO    wandb-upload_1:5402 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/files/requirements.txt
+2023-03-09 23:04:21,623 INFO    Thread-12 :5402 [sender.py:transition_state():587] send defer: 11
+2023-03-09 23:04:21,623 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:21,623 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 11
+2023-03-09 23:04:21,624 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:21,624 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 11
+2023-03-09 23:04:21,624 INFO    SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher
+2023-03-09 23:04:21,624 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 12
+2023-03-09 23:04:21,624 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:21,624 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 12
+2023-03-09 23:04:21,624 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:21,624 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 12
+2023-03-09 23:04:21,809 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:22,109 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 13
+2023-03-09 23:04:22,109 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:22,109 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:22,110 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 13
+2023-03-09 23:04:22,110 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:22,110 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 13
+2023-03-09 23:04:22,110 INFO    SenderThread:5402 [sender.py:transition_state():587] send defer: 14
+2023-03-09 23:04:22,110 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:04:22,111 INFO    HandlerThread:5402 [handler.py:handle_request_defer():170] handle defer: 14
+2023-03-09 23:04:22,111 DEBUG   SenderThread:5402 [sender.py:send():336] send: final
+2023-03-09 23:04:22,111 DEBUG   SenderThread:5402 [sender.py:send():336] send: footer
+2023-03-09 23:04:22,111 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:04:22,111 INFO    SenderThread:5402 [sender.py:send_request_defer():583] handle sender defer: 14
+2023-03-09 23:04:22,112 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:22,112 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:22,113 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:04:22,113 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:04:22,113 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: server_info
+2023-03-09 23:04:22,114 DEBUG   SenderThread:5402 [sender.py:send_request():363] send_request: server_info
+2023-03-09 23:04:22,150 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: get_summary
+2023-03-09 23:04:22,165 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: sampled_history
+2023-03-09 23:04:23,210 INFO    MainThread:5402 [wandb_run.py:_footer_history_summary_info():3429] rendering history
+2023-03-09 23:04:23,210 INFO    MainThread:5402 [wandb_run.py:_footer_history_summary_info():3461] rendering summary
+2023-03-09 23:04:23,210 INFO    MainThread:5402 [wandb_run.py:_footer_sync_info():3387] logging synced files
+2023-03-09 23:04:23,211 DEBUG   HandlerThread:5402 [handler.py:handle_request():144] handle_request: shutdown
+2023-03-09 23:04:23,212 INFO    HandlerThread:5402 [handler.py:finish():842] shutting down handler
+2023-03-09 23:04:24,208 INFO    WriterThread:5402 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/run-tfw61kps.wandb
+2023-03-09 23:04:24,209 INFO    SenderThread:5402 [sender.py:finish():1504] shutting down sender
+2023-03-09 23:04:24,210 INFO    SenderThread:5402 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:04:24,210 INFO    SenderThread:5402 [file_pusher.py:join():169] waiting for file pusher

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log

@@ -0,0 +1,29 @@
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Configure stats pid to 3604
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug.log
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230402-tfw61kps/logs/debug-internal.log
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:init():602] starting backend
+2023-03-09 23:04:02,230 INFO    MainThread:3604 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:04:02,242 INFO    MainThread:3604 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:04:02,245 INFO    MainThread:3604 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:04:02,249 INFO    MainThread:3604 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:04:02,251 INFO    MainThread:3604 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:04:03,555 INFO    MainThread:3604 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:04:08,608 INFO    MainThread:3604 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:04:08,608 INFO    MainThread:3604 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:04:09,562 INFO    MainThread:3604 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:04:09,562 INFO    MainThread:3604 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:04:09,562 INFO    MainThread:3604 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:04:09,562 INFO    MainThread:3604 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:04:09,563 INFO    MainThread:3604 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:04:09,564 INFO    MainThread:3604 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}
+2023-03-09 23:04:24,411 WARNING MsgRouterThr:3604 [router.py:message_loop():77] message_loop has been closed

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml

@@ -0,0 +1,40 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678374396.698895
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log

@@ -0,0 +1,14 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Device:          cuda
+        Images scaling:  0.5
+Traceback (most recent call last):
+  File "train.py", line 169, in <module>
+    img_scale=args.scale,
+  File "train.py", line 70, in train_model
+    optimizer = ms.nn.RMSprop(params=model.parameters(), learning_rate=learning_rate, momentum=momentum, weight_decay=weight_decay)
+AttributeError: module 'mindspore.nn' has no attribute 'RMSprop'

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:06:43.530245",
+    "startedAt": "2023-03-09T15:06:36.660035",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2499.997949999998,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.992,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.979,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.992,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.988,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.985,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.987,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.992,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.981,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.985,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.992,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.984,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.012,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.992,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.006,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.008,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.006,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.996,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.981,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.987,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.049,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.006,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.984,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.063,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.004,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.014,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.002,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6544303894043
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json

@@ -0,0 +1 @@
+{"_wandb": {"runtime": 1}}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log

@@ -0,0 +1,177 @@
+2023-03-09 23:06:36,705 INFO    StreamThr :7419 [internal.py:wandb_internal():90] W&B internal server running at pid: 7419, started at: 2023-03-09 23:06:36.703095
+2023-03-09 23:06:36,706 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:06:36,708 INFO    WriterThread:7419 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb
+2023-03-09 23:06:36,712 DEBUG   SenderThread:7419 [sender.py:send():336] send: header
+2023-03-09 23:06:36,713 DEBUG   SenderThread:7419 [sender.py:send():336] send: run
+2023-03-09 23:06:36,804 INFO    SenderThread:7419 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/3p0zeu2z
+2023-03-09 23:06:38,422 INFO    SenderThread:7419 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files
+2023-03-09 23:06:38,422 INFO    SenderThread:7419 [sender.py:_start_run_threads():1081] run started: 3p0zeu2z with start time 1678374396.698895
+2023-03-09 23:06:38,423 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:06:38,426 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:06:38,444 INFO    SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:06:38,444 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:06:39,426 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json
+2023-03-09 23:06:43,427 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:06:43,446 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:06:43,504 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:06:43,509 DEBUG   HandlerThread:7419 [system_info.py:__init__():31] System info init
+2023-03-09 23:06:43,509 DEBUG   HandlerThread:7419 [system_info.py:__init__():46] System info init done
+2023-03-09 23:06:43,509 INFO    HandlerThread:7419 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:06:43,510 INFO    SystemMonitor:7419 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:06:43,510 INFO    HandlerThread:7419 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:06:43,510 INFO    SystemMonitor:7419 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:06:43,511 INFO    SystemMonitor:7419 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:06:43,511 INFO    SystemMonitor:7419 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:06:43,512 INFO    SystemMonitor:7419 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:06:43,512 INFO    SystemMonitor:7419 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:06:43,530 DEBUG   HandlerThread:7419 [system_info.py:probe():195] Probing system
+2023-03-09 23:06:43,549 DEBUG   HandlerThread:7419 [git.py:repo():40] git repository is invalid
+2023-03-09 23:06:43,549 DEBUG   HandlerThread:7419 [system_info.py:probe():240] Probing system done
+2023-03-09 23:06:43,549 DEBUG   HandlerThread:7419 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:06:43.530245', 'startedAt': '2023-03-09T15:06:36.660035', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2499.997949999998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.979, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.985, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.992, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.996, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.981, 'min': 0.0, 'max': 0.0}, {'current': 2499.987, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.049, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2499.984, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.014, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6544303894043}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:06:43,549 INFO    HandlerThread:7419 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:06:43,549 INFO    HandlerThread:7419 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:06:43,549 DEBUG   HandlerThread:7419 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:06:43,576 DEBUG   HandlerThread:7419 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:06:43,577 DEBUG   HandlerThread:7419 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:06:44,293 DEBUG   HandlerThread:7419 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:06:44,346 INFO    HandlerThread:7419 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:06:44,349 DEBUG   SenderThread:7419 [sender.py:send():336] send: files
+2023-03-09 23:06:44,349 INFO    SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:06:44,368 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:06:44,384 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:06:44,593 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml
+2023-03-09 23:06:44,593 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json
+2023-03-09 23:06:44,593 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt
+2023-03-09 23:06:44,866 DEBUG   SenderThread:7419 [sender.py:send():336] send: telemetry
+2023-03-09 23:06:44,871 DEBUG   SenderThread:7419 [sender.py:send():336] send: config
+2023-03-09 23:06:44,914 DEBUG   SenderThread:7419 [sender.py:send():336] send: exit
+2023-03-09 23:06:44,920 INFO    SenderThread:7419 [sender.py:send_exit():559] handling exit code: 1
+2023-03-09 23:06:44,920 INFO    SenderThread:7419 [sender.py:send_exit():561] handling runtime: 1
+2023-03-09 23:06:45,056 INFO    SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:06:45,067 INFO    SenderThread:7419 [sender.py:send_exit():567] send defer
+2023-03-09 23:06:45,068 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,069 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 0
+2023-03-09 23:06:45,069 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,069 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 0
+2023-03-09 23:06:45,069 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 1
+2023-03-09 23:06:45,069 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,069 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 1
+2023-03-09 23:06:45,070 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,070 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 1
+2023-03-09 23:06:45,070 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 2
+2023-03-09 23:06:45,070 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,070 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 2
+2023-03-09 23:06:45,070 INFO    HandlerThread:7419 [system_monitor.py:finish():193] Stopping system monitor
+2023-03-09 23:06:45,070 DEBUG   SystemMonitor:7419 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:06:45,071 DEBUG   SystemMonitor:7419 [system_monitor.py:_start():168] Finished system metrics aggregation loop
+2023-03-09 23:06:45,071 DEBUG   SystemMonitor:7419 [system_monitor.py:_start():172] Publishing last batch of metrics
+2023-03-09 23:06:45,072 INFO    HandlerThread:7419 [interfaces.py:finish():199] Joined cpu monitor
+2023-03-09 23:06:45,072 INFO    HandlerThread:7419 [interfaces.py:finish():199] Joined disk monitor
+2023-03-09 23:06:45,086 INFO    HandlerThread:7419 [interfaces.py:finish():199] Joined gpu monitor
+2023-03-09 23:06:45,086 INFO    HandlerThread:7419 [interfaces.py:finish():199] Joined memory monitor
+2023-03-09 23:06:45,086 INFO    HandlerThread:7419 [interfaces.py:finish():199] Joined network monitor
+2023-03-09 23:06:45,087 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,087 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 2
+2023-03-09 23:06:45,087 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 3
+2023-03-09 23:06:45,087 DEBUG   SenderThread:7419 [sender.py:send():336] send: stats
+2023-03-09 23:06:45,087 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,087 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 3
+2023-03-09 23:06:45,088 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,088 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 3
+2023-03-09 23:06:45,088 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 4
+2023-03-09 23:06:45,088 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,088 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 4
+2023-03-09 23:06:45,089 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,089 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 4
+2023-03-09 23:06:45,089 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 5
+2023-03-09 23:06:45,089 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,089 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 5
+2023-03-09 23:06:45,089 DEBUG   SenderThread:7419 [sender.py:send():336] send: summary
+2023-03-09 23:06:45,107 INFO    SenderThread:7419 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:06:45,108 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,108 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 5
+2023-03-09 23:06:45,108 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 6
+2023-03-09 23:06:45,108 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,108 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 6
+2023-03-09 23:06:45,108 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,108 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 6
+2023-03-09 23:06:45,114 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:06:45,573 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 7
+2023-03-09 23:06:45,573 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:45,573 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 7
+2023-03-09 23:06:45,574 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:45,574 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 7
+2023-03-09 23:06:45,591 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:06:45,595 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json
+2023-03-09 23:06:45,595 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml
+2023-03-09 23:06:45,595 INFO    Thread-13 :7419 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log
+2023-03-09 23:06:45,608 INFO    wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /tmp/tmph6vgygn9wandb/2ibjar1l-wandb-metadata.json
+2023-03-09 23:06:46,926 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 8
+2023-03-09 23:06:46,927 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:06:46,927 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:46,927 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 8
+2023-03-09 23:06:46,928 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:46,928 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 8
+2023-03-09 23:06:46,928 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 9
+2023-03-09 23:06:46,928 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:46,928 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 9
+2023-03-09 23:06:46,928 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:46,929 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 9
+2023-03-09 23:06:46,929 INFO    SenderThread:7419 [dir_watcher.py:finish():365] shutting down directory watcher
+2023-03-09 23:06:47,597 INFO    SenderThread:7419 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log
+2023-03-09 23:06:47,597 INFO    SenderThread:7419 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files
+2023-03-09 23:06:47,598 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/conda-environment.yaml conda-environment.yaml
+2023-03-09 23:06:47,598 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log output.log
+2023-03-09 23:06:47,598 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt requirements.txt
+2023-03-09 23:06:47,603 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml config.yaml
+2023-03-09 23:06:47,629 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json wandb-summary.json
+2023-03-09 23:06:47,720 INFO    SenderThread:7419 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-metadata.json wandb-metadata.json
+2023-03-09 23:06:47,720 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 10
+2023-03-09 23:06:47,857 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:47,858 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 10
+2023-03-09 23:06:47,891 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:47,896 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 10
+2023-03-09 23:06:47,896 INFO    SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:06:48,926 INFO    wandb-upload_0:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/output.log
+2023-03-09 23:06:49,240 INFO    wandb-upload_3:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/config.yaml
+2023-03-09 23:06:49,315 INFO    wandb-upload_4:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/wandb-summary.json
+2023-03-09 23:06:50,115 INFO    wandb-upload_2:7419 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/files/requirements.txt
+2023-03-09 23:06:50,316 INFO    Thread-12 :7419 [sender.py:transition_state():587] send defer: 11
+2023-03-09 23:06:50,316 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:50,316 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 11
+2023-03-09 23:06:50,317 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:50,317 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 11
+2023-03-09 23:06:50,317 INFO    SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher
+2023-03-09 23:06:50,317 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 12
+2023-03-09 23:06:50,317 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:50,317 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 12
+2023-03-09 23:06:50,317 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:50,318 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 12
+2023-03-09 23:06:50,626 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:06:50,662 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 13
+2023-03-09 23:06:50,662 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:50,662 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 13
+2023-03-09 23:06:50,662 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:50,663 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 13
+2023-03-09 23:06:50,663 INFO    SenderThread:7419 [sender.py:transition_state():587] send defer: 14
+2023-03-09 23:06:50,663 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:06:50,663 INFO    HandlerThread:7419 [handler.py:handle_request_defer():170] handle defer: 14
+2023-03-09 23:06:50,663 DEBUG   SenderThread:7419 [sender.py:send():336] send: final
+2023-03-09 23:06:50,664 DEBUG   SenderThread:7419 [sender.py:send():336] send: footer
+2023-03-09 23:06:50,664 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:06:50,664 INFO    SenderThread:7419 [sender.py:send_request_defer():583] handle sender defer: 14
+2023-03-09 23:06:50,664 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:06:50,665 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:06:50,665 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: server_info
+2023-03-09 23:06:50,666 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: get_summary
+2023-03-09 23:06:50,666 DEBUG   SenderThread:7419 [sender.py:send_request():363] send_request: server_info
+2023-03-09 23:06:50,717 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: sampled_history
+2023-03-09 23:06:51,503 INFO    MainThread:7419 [wandb_run.py:_footer_history_summary_info():3429] rendering history
+2023-03-09 23:06:51,503 INFO    MainThread:7419 [wandb_run.py:_footer_history_summary_info():3461] rendering summary
+2023-03-09 23:06:51,503 INFO    MainThread:7419 [wandb_run.py:_footer_sync_info():3387] logging synced files
+2023-03-09 23:06:51,504 DEBUG   HandlerThread:7419 [handler.py:handle_request():144] handle_request: shutdown
+2023-03-09 23:06:51,504 INFO    HandlerThread:7419 [handler.py:finish():842] shutting down handler
+2023-03-09 23:06:51,666 INFO    WriterThread:7419 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/run-3p0zeu2z.wandb
+2023-03-09 23:06:52,503 INFO    SenderThread:7419 [sender.py:finish():1504] shutting down sender
+2023-03-09 23:06:52,503 INFO    SenderThread:7419 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:06:52,503 INFO    SenderThread:7419 [file_pusher.py:join():169] waiting for file pusher

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log

@@ -0,0 +1,28 @@
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Configure stats pid to 5620
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:06:36,685 INFO    MainThread:5620 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug.log
+2023-03-09 23:06:36,686 INFO    MainThread:5620 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_230636-3p0zeu2z/logs/debug-internal.log
+2023-03-09 23:06:36,686 INFO    MainThread:5620 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:06:36,686 INFO    MainThread:5620 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:06:36,686 INFO    MainThread:5620 [wandb_init.py:init():602] starting backend
+2023-03-09 23:06:36,686 INFO    MainThread:5620 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:06:36,697 INFO    MainThread:5620 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:06:36,698 INFO    MainThread:5620 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:06:36,703 INFO    MainThread:5620 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:06:36,704 INFO    MainThread:5620 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:06:38,425 INFO    MainThread:5620 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:06:43,446 INFO    MainThread:5620 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:06:43,447 INFO    MainThread:5620 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:06:44,356 INFO    MainThread:5620 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:06:44,356 INFO    MainThread:5620 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:06:44,357 INFO    MainThread:5620 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:06:44,357 INFO    MainThread:5620 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:06:44,358 INFO    MainThread:5620 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:06:44,358 INFO    MainThread:5620 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}
+2023-03-09 23:06:52,524 WARNING MsgRouterThr:5620 [router.py:message_loop():77] message_loop has been closed

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml

@@ -0,0 +1,40 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678374643.540464
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log

@@ -0,0 +1,14 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Device:          cuda
+        Images scaling:  0.5
+Traceback (most recent call last):
+  File "train.py", line 169, in <module>
+    img_scale=args.scale,
+  File "train.py", line 88, in train_model
+    grad_fn = ops.value_and_grad(forward_fn, None, optimizer.parameters, has_aux=True)
+NameError: name 'ops' is not defined

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:10:50.243518",
+    "startedAt": "2023-03-09T15:10:43.500810",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.419575,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.322,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.664,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.917,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.586,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.207,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.024,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.054,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.294,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2497.793,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.641,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.912,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.087,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.927,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.004,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.109,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.008,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.666,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.422,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.34,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.446,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.667,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.071,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.954,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.346,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.336,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.318,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.379,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.717,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.668,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.274,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.083,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.294,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.426,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.706,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.578,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.8,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.034,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.781,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.461,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.789,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.345,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.362,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.448,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.053,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.154,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.213,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.574,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.754,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.345,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.409,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.953,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.981,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.045,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.47,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.703,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.13,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.197,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.917,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.623,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.272,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.349,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.464,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.376,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.544,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.74,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.053,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6549377441406
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json

@@ -0,0 +1 @@
+{"_wandb": {"runtime": 1}}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log

@@ -0,0 +1,183 @@
+2023-03-09 23:10:43,546 INFO    StreamThr :9187 [internal.py:wandb_internal():90] W&B internal server running at pid: 9187, started at: 2023-03-09 23:10:43.544652
+2023-03-09 23:10:43,548 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:10:43,549 INFO    WriterThread:9187 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb
+2023-03-09 23:10:43,554 DEBUG   SenderThread:9187 [sender.py:send():336] send: header
+2023-03-09 23:10:43,554 DEBUG   SenderThread:9187 [sender.py:send():336] send: run
+2023-03-09 23:10:43,617 INFO    SenderThread:9187 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/osho06z5
+2023-03-09 23:10:45,099 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:10:45,100 INFO    SenderThread:9187 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files
+2023-03-09 23:10:45,100 INFO    SenderThread:9187 [sender.py:_start_run_threads():1081] run started: osho06z5 with start time 1678374643.540464
+2023-03-09 23:10:45,100 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:10:45,131 INFO    SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:10:45,132 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:10:46,104 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json
+2023-03-09 23:10:50,100 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:10:50,133 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:10:50,192 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:10:50,205 DEBUG   HandlerThread:9187 [system_info.py:__init__():31] System info init
+2023-03-09 23:10:50,205 DEBUG   HandlerThread:9187 [system_info.py:__init__():46] System info init done
+2023-03-09 23:10:50,205 INFO    HandlerThread:9187 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:10:50,206 INFO    SystemMonitor:9187 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:10:50,206 INFO    HandlerThread:9187 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:10:50,206 INFO    SystemMonitor:9187 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:10:50,207 INFO    SystemMonitor:9187 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:10:50,207 INFO    SystemMonitor:9187 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:10:50,208 INFO    SystemMonitor:9187 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:10:50,208 INFO    SystemMonitor:9187 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:10:50,243 DEBUG   HandlerThread:9187 [system_info.py:probe():195] Probing system
+2023-03-09 23:10:50,266 DEBUG   HandlerThread:9187 [git.py:repo():40] git repository is invalid
+2023-03-09 23:10:50,266 DEBUG   HandlerThread:9187 [system_info.py:probe():240] Probing system done
+2023-03-09 23:10:50,266 DEBUG   HandlerThread:9187 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:10:50.243518', 'startedAt': '2023-03-09T15:10:43.500810', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.419575, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.664, 'min': 0.0, 'max': 0.0}, {'current': 2499.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.586, 'min': 0.0, 'max': 0.0}, {'current': 2500.207, 'min': 0.0, 'max': 0.0}, {'current': 2500.024, 'min': 0.0, 'max': 0.0}, {'current': 2500.054, 'min': 0.0, 'max': 0.0}, {'current': 2501.294, 'min': 0.0, 'max': 0.0}, {'current': 2497.793, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.641, 'min': 0.0, 'max': 0.0}, {'current': 2503.912, 'min': 0.0, 'max': 0.0}, {'current': 2502.087, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2502.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2503.109, 'min': 0.0, 'max': 0.0}, {'current': 2501.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.666, 'min': 0.0, 'max': 0.0}, {'current': 2501.422, 'min': 0.0, 'max': 0.0}, {'current': 2500.34, 'min': 0.0, 'max': 0.0}, {'current': 2500.446, 'min': 0.0, 'max': 0.0}, {'current': 2499.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.071, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.954, 'min': 0.0, 'max': 0.0}, {'current': 2500.346, 'min': 0.0, 'max': 0.0}, {'current': 2500.336, 'min': 0.0, 'max': 0.0}, {'current': 2501.318, 'min': 0.0, 'max': 0.0}, {'current': 2500.379, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.668, 'min': 0.0, 'max': 0.0}, {'current': 2499.274, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.083, 'min': 0.0, 'max': 0.0}, {'current': 2499.294, 'min': 0.0, 'max': 0.0}, {'current': 2505.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.706, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.578, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.8, 'min': 0.0, 'max': 0.0}, {'current': 2499.034, 'min': 0.0, 'max': 0.0}, {'current': 2504.781, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2500.789, 'min': 0.0, 'max': 0.0}, {'current': 2501.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.362, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.448, 'min': 0.0, 'max': 0.0}, {'current': 2499.053, 'min': 0.0, 'max': 0.0}, {'current': 2503.154, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.213, 'min': 0.0, 'max': 0.0}, {'current': 2504.574, 'min': 0.0, 'max': 0.0}, {'current': 2506.754, 'min': 0.0, 'max': 0.0}, {'current': 2500.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.409, 'min': 0.0, 'max': 0.0}, {'current': 2504.953, 'min': 0.0, 'max': 0.0}, {'current': 2501.981, 'min': 0.0, 'max': 0.0}, {'current': 2500.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.47, 'min': 0.0, 'max': 0.0}, {'current': 2500.703, 'min': 0.0, 'max': 0.0}, {'current': 2503.13, 'min': 0.0, 'max': 0.0}, {'current': 2500.197, 'min': 0.0, 'max': 0.0}, {'current': 2505.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.99, 'min': 0.0, 'max': 0.0}, {'current': 2503.272, 'min': 0.0, 'max': 0.0}, {'current': 2502.349, 'min': 0.0, 'max': 0.0}, {'current': 2504.464, 'min': 0.0, 'max': 0.0}, {'current': 2503.376, 'min': 0.0, 'max': 0.0}, {'current': 2503.997, 'min': 0.0, 'max': 0.0}, {'current': 2505.544, 'min': 0.0, 'max': 0.0}, {'current': 2505.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.053, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6549377441406}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:10:50,267 INFO    HandlerThread:9187 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:10:50,267 INFO    HandlerThread:9187 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:10:50,267 DEBUG   HandlerThread:9187 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:10:50,309 DEBUG   HandlerThread:9187 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:10:50,309 DEBUG   HandlerThread:9187 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:10:51,019 DEBUG   HandlerThread:9187 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:10:51,065 INFO    HandlerThread:9187 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:10:51,068 DEBUG   SenderThread:9187 [sender.py:send():336] send: files
+2023-03-09 23:10:51,068 INFO    SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:10:51,084 DEBUG   SenderThread:9187 [sender.py:send():336] send: telemetry
+2023-03-09 23:10:51,084 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:10:51,094 DEBUG   SenderThread:9187 [sender.py:send():336] send: config
+2023-03-09 23:10:51,130 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:10:51,233 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json
+2023-03-09 23:10:51,262 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml
+2023-03-09 23:10:51,268 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt
+2023-03-09 23:10:51,721 DEBUG   SenderThread:9187 [sender.py:send():336] send: exit
+2023-03-09 23:10:51,721 INFO    SenderThread:9187 [sender.py:send_exit():559] handling exit code: 1
+2023-03-09 23:10:51,721 INFO    SenderThread:9187 [sender.py:send_exit():561] handling runtime: 1
+2023-03-09 23:10:51,797 INFO    SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:10:51,808 INFO    SenderThread:9187 [sender.py:send_exit():567] send defer
+2023-03-09 23:10:51,828 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,832 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 0
+2023-03-09 23:10:51,833 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,833 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 0
+2023-03-09 23:10:51,833 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 1
+2023-03-09 23:10:51,833 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,833 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 1
+2023-03-09 23:10:51,834 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,834 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 1
+2023-03-09 23:10:51,834 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 2
+2023-03-09 23:10:51,834 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,834 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 2
+2023-03-09 23:10:51,834 INFO    HandlerThread:9187 [system_monitor.py:finish():193] Stopping system monitor
+2023-03-09 23:10:51,834 DEBUG   SystemMonitor:9187 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:10:51,835 DEBUG   SystemMonitor:9187 [system_monitor.py:_start():168] Finished system metrics aggregation loop
+2023-03-09 23:10:51,835 DEBUG   SystemMonitor:9187 [system_monitor.py:_start():172] Publishing last batch of metrics
+2023-03-09 23:10:51,837 INFO    HandlerThread:9187 [interfaces.py:finish():199] Joined cpu monitor
+2023-03-09 23:10:51,838 INFO    HandlerThread:9187 [interfaces.py:finish():199] Joined disk monitor
+2023-03-09 23:10:51,850 INFO    HandlerThread:9187 [interfaces.py:finish():199] Joined gpu monitor
+2023-03-09 23:10:51,851 INFO    HandlerThread:9187 [interfaces.py:finish():199] Joined memory monitor
+2023-03-09 23:10:51,851 INFO    HandlerThread:9187 [interfaces.py:finish():199] Joined network monitor
+2023-03-09 23:10:51,852 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,852 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 2
+2023-03-09 23:10:51,852 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 3
+2023-03-09 23:10:51,852 DEBUG   SenderThread:9187 [sender.py:send():336] send: stats
+2023-03-09 23:10:51,853 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,853 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 3
+2023-03-09 23:10:51,853 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,853 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 3
+2023-03-09 23:10:51,853 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 4
+2023-03-09 23:10:51,854 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,854 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 4
+2023-03-09 23:10:51,854 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,854 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 4
+2023-03-09 23:10:51,854 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 5
+2023-03-09 23:10:51,854 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,854 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 5
+2023-03-09 23:10:51,854 DEBUG   SenderThread:9187 [sender.py:send():336] send: summary
+2023-03-09 23:10:51,873 INFO    SenderThread:9187 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:10:51,874 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,874 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 5
+2023-03-09 23:10:51,874 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 6
+2023-03-09 23:10:51,874 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:51,874 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 6
+2023-03-09 23:10:51,874 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:51,874 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 6
+2023-03-09 23:10:51,879 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:10:52,221 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json
+2023-03-09 23:10:52,221 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log
+2023-03-09 23:10:52,323 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 7
+2023-03-09 23:10:52,323 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:52,323 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 7
+2023-03-09 23:10:52,324 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:52,324 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 7
+2023-03-09 23:10:52,324 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:10:52,402 INFO    wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /tmp/tmpfner7v0awandb/mcayg890-wandb-metadata.json
+2023-03-09 23:10:53,222 INFO    Thread-13 :9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml
+2023-03-09 23:10:53,747 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 8
+2023-03-09 23:10:53,753 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:10:53,753 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:53,758 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 8
+2023-03-09 23:10:53,789 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:53,801 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 8
+2023-03-09 23:10:53,827 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 9
+2023-03-09 23:10:53,840 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:53,840 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 9
+2023-03-09 23:10:53,840 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:53,840 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 9
+2023-03-09 23:10:53,840 INFO    SenderThread:9187 [dir_watcher.py:finish():365] shutting down directory watcher
+2023-03-09 23:10:54,228 INFO    SenderThread:9187 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log
+2023-03-09 23:10:54,228 INFO    SenderThread:9187 [dir_watcher.py:finish():395] scan: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files
+2023-03-09 23:10:54,229 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/conda-environment.yaml conda-environment.yaml
+2023-03-09 23:10:54,229 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log output.log
+2023-03-09 23:10:54,229 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt requirements.txt
+2023-03-09 23:10:54,235 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml config.yaml
+2023-03-09 23:10:54,255 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json wandb-summary.json
+2023-03-09 23:10:54,304 INFO    SenderThread:9187 [dir_watcher.py:finish():409] scan save: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-metadata.json wandb-metadata.json
+2023-03-09 23:10:54,304 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 10
+2023-03-09 23:10:54,523 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:10:54,529 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 10
+2023-03-09 23:10:54,542 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:10:54,542 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 10
+2023-03-09 23:10:54,542 INFO    SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:10:55,533 INFO    wandb-upload_0:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/requirements.txt
+2023-03-09 23:10:55,794 INFO    wandb-upload_3:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/config.yaml
+2023-03-09 23:10:55,904 INFO    wandb-upload_1:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/output.log
+2023-03-09 23:10:57,354 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:10:59,356 ERROR   wandb-upload_4:9187 [internal_api.py:upload_file():1997] upload_file exception https://storage.googleapis.com/wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-production.appspot.com/anony-moose-510831/U-Net/osho06z5/wandb-summary.json?Expires=1678461146&GoogleAccessId=wandb-production%40appspot.gserviceaccount.com&Signature=CTSL8FePXMav4o0ERsq5ED1l8Tn67rd2agZ4RCkpiEQpJa1xFovJjLyERirrlgyyR1IYGJ7YC8mvdwxMo04QdsvFfxdzl7sWSnMNlut4cDIOaHUy%2Fa0iI7AZWLotW9oKV7SZ3I1Bv%2BPgEVAEbFLJdW8hzzhP7dugqeh%2BKd2ShA7aK1Cuthgknx57msxZK3AMzQl0HTF8L0%2BQlGeHfr0HprOeK7q3Z8jA0oqanCh3LXXD2QAwh1OVukYr7wUK0XtfOhTj4LuLII7YGgQ7nSJ0BJiGEe9QQBPacsnmf5v793Jb6gUxVLOEgTAckd0ndapR0JHvZsuW00OJKmZTV6R09g%3D%3D (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb81c30ea10>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
+2023-03-09 23:10:59,356 ERROR   wandb-upload_4:9187 [internal_api.py:upload_file():1999] upload_file request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '26'}
+2023-03-09 23:10:59,356 ERROR   wandb-upload_4:9187 [internal_api.py:upload_file():2001] upload_file response body: 
+2023-03-09 23:11:00,543 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:11:02,356 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:11:03,317 INFO    wandb-upload_4:9187 [upload_job.py:push():138] Uploaded file /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/files/wandb-summary.json
+2023-03-09 23:11:03,517 INFO    Thread-12 :9187 [sender.py:transition_state():587] send defer: 11
+2023-03-09 23:11:03,518 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:11:03,518 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 11
+2023-03-09 23:11:03,518 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:11:03,518 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 11
+2023-03-09 23:11:03,518 INFO    SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher
+2023-03-09 23:11:03,518 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 12
+2023-03-09 23:11:03,519 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:11:03,519 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 12
+2023-03-09 23:11:03,519 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:11:03,519 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 12
+2023-03-09 23:11:04,731 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 13
+2023-03-09 23:11:04,731 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:11:04,731 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 13
+2023-03-09 23:11:04,731 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:11:04,731 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 13
+2023-03-09 23:11:04,731 INFO    SenderThread:9187 [sender.py:transition_state():587] send defer: 14
+2023-03-09 23:11:04,732 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: defer
+2023-03-09 23:11:04,732 INFO    HandlerThread:9187 [handler.py:handle_request_defer():170] handle defer: 14
+2023-03-09 23:11:04,732 DEBUG   SenderThread:9187 [sender.py:send():336] send: final
+2023-03-09 23:11:04,732 DEBUG   SenderThread:9187 [sender.py:send():336] send: footer
+2023-03-09 23:11:04,732 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: defer
+2023-03-09 23:11:04,732 INFO    SenderThread:9187 [sender.py:send_request_defer():583] handle sender defer: 14
+2023-03-09 23:11:04,733 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: poll_exit
+2023-03-09 23:11:04,733 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: poll_exit
+2023-03-09 23:11:04,734 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: server_info
+2023-03-09 23:11:04,734 DEBUG   SenderThread:9187 [sender.py:send_request():363] send_request: server_info
+2023-03-09 23:11:04,770 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: get_summary
+2023-03-09 23:11:04,800 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: sampled_history
+2023-03-09 23:11:05,697 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:11:05,697 INFO    MainThread:9187 [wandb_run.py:_footer_history_summary_info():3429] rendering history
+2023-03-09 23:11:05,697 INFO    MainThread:9187 [wandb_run.py:_footer_history_summary_info():3461] rendering summary
+2023-03-09 23:11:05,698 INFO    MainThread:9187 [wandb_run.py:_footer_sync_info():3387] logging synced files
+2023-03-09 23:11:05,698 DEBUG   HandlerThread:9187 [handler.py:handle_request():144] handle_request: shutdown
+2023-03-09 23:11:05,698 INFO    HandlerThread:9187 [handler.py:finish():842] shutting down handler
+2023-03-09 23:11:06,697 INFO    SenderThread:9187 [sender.py:finish():1504] shutting down sender
+2023-03-09 23:11:06,697 INFO    SenderThread:9187 [file_pusher.py:finish():164] shutting down file pusher
+2023-03-09 23:11:06,697 INFO    SenderThread:9187 [file_pusher.py:join():169] waiting for file pusher
+2023-03-09 23:11:06,697 INFO    WriterThread:9187 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/run-osho06z5.wandb

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log

@@ -0,0 +1,28 @@
+2023-03-09 23:10:43,526 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Configure stats pid to 7468
+2023-03-09 23:10:43,526 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:10:43,526 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:10:43,526 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug.log
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231043-osho06z5/logs/debug-internal.log
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:init():602] starting backend
+2023-03-09 23:10:43,527 INFO    MainThread:7468 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:10:43,538 INFO    MainThread:7468 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:10:43,540 INFO    MainThread:7468 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:10:43,544 INFO    MainThread:7468 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:10:43,545 INFO    MainThread:7468 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:10:45,099 INFO    MainThread:7468 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:10:50,134 INFO    MainThread:7468 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:10:50,134 INFO    MainThread:7468 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:10:51,075 INFO    MainThread:7468 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:10:51,075 INFO    MainThread:7468 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:10:51,075 INFO    MainThread:7468 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:10:51,075 INFO    MainThread:7468 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:10:51,076 INFO    MainThread:7468 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:10:51,076 INFO    MainThread:7468 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}
+2023-03-09 23:11:06,747 WARNING MsgRouterThr:7468 [router.py:message_loop():77] message_loop has been closed

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml

@@ -0,0 +1,40 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678374750.437312
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log

@@ -0,0 +1,8 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Device:          cuda
+        Images scaling:  0.5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:12:37.409031",
+    "startedAt": "2023-03-09T15:12:30.397388",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.8709499999995,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2503.176,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.665,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.337,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.575,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.932,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.175,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.232,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.76,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.386,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.645,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.742,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.602,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.547,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.785,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.892,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.913,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.64,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.636,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.065,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.447,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.937,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.397,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.317,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.778,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.691,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.442,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.403,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.367,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.909,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.332,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.151,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.712,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.725,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.245,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.616,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.382,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.774,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2509.295,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.964,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.445,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.114,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.765,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.667,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.401,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.018,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.418,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.384,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.864,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.165,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.535,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.818,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.705,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.344,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.589,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.395,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.818,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.757,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.398,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.645,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.011,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.031,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.391,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.45,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.348,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.345,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.644,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.027,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.544,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.622,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.011,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.421,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6551322937012
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log

@@ -0,0 +1,155 @@
+2023-03-09 23:12:30,443 INFO    StreamThr :11017 [internal.py:wandb_internal():90] W&B internal server running at pid: 11017, started at: 2023-03-09 23:12:30.441643
+2023-03-09 23:12:30,445 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:12:30,446 INFO    WriterThread:11017 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/run-qbg0lcfa.wandb
+2023-03-09 23:12:30,452 DEBUG   SenderThread:11017 [sender.py:send():336] send: header
+2023-03-09 23:12:30,452 DEBUG   SenderThread:11017 [sender.py:send():336] send: run
+2023-03-09 23:12:30,506 INFO    SenderThread:11017 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/qbg0lcfa
+2023-03-09 23:12:32,285 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:12:32,285 INFO    SenderThread:11017 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files
+2023-03-09 23:12:32,285 INFO    SenderThread:11017 [sender.py:_start_run_threads():1081] run started: qbg0lcfa with start time 1678374750.437312
+2023-03-09 23:12:32,285 DEBUG   SenderThread:11017 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:12:32,305 INFO    SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:12:32,305 DEBUG   SenderThread:11017 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:12:33,289 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-summary.json
+2023-03-09 23:12:37,286 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:12:37,306 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:12:37,368 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:12:37,373 DEBUG   HandlerThread:11017 [system_info.py:__init__():31] System info init
+2023-03-09 23:12:37,373 DEBUG   HandlerThread:11017 [system_info.py:__init__():46] System info init done
+2023-03-09 23:12:37,373 INFO    HandlerThread:11017 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:12:37,374 INFO    SystemMonitor:11017 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:12:37,374 INFO    HandlerThread:11017 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:12:37,374 INFO    SystemMonitor:11017 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:12:37,375 INFO    SystemMonitor:11017 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:12:37,375 INFO    SystemMonitor:11017 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:12:37,376 INFO    SystemMonitor:11017 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:12:37,376 INFO    SystemMonitor:11017 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:12:37,408 DEBUG   HandlerThread:11017 [system_info.py:probe():195] Probing system
+2023-03-09 23:12:37,420 DEBUG   HandlerThread:11017 [git.py:repo():40] git repository is invalid
+2023-03-09 23:12:37,420 DEBUG   HandlerThread:11017 [system_info.py:probe():240] Probing system done
+2023-03-09 23:12:37,421 DEBUG   HandlerThread:11017 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:12:37.409031', 'startedAt': '2023-03-09T15:12:30.397388', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.8709499999995, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2503.176, 'min': 0.0, 'max': 0.0}, {'current': 2499.665, 'min': 0.0, 'max': 0.0}, {'current': 2500.337, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2502.932, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.175, 'min': 0.0, 'max': 0.0}, {'current': 2499.232, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.76, 'min': 0.0, 'max': 0.0}, {'current': 2500.386, 'min': 0.0, 'max': 0.0}, {'current': 2499.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.742, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.602, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2502.547, 'min': 0.0, 'max': 0.0}, {'current': 2498.785, 'min': 0.0, 'max': 0.0}, {'current': 2503.892, 'min': 0.0, 'max': 0.0}, {'current': 2499.913, 'min': 0.0, 'max': 0.0}, {'current': 2499.64, 'min': 0.0, 'max': 0.0}, {'current': 2500.636, 'min': 0.0, 'max': 0.0}, {'current': 2505.065, 'min': 0.0, 'max': 0.0}, {'current': 2498.447, 'min': 0.0, 'max': 0.0}, {'current': 2500.937, 'min': 0.0, 'max': 0.0}, {'current': 2505.397, 'min': 0.0, 'max': 0.0}, {'current': 2499.317, 'min': 0.0, 'max': 0.0}, {'current': 2500.778, 'min': 0.0, 'max': 0.0}, {'current': 2500.691, 'min': 0.0, 'max': 0.0}, {'current': 2501.442, 'min': 0.0, 'max': 0.0}, {'current': 2501.403, 'min': 0.0, 'max': 0.0}, {'current': 2500.367, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.909, 'min': 0.0, 'max': 0.0}, {'current': 2502.332, 'min': 0.0, 'max': 0.0}, {'current': 2501.151, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.712, 'min': 0.0, 'max': 0.0}, {'current': 2503.725, 'min': 0.0, 'max': 0.0}, {'current': 2503.245, 'min': 0.0, 'max': 0.0}, {'current': 2499.616, 'min': 0.0, 'max': 0.0}, {'current': 2503.382, 'min': 0.0, 'max': 0.0}, {'current': 2504.774, 'min': 0.0, 'max': 0.0}, {'current': 2504.997, 'min': 0.0, 'max': 0.0}, {'current': 2509.295, 'min': 0.0, 'max': 0.0}, {'current': 2503.964, 'min': 0.0, 'max': 0.0}, {'current': 2500.445, 'min': 0.0, 'max': 0.0}, {'current': 2501.114, 'min': 0.0, 'max': 0.0}, {'current': 2499.765, 'min': 0.0, 'max': 0.0}, {'current': 2500.667, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2506.018, 'min': 0.0, 'max': 0.0}, {'current': 2500.418, 'min': 0.0, 'max': 0.0}, {'current': 2500.384, 'min': 0.0, 'max': 0.0}, {'current': 2505.864, 'min': 0.0, 'max': 0.0}, {'current': 2505.165, 'min': 0.0, 'max': 0.0}, {'current': 2507.535, 'min': 0.0, 'max': 0.0}, {'current': 2506.818, 'min': 0.0, 'max': 0.0}, {'current': 2504.705, 'min': 0.0, 'max': 0.0}, {'current': 2500.344, 'min': 0.0, 'max': 0.0}, {'current': 2503.589, 'min': 0.0, 'max': 0.0}, {'current': 2504.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.818, 'min': 0.0, 'max': 0.0}, {'current': 2503.757, 'min': 0.0, 'max': 0.0}, {'current': 2504.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.645, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.031, 'min': 0.0, 'max': 0.0}, {'current': 2500.391, 'min': 0.0, 'max': 0.0}, {'current': 2500.45, 'min': 0.0, 'max': 0.0}, {'current': 2505.348, 'min': 0.0, 'max': 0.0}, {'current': 2505.345, 'min': 0.0, 'max': 0.0}, {'current': 2503.644, 'min': 0.0, 'max': 0.0}, {'current': 2507.027, 'min': 0.0, 'max': 0.0}, {'current': 2499.544, 'min': 0.0, 'max': 0.0}, {'current': 2500.622, 'min': 0.0, 'max': 0.0}, {'current': 2501.011, 'min': 0.0, 'max': 0.0}, {'current': 2501.421, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6551322937012}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:12:37,421 INFO    HandlerThread:11017 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:12:37,421 INFO    HandlerThread:11017 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:12:37,421 DEBUG   HandlerThread:11017 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:12:37,486 DEBUG   HandlerThread:11017 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:12:37,486 DEBUG   HandlerThread:11017 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:12:38,197 DEBUG   HandlerThread:11017 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:12:38,229 INFO    HandlerThread:11017 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:12:38,232 DEBUG   SenderThread:11017 [sender.py:send():336] send: files
+2023-03-09 23:12:38,232 INFO    SenderThread:11017 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:12:38,248 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:12:38,268 DEBUG   SenderThread:11017 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:12:38,472 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/conda-environment.yaml
+2023-03-09 23:12:38,472 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/wandb-metadata.json
+2023-03-09 23:12:38,472 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/requirements.txt
+2023-03-09 23:12:38,720 DEBUG   SenderThread:11017 [sender.py:send():336] send: telemetry
+2023-03-09 23:12:38,721 DEBUG   SenderThread:11017 [sender.py:send():336] send: config
+2023-03-09 23:12:39,475 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log
+2023-03-09 23:12:39,490 INFO    wandb-upload_0:11017 [upload_job.py:push():138] Uploaded file /tmp/tmp03hehnd5wandb/0g7l9wyo-wandb-metadata.json
+2023-03-09 23:12:41,477 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/output.log
+2023-03-09 23:12:42,727 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:12:47,727 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:12:52,728 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:12:53,241 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:12:53,242 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:12:57,729 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:12:58,243 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:02,735 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:03,257 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:03,497 INFO    Thread-13 :11017 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/files/config.yaml
+2023-03-09 23:13:08,259 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:08,259 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:13,260 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:13,261 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:18,261 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:18,262 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:23,261 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:23,263 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:28,262 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:28,265 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:33,263 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:33,266 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:37,377 DEBUG   SystemMonitor:11017 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:13:37,381 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:13:38,267 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:38,382 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:43,269 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:43,383 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:48,270 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:48,384 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:53,272 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:53,384 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:13:58,273 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:13:58,385 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:03,274 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:03,386 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:07,384 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:14:08,275 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:09,386 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:13,277 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:14,386 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:18,278 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:19,387 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:23,294 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:24,409 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:28,309 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:29,410 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:33,509 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:34,411 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:37,414 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:14:38,609 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:39,426 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:43,622 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:44,608 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:48,623 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:49,609 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:53,709 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:54,610 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:14:58,711 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:14:59,611 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:03,712 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:04,611 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:07,416 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:15:08,713 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:10,418 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:13,715 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:15,419 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:18,716 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:20,420 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:23,717 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:25,420 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:28,718 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:30,421 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:33,720 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:35,422 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:37,419 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:15:38,722 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:41,421 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:41,969 WARNING FileStreamThread:11017 [file_stream.py:request_with_retry():675] requests_with_retry encountered retryable exception: 502 Server Error: Bad Gateway for url: https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream. func: <bound method Session.post of <requests.sessions.Session object at 0x7f3cd90d0c50>>, args: ('https://api.wandb.ai/files/anony-moose-510831/U-Net/qbg0lcfa/file_stream',), kwargs: {'json': {'files': {'wandb-events.jsonl': {'offset': 4, 'content': ['{"system.disk": 34.0, "system.network.sent": 648043.93, "system.network.recv": 292258.87, "system.proc.memory.availableMB": 329995.35, "system.memory": 14.43, "system.proc.memory.rssMB": 22483.33, "system.proc.memory.percent": 5.83, "system.cpu": 0.01, "system.cpu.0.cpu_percent": 13.63, "system.cpu.1.cpu_percent": 14.26, "system.cpu.2.cpu_percent": 13.02, "system.cpu.3.cpu_percent": 11.75, "system.cpu.4.cpu_percent": 11.36, "system.cpu.5.cpu_percent": 13.88, "system.cpu.6.cpu_percent": 11.97, "system.cpu.7.cpu_percent": 11.55, "system.cpu.8.cpu_percent": 10.73, "system.cpu.9.cpu_percent": 11.03, "system.cpu.10.cpu_percent": 12.05, "system.cpu.11.cpu_percent": 12.08, "system.cpu.12.cpu_percent": 12.5, "system.cpu.13.cpu_percent": 10.96, "system.cpu.14.cpu_percent": 11.1, "system.cpu.15.cpu_percent": 11.49, "system.cpu.16.cpu_percent": 11.22, "system.cpu.17.cpu_percent": 11.7, "system.cpu.18.cpu_percent": 12.28, "system.cpu.19.cpu_percent": 10.81, "system.cpu.20.cpu_percent": 13.6, "system.cpu.21.cpu_percent": 13.91, "system.cpu.22.cpu_percent": 13.94, "system.cpu.23.cpu_percent": 11.47, "system.cpu.24.cpu_percent": 10.25, "system.cpu.25.cpu_percent": 12.91, "system.cpu.26.cpu_percent": 10.18, "system.cpu.27.cpu_percent": 10.71, "system.cpu.28.cpu_percent": 10.51, "system.cpu.29.cpu_percent": 9.33, "system.cpu.30.cpu_percent": 10.98, "system.cpu.31.cpu_percent": 10.93, "system.cpu.32.cpu_percent": 9.93, "system.cpu.33.cpu_percent": 10.65, "system.cpu.34.cpu_percent": 11.46, "system.cpu.35.cpu_percent": 9.93, "system.cpu.36.cpu_percent": 10.47, "system.cpu.37.cpu_percent": 10.19, "system.cpu.38.cpu_percent": 9.21, "system.cpu.39.cpu_percent": 10.19, "system.cpu.40.cpu_percent": 12.19, "system.cpu.41.cpu_percent": 12.61, "system.cpu.42.cpu_percent": 18.98, "system.cpu.43.cpu_percent": 11.38, "system.cpu.44.cpu_percent": 11.84, "system.cpu.45.cpu_percent": 11.58, "system.cpu.46.cpu_percent": 10.05, "system.cpu.47.cpu_percent": 11.54, "system.cpu.48.cpu_percent": 11.37, "system.cpu.49.cpu_percent": 12.76, "system.cpu.50.cpu_percent": 11.16, "system.cpu.51.cpu_percent": 11.69, "system.cpu.52.cpu_percent": 10.77, "system.cpu.53.cpu_percent": 10.6, "system.cpu.54.cpu_percent": 10.82, "system.cpu.55.cpu_percent": 11.67, "system.cpu.56.cpu_percent": 17.88, "system.cpu.57.cpu_percent": 14.37, "system.cpu.58.cpu_percent": 10.75, "system.cpu.59.cpu_percent": 11.79, "system.cpu.60.cpu_percent": 11.43, "system.cpu.61.cpu_percent": 10.05, "system.cpu.62.cpu_percent": 9.11, "system.cpu.63.cpu_percent": 9.93, "system.cpu.64.cpu_percent": 11.24, "system.cpu.65.cpu_percent": 16.09, "system.cpu.66.cpu_percent": 12.53, "system.cpu.67.cpu_percent": 17.75, "system.cpu.68.cpu_percent": 12.15, "system.cpu.69.cpu_percent": 10.11, "system.cpu.70.cpu_percent": 9.68, "system.cpu.71.cpu_percent": 10.84, "system.cpu.72.cpu_percent": 16.69, "system.cpu.73.cpu_percent": 10.49, "system.cpu.74.cpu_percent": 10.3, "system.cpu.75.cpu_percent": 12.14, "system.cpu.76.cpu_percent": 12.36, "system.cpu.77.cpu_percent": 9.46, "system.cpu.78.cpu_percent": 10.52, "system.cpu.79.cpu_percent": 11.16, "system.proc.cpu.threads": 88, "system.gpu.0.memoryAllocated": 32.36, "system.gpu.1.memoryAllocated": 0.02, "system.gpu.0.memory": 0.0, "system.gpu.1.memory": 0.0, "system.gpu.0.gpu": 0.0, "system.gpu.1.gpu": 0.0, "system.gpu.0.temp": 56.0, "system.gpu.1.temp": 39.0, "system.gpu.0.powerWatts": 29.95, "system.gpu.1.powerWatts": 9.91, "system.gpu.0.powerPercent": 42.78, "system.gpu.1.powerPercent": 14.18, "_wandb": true, "_timestamp": 1678374937.415186, "_runtime": 186.977874}']}}, 'dropped': 0}}
+2023-03-09 23:15:43,723 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:46,421 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:48,724 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:51,422 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:53,725 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:15:56,423 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:15:58,726 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:01,423 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:03,728 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:06,424 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:07,490 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:16:08,729 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:11,492 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:13,730 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:16,492 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:18,732 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:21,493 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:23,733 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:26,494 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:28,734 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:31,494 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:33,736 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:36,495 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:37,493 DEBUG   SenderThread:11017 [sender.py:send():336] send: stats
+2023-03-09 23:16:38,737 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:16:42,508 DEBUG   HandlerThread:11017 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:16:45,728 INFO    MainThread:11017 [internal.py:handle_exit():77] Internal process exited

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log

@@ -0,0 +1,27 @@
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Configure stats pid to 9232
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:12:30,423 INFO    MainThread:9232 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug.log
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_231230-qbg0lcfa/logs/debug-internal.log
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:init():602] starting backend
+2023-03-09 23:12:30,424 INFO    MainThread:9232 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:12:30,435 INFO    MainThread:9232 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:12:30,437 INFO    MainThread:9232 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:12:30,440 INFO    MainThread:9232 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:12:30,441 INFO    MainThread:9232 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:12:32,284 INFO    MainThread:9232 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:12:37,307 INFO    MainThread:9232 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:12:37,307 INFO    MainThread:9232 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:12:38,240 INFO    MainThread:9232 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:12:38,241 INFO    MainThread:9232 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:12:38,241 INFO    MainThread:9232 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:12:38,241 INFO    MainThread:9232 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:12:38,242 INFO    MainThread:9232 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:12:38,243 INFO    MainThread:9232 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml

@@ -0,0 +1,40 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678375959.443294
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/output.log

@@ -0,0 +1,8 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Device:          cuda
+        Images scaling:  0.5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:32:46.785082",
+    "startedAt": "2023-03-09T15:32:39.401025",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.3775624999994,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.614,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.081,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.009,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.73,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.934,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.158,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.351,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.211,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.412,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.243,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.648,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.74,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.191,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.465,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.182,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.405,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.933,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.646,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.576,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.25,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.654,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.366,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.578,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.947,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.823,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.483,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.788,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.937,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.939,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.089,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.405,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.625,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.759,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.618,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.609,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.887,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.558,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.769,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.432,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.978,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.278,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.876,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.435,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.423,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.059,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.103,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.258,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.683,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.349,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.848,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.132,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.036,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.927,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.827,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.608,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.265,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.142,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.334,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.609,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.275,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.014,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.764,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.409,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.183,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.052,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.383,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.035,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.466,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.834,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.654,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.935,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6485137939453
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log

@@ -0,0 +1,94 @@
+2023-03-09 23:32:39,448 INFO    StreamThr :21832 [internal.py:wandb_internal():90] W&B internal server running at pid: 21832, started at: 2023-03-09 23:32:39.446798
+2023-03-09 23:32:39,450 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:32:39,451 INFO    WriterThread:21832 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/run-58pb6znt.wandb
+2023-03-09 23:32:39,456 DEBUG   SenderThread:21832 [sender.py:send():336] send: header
+2023-03-09 23:32:39,456 DEBUG   SenderThread:21832 [sender.py:send():336] send: run
+2023-03-09 23:32:39,536 INFO    SenderThread:21832 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/58pb6znt
+2023-03-09 23:32:41,675 INFO    SenderThread:21832 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files
+2023-03-09 23:32:41,675 INFO    SenderThread:21832 [sender.py:_start_run_threads():1081] run started: 58pb6znt with start time 1678375959.443294
+2023-03-09 23:32:41,676 DEBUG   SenderThread:21832 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:32:41,676 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:32:41,715 INFO    SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:32:41,716 DEBUG   SenderThread:21832 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:32:42,679 INFO    Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-summary.json
+2023-03-09 23:32:46,676 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:32:46,717 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:32:46,737 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:32:46,742 DEBUG   HandlerThread:21832 [system_info.py:__init__():31] System info init
+2023-03-09 23:32:46,742 DEBUG   HandlerThread:21832 [system_info.py:__init__():46] System info init done
+2023-03-09 23:32:46,742 INFO    HandlerThread:21832 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:32:46,742 INFO    SystemMonitor:21832 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:32:46,742 INFO    HandlerThread:21832 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:32:46,743 INFO    SystemMonitor:21832 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:32:46,744 INFO    SystemMonitor:21832 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:32:46,744 INFO    SystemMonitor:21832 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:32:46,745 INFO    SystemMonitor:21832 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:32:46,746 INFO    SystemMonitor:21832 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:32:46,785 DEBUG   HandlerThread:21832 [system_info.py:probe():195] Probing system
+2023-03-09 23:32:46,806 DEBUG   HandlerThread:21832 [git.py:repo():40] git repository is invalid
+2023-03-09 23:32:46,806 DEBUG   HandlerThread:21832 [system_info.py:probe():240] Probing system done
+2023-03-09 23:32:46,806 DEBUG   HandlerThread:21832 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:32:46.785082', 'startedAt': '2023-03-09T15:32:39.401025', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.3775624999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.614, 'min': 0.0, 'max': 0.0}, {'current': 2501.081, 'min': 0.0, 'max': 0.0}, {'current': 2500.009, 'min': 0.0, 'max': 0.0}, {'current': 2500.73, 'min': 0.0, 'max': 0.0}, {'current': 2502.934, 'min': 0.0, 'max': 0.0}, {'current': 2500.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.351, 'min': 0.0, 'max': 0.0}, {'current': 2500.211, 'min': 0.0, 'max': 0.0}, {'current': 2501.412, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.648, 'min': 0.0, 'max': 0.0}, {'current': 2499.74, 'min': 0.0, 'max': 0.0}, {'current': 2500.191, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2498.465, 'min': 0.0, 'max': 0.0}, {'current': 2500.182, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2501.933, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.646, 'min': 0.0, 'max': 0.0}, {'current': 2499.576, 'min': 0.0, 'max': 0.0}, {'current': 2499.25, 'min': 0.0, 'max': 0.0}, {'current': 2499.654, 'min': 0.0, 'max': 0.0}, {'current': 2500.366, 'min': 0.0, 'max': 0.0}, {'current': 2501.578, 'min': 0.0, 'max': 0.0}, {'current': 2499.947, 'min': 0.0, 'max': 0.0}, {'current': 2503.823, 'min': 0.0, 'max': 0.0}, {'current': 2501.483, 'min': 0.0, 'max': 0.0}, {'current': 2499.788, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.937, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2500.089, 'min': 0.0, 'max': 0.0}, {'current': 2500.405, 'min': 0.0, 'max': 0.0}, {'current': 2499.625, 'min': 0.0, 'max': 0.0}, {'current': 2504.001, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2500.887, 'min': 0.0, 'max': 0.0}, {'current': 2504.558, 'min': 0.0, 'max': 0.0}, {'current': 2500.769, 'min': 0.0, 'max': 0.0}, {'current': 2500.432, 'min': 0.0, 'max': 0.0}, {'current': 2504.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.278, 'min': 0.0, 'max': 0.0}, {'current': 2503.876, 'min': 0.0, 'max': 0.0}, {'current': 2500.435, 'min': 0.0, 'max': 0.0}, {'current': 2504.423, 'min': 0.0, 'max': 0.0}, {'current': 2500.059, 'min': 0.0, 'max': 0.0}, {'current': 2505.103, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2500.258, 'min': 0.0, 'max': 0.0}, {'current': 2503.683, 'min': 0.0, 'max': 0.0}, {'current': 2504.349, 'min': 0.0, 'max': 0.0}, {'current': 2499.848, 'min': 0.0, 'max': 0.0}, {'current': 2505.132, 'min': 0.0, 'max': 0.0}, {'current': 2505.036, 'min': 0.0, 'max': 0.0}, {'current': 2504.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.827, 'min': 0.0, 'max': 0.0}, {'current': 2505.608, 'min': 0.0, 'max': 0.0}, {'current': 2501.265, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.142, 'min': 0.0, 'max': 0.0}, {'current': 2506.334, 'min': 0.0, 'max': 0.0}, {'current': 2499.609, 'min': 0.0, 'max': 0.0}, {'current': 2504.275, 'min': 0.0, 'max': 0.0}, {'current': 2501.014, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2498.764, 'min': 0.0, 'max': 0.0}, {'current': 2500.409, 'min': 0.0, 'max': 0.0}, {'current': 2505.183, 'min': 0.0, 'max': 0.0}, {'current': 2500.052, 'min': 0.0, 'max': 0.0}, {'current': 2500.383, 'min': 0.0, 'max': 0.0}, {'current': 2503.035, 'min': 0.0, 'max': 0.0}, {'current': 2504.466, 'min': 0.0, 'max': 0.0}, {'current': 2503.834, 'min': 0.0, 'max': 0.0}, {'current': 2500.654, 'min': 0.0, 'max': 0.0}, {'current': 2503.935, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6485137939453}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:32:46,807 INFO    HandlerThread:21832 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:32:46,807 INFO    HandlerThread:21832 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:32:46,807 DEBUG   HandlerThread:21832 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:32:46,861 DEBUG   HandlerThread:21832 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:32:46,861 DEBUG   HandlerThread:21832 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:32:47,586 DEBUG   HandlerThread:21832 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:32:47,646 INFO    HandlerThread:21832 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:32:47,649 DEBUG   SenderThread:21832 [sender.py:send():336] send: files
+2023-03-09 23:32:47,649 INFO    SenderThread:21832 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:32:47,666 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:32:47,666 DEBUG   SenderThread:21832 [sender.py:send():336] send: telemetry
+2023-03-09 23:32:47,682 DEBUG   SenderThread:21832 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:32:47,868 INFO    Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/conda-environment.yaml
+2023-03-09 23:32:47,878 INFO    Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/requirements.txt
+2023-03-09 23:32:47,888 INFO    Thread-13 :21832 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/wandb-metadata.json
+2023-03-09 23:32:48,228 DEBUG   SenderThread:21832 [sender.py:send():336] send: config
+2023-03-09 23:32:48,916 INFO    wandb-upload_0:21832 [upload_job.py:push():138] Uploaded file /tmp/tmpckj69pbswandb/vyd13lcr-wandb-metadata.json
+2023-03-09 23:32:52,258 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:32:57,259 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:02,260 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:02,657 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:02,657 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:07,260 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:07,659 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:12,266 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:12,660 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:13,878 INFO    Thread-13 :21832 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/files/config.yaml
+2023-03-09 23:33:17,661 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:18,617 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:22,662 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:23,618 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:27,663 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:28,619 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:32,665 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:33,619 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:37,666 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:38,620 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:42,667 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:43,621 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:46,746 DEBUG   SystemMonitor:21832 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:33:46,751 DEBUG   SenderThread:21832 [sender.py:send():336] send: stats
+2023-03-09 23:33:47,668 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:48,752 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:52,669 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:53,753 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:33:57,671 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:33:58,754 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:02,672 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:34:03,754 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:07,673 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:34:08,755 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:12,674 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:34:13,756 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:16,754 DEBUG   SenderThread:21832 [sender.py:send():336] send: stats
+2023-03-09 23:34:17,676 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:34:19,756 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:22,677 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:34:24,756 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:28,873 ERROR   memory    :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104)
+2023-03-09 23:34:28,873 ERROR   memory    :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104)
+2023-03-09 23:34:28,915 ERROR   cpu       :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104)
+2023-03-09 23:34:28,917 ERROR   cpu       :21832 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=20104)
+2023-03-09 23:34:29,757 DEBUG   HandlerThread:21832 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:34:29,823 INFO    MainThread:21832 [internal.py:handle_exit():77] Internal process exited

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log

@@ -0,0 +1,27 @@
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Configure stats pid to 20104
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug.log
+2023-03-09 23:32:39,428 INFO    MainThread:20104 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233239-58pb6znt/logs/debug-internal.log
+2023-03-09 23:32:39,429 INFO    MainThread:20104 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:32:39,429 INFO    MainThread:20104 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:32:39,429 INFO    MainThread:20104 [wandb_init.py:init():602] starting backend
+2023-03-09 23:32:39,429 INFO    MainThread:20104 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:32:39,440 INFO    MainThread:20104 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:32:39,442 INFO    MainThread:20104 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:32:39,447 INFO    MainThread:20104 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:32:39,448 INFO    MainThread:20104 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:32:41,675 INFO    MainThread:20104 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:32:46,718 INFO    MainThread:20104 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:32:46,718 INFO    MainThread:20104 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:32:47,656 INFO    MainThread:20104 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:32:47,656 INFO    MainThread:20104 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:32:47,656 INFO    MainThread:20104 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:32:47,656 INFO    MainThread:20104 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:32:47,657 INFO    MainThread:20104 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:32:47,658 INFO    MainThread:20104 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/config.yaml

@@ -0,0 +1,22 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: true
+    python_version: 3.7.11
+    start_time: 1678376153.802776
+    t:
+      1:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 2
+      - 5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log

@@ -0,0 +1,7 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Images scaling:  0.5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:36:00.197376",
+    "startedAt": "2023-03-09T15:35:53.764072",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "hbfd862b3e0541989dd59bcbcb44c6eb-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.5297500000006,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2500.096,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.615,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.893,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.591,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.521,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.243,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.638,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.144,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.868,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.649,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.109,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.04,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.822,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.83,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.476,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.859,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.572,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.283,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.012,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.618,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.178,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.045,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.773,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.834,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.236,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.061,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.96,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.082,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.189,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.323,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.69,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.152,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.988,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.041,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.322,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.397,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.099,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.921,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.55,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.546,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.646,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.091,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.889,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.029,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.008,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.061,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.807,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.396,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.189,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.804,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.96,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.432,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.638,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.016,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.045,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.726,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.212,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.968,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.027,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.15,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.153,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.227,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.381,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.1,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.225,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.803,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.722,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.718,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.523,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.426,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.206,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.864,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.673,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.64935302734375
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log

@@ -0,0 +1,51 @@
+2023-03-09 23:35:53,808 INFO    StreamThr :24054 [internal.py:wandb_internal():90] W&B internal server running at pid: 24054, started at: 2023-03-09 23:35:53.806812
+2023-03-09 23:35:53,810 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:35:53,811 INFO    WriterThread:24054 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/run-yc1qtvil.wandb
+2023-03-09 23:35:53,828 DEBUG   SenderThread:24054 [sender.py:send():336] send: header
+2023-03-09 23:35:53,828 DEBUG   SenderThread:24054 [sender.py:send():336] send: run
+2023-03-09 23:35:53,864 INFO    SenderThread:24054 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/yc1qtvil
+2023-03-09 23:35:55,068 INFO    SenderThread:24054 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files
+2023-03-09 23:35:55,069 INFO    SenderThread:24054 [sender.py:_start_run_threads():1081] run started: yc1qtvil with start time 1678376153.802776
+2023-03-09 23:35:55,069 DEBUG   SenderThread:24054 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:35:55,069 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:35:55,122 INFO    SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:35:55,123 DEBUG   SenderThread:24054 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:35:56,073 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-summary.json
+2023-03-09 23:36:00,070 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:36:00,124 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:36:00,144 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:36:00,154 DEBUG   HandlerThread:24054 [system_info.py:__init__():31] System info init
+2023-03-09 23:36:00,154 DEBUG   HandlerThread:24054 [system_info.py:__init__():46] System info init done
+2023-03-09 23:36:00,155 INFO    HandlerThread:24054 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:36:00,155 INFO    SystemMonitor:24054 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:36:00,155 INFO    HandlerThread:24054 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:36:00,156 INFO    SystemMonitor:24054 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:36:00,156 INFO    SystemMonitor:24054 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:36:00,157 INFO    SystemMonitor:24054 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:36:00,157 INFO    SystemMonitor:24054 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:36:00,158 INFO    SystemMonitor:24054 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:36:00,197 DEBUG   HandlerThread:24054 [system_info.py:probe():195] Probing system
+2023-03-09 23:36:00,234 DEBUG   HandlerThread:24054 [git.py:repo():40] git repository is invalid
+2023-03-09 23:36:00,234 DEBUG   HandlerThread:24054 [system_info.py:probe():240] Probing system done
+2023-03-09 23:36:00,234 DEBUG   HandlerThread:24054 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:36:00.197376', 'startedAt': '2023-03-09T15:35:53.764072', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'hbfd862b3e0541989dd59bcbcb44c6eb-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.5297500000006, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.096, 'min': 0.0, 'max': 0.0}, {'current': 2502.615, 'min': 0.0, 'max': 0.0}, {'current': 2500.893, 'min': 0.0, 'max': 0.0}, {'current': 2499.591, 'min': 0.0, 'max': 0.0}, {'current': 2499.521, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2501.243, 'min': 0.0, 'max': 0.0}, {'current': 2499.638, 'min': 0.0, 'max': 0.0}, {'current': 2499.144, 'min': 0.0, 'max': 0.0}, {'current': 2501.868, 'min': 0.0, 'max': 0.0}, {'current': 2499.649, 'min': 0.0, 'max': 0.0}, {'current': 2501.109, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.822, 'min': 0.0, 'max': 0.0}, {'current': 2498.83, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.859, 'min': 0.0, 'max': 0.0}, {'current': 2499.572, 'min': 0.0, 'max': 0.0}, {'current': 2499.283, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.618, 'min': 0.0, 'max': 0.0}, {'current': 2501.178, 'min': 0.0, 'max': 0.0}, {'current': 2501.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.773, 'min': 0.0, 'max': 0.0}, {'current': 2501.834, 'min': 0.0, 'max': 0.0}, {'current': 2506.236, 'min': 0.0, 'max': 0.0}, {'current': 2501.061, 'min': 0.0, 'max': 0.0}, {'current': 2502.96, 'min': 0.0, 'max': 0.0}, {'current': 2501.082, 'min': 0.0, 'max': 0.0}, {'current': 2499.189, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.323, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2502.152, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.041, 'min': 0.0, 'max': 0.0}, {'current': 2500.322, 'min': 0.0, 'max': 0.0}, {'current': 2500.397, 'min': 0.0, 'max': 0.0}, {'current': 2504.099, 'min': 0.0, 'max': 0.0}, {'current': 2503.921, 'min': 0.0, 'max': 0.0}, {'current': 2500.55, 'min': 0.0, 'max': 0.0}, {'current': 2500.546, 'min': 0.0, 'max': 0.0}, {'current': 2505.646, 'min': 0.0, 'max': 0.0}, {'current': 2501.091, 'min': 0.0, 'max': 0.0}, {'current': 2499.889, 'min': 0.0, 'max': 0.0}, {'current': 2500.029, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2504.061, 'min': 0.0, 'max': 0.0}, {'current': 2505.807, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2500.189, 'min': 0.0, 'max': 0.0}, {'current': 2506.804, 'min': 0.0, 'max': 0.0}, {'current': 2500.96, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.432, 'min': 0.0, 'max': 0.0}, {'current': 2506.638, 'min': 0.0, 'max': 0.0}, {'current': 2507.016, 'min': 0.0, 'max': 0.0}, {'current': 2504.045, 'min': 0.0, 'max': 0.0}, {'current': 2500.726, 'min': 0.0, 'max': 0.0}, {'current': 2504.212, 'min': 0.0, 'max': 0.0}, {'current': 2499.968, 'min': 0.0, 'max': 0.0}, {'current': 2500.027, 'min': 0.0, 'max': 0.0}, {'current': 2504.15, 'min': 0.0, 'max': 0.0}, {'current': 2505.153, 'min': 0.0, 'max': 0.0}, {'current': 2499.227, 'min': 0.0, 'max': 0.0}, {'current': 2500.381, 'min': 0.0, 'max': 0.0}, {'current': 2504.1, 'min': 0.0, 'max': 0.0}, {'current': 2504.225, 'min': 0.0, 'max': 0.0}, {'current': 2500.803, 'min': 0.0, 'max': 0.0}, {'current': 2504.722, 'min': 0.0, 'max': 0.0}, {'current': 2504.718, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2500.426, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.206, 'min': 0.0, 'max': 0.0}, {'current': 2501.864, 'min': 0.0, 'max': 0.0}, {'current': 2504.673, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64935302734375}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:36:00,235 INFO    HandlerThread:24054 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:36:00,235 INFO    HandlerThread:24054 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:36:00,235 DEBUG   HandlerThread:24054 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:36:00,274 DEBUG   HandlerThread:24054 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:36:00,274 DEBUG   HandlerThread:24054 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:36:00,993 DEBUG   HandlerThread:24054 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:36:01,023 INFO    HandlerThread:24054 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:36:01,027 DEBUG   SenderThread:24054 [sender.py:send():336] send: files
+2023-03-09 23:36:01,027 INFO    SenderThread:24054 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:36:01,036 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:36:01,049 DEBUG   SenderThread:24054 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:36:01,270 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/conda-environment.yaml
+2023-03-09 23:36:01,271 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/wandb-metadata.json
+2023-03-09 23:36:01,271 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/requirements.txt
+2023-03-09 23:36:01,524 DEBUG   SenderThread:24054 [sender.py:send():336] send: telemetry
+2023-03-09 23:36:01,524 DEBUG   SenderThread:24054 [sender.py:send():336] send: config
+2023-03-09 23:36:02,264 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log
+2023-03-09 23:36:02,406 INFO    wandb-upload_0:24054 [upload_job.py:push():138] Uploaded file /tmp/tmpzwt8zs5awandb/qutsputw-wandb-metadata.json
+2023-03-09 23:36:04,279 INFO    Thread-13 :24054 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/files/output.log
+2023-03-09 23:36:05,528 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:36:10,529 DEBUG   HandlerThread:24054 [handler.py:handle_request():144] handle_request: status_report

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log

@@ -0,0 +1,27 @@
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Configure stats pid to 22330
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug.log
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_233553-yc1qtvil/logs/debug-internal.log
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:init():602] starting backend
+2023-03-09 23:35:53,789 INFO    MainThread:22330 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:35:53,800 INFO    MainThread:22330 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:35:53,802 INFO    MainThread:22330 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:35:53,806 INFO    MainThread:22330 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:35:53,807 INFO    MainThread:22330 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:35:55,067 INFO    MainThread:22330 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:36:00,125 INFO    MainThread:22330 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:36:00,125 INFO    MainThread:22330 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:36:01,034 INFO    MainThread:22330 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:36:01,034 INFO    MainThread:22330 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:36:01,034 INFO    MainThread:22330 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:36:01,034 INFO    MainThread:22330 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:36:01,035 INFO    MainThread:22330 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:36:01,036 INFO    MainThread:22330 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml

@@ -0,0 +1,39 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: false
+    python_version: 3.7.11
+    start_time: 1678376631.533235
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log

@@ -0,0 +1,7 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Images scaling:  0.5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:44:01.780995",
+    "startedAt": "2023-03-09T15:43:51.492578",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "n1d6fa00775e48a18f80acc1355070be-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2501.7918,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2500.007,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.205,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.476,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.178,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.693,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.211,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.485,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.073,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.962,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.188,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.333,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.473,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.557,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.576,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.927,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.451,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.543,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.573,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.677,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.653,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.368,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.888,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.582,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.338,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.318,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.259,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.245,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.213,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.943,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.789,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.016,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.944,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.978,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.631,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.601,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.717,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.733,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.492,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.656,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.187,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.269,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.918,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.545,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.833,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.211,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.345,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.134,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.032,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.317,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.961,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.426,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.555,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.063,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.827,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.429,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.572,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.604,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.86,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.01,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.523,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.704,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.206,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.531,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.875,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.19,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.872,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.063,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.075,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.623,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.85,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.509,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.271,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6454200744629
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log

@@ -0,0 +1,114 @@
+2023-03-09 23:43:51,539 INFO    StreamThr :1879 [internal.py:wandb_internal():90] W&B internal server running at pid: 1879, started at: 2023-03-09 23:43:51.537061
+2023-03-09 23:43:51,540 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:43:51,541 INFO    WriterThread:1879 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/run-w9y25r7k.wandb
+2023-03-09 23:43:51,546 DEBUG   SenderThread:1879 [sender.py:send():336] send: header
+2023-03-09 23:43:51,546 DEBUG   SenderThread:1879 [sender.py:send():336] send: run
+2023-03-09 23:43:51,616 INFO    SenderThread:1879 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/w9y25r7k
+2023-03-09 23:43:56,543 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:43:56,651 INFO    SenderThread:1879 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files
+2023-03-09 23:43:56,651 INFO    SenderThread:1879 [sender.py:_start_run_threads():1081] run started: w9y25r7k with start time 1678376631.533235
+2023-03-09 23:43:56,652 DEBUG   SenderThread:1879 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:43:56,652 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:43:56,654 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:43:56,675 INFO    SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:43:56,676 DEBUG   SenderThread:1879 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:43:57,655 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-summary.json
+2023-03-09 23:44:01,654 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:01,677 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:01,736 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:44:01,741 DEBUG   HandlerThread:1879 [system_info.py:__init__():31] System info init
+2023-03-09 23:44:01,741 DEBUG   HandlerThread:1879 [system_info.py:__init__():46] System info init done
+2023-03-09 23:44:01,741 INFO    HandlerThread:1879 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:44:01,741 INFO    SystemMonitor:1879 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:44:01,742 INFO    HandlerThread:1879 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:44:01,742 INFO    SystemMonitor:1879 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:44:01,743 INFO    SystemMonitor:1879 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:44:01,743 INFO    SystemMonitor:1879 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:44:01,744 INFO    SystemMonitor:1879 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:44:01,744 INFO    SystemMonitor:1879 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:44:01,780 DEBUG   HandlerThread:1879 [system_info.py:probe():195] Probing system
+2023-03-09 23:44:01,790 DEBUG   HandlerThread:1879 [git.py:repo():40] git repository is invalid
+2023-03-09 23:44:01,790 DEBUG   HandlerThread:1879 [system_info.py:probe():240] Probing system done
+2023-03-09 23:44:01,790 DEBUG   HandlerThread:1879 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:44:01.780995', 'startedAt': '2023-03-09T15:43:51.492578', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2501.7918, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2503.205, 'min': 0.0, 'max': 0.0}, {'current': 2500.476, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.178, 'min': 0.0, 'max': 0.0}, {'current': 2498.693, 'min': 0.0, 'max': 0.0}, {'current': 2499.211, 'min': 0.0, 'max': 0.0}, {'current': 2502.485, 'min': 0.0, 'max': 0.0}, {'current': 2500.073, 'min': 0.0, 'max': 0.0}, {'current': 2503.962, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2502.188, 'min': 0.0, 'max': 0.0}, {'current': 2503.333, 'min': 0.0, 'max': 0.0}, {'current': 2500.473, 'min': 0.0, 'max': 0.0}, {'current': 2499.557, 'min': 0.0, 'max': 0.0}, {'current': 2501.576, 'min': 0.0, 'max': 0.0}, {'current': 2501.927, 'min': 0.0, 'max': 0.0}, {'current': 2500.451, 'min': 0.0, 'max': 0.0}, {'current': 2499.543, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2499.677, 'min': 0.0, 'max': 0.0}, {'current': 2499.653, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2502.888, 'min': 0.0, 'max': 0.0}, {'current': 2503.582, 'min': 0.0, 'max': 0.0}, {'current': 2499.338, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.318, 'min': 0.0, 'max': 0.0}, {'current': 2501.259, 'min': 0.0, 'max': 0.0}, {'current': 2501.245, 'min': 0.0, 'max': 0.0}, {'current': 2501.213, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2499.789, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2499.944, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.631, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.601, 'min': 0.0, 'max': 0.0}, {'current': 2500.717, 'min': 0.0, 'max': 0.0}, {'current': 2505.733, 'min': 0.0, 'max': 0.0}, {'current': 2504.492, 'min': 0.0, 'max': 0.0}, {'current': 2505.656, 'min': 0.0, 'max': 0.0}, {'current': 2498.187, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2499.918, 'min': 0.0, 'max': 0.0}, {'current': 2504.545, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2504.833, 'min': 0.0, 'max': 0.0}, {'current': 2506.211, 'min': 0.0, 'max': 0.0}, {'current': 2506.345, 'min': 0.0, 'max': 0.0}, {'current': 2505.134, 'min': 0.0, 'max': 0.0}, {'current': 2501.032, 'min': 0.0, 'max': 0.0}, {'current': 2507.317, 'min': 0.0, 'max': 0.0}, {'current': 2505.961, 'min': 0.0, 'max': 0.0}, {'current': 2506.426, 'min': 0.0, 'max': 0.0}, {'current': 2498.555, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2503.827, 'min': 0.0, 'max': 0.0}, {'current': 2504.429, 'min': 0.0, 'max': 0.0}, {'current': 2502.572, 'min': 0.0, 'max': 0.0}, {'current': 2503.604, 'min': 0.0, 'max': 0.0}, {'current': 2504.86, 'min': 0.0, 'max': 0.0}, {'current': 2500.01, 'min': 0.0, 'max': 0.0}, {'current': 2504.523, 'min': 0.0, 'max': 0.0}, {'current': 2503.704, 'min': 0.0, 'max': 0.0}, {'current': 2503.206, 'min': 0.0, 'max': 0.0}, {'current': 2500.531, 'min': 0.0, 'max': 0.0}, {'current': 2502.875, 'min': 0.0, 'max': 0.0}, {'current': 2500.19, 'min': 0.0, 'max': 0.0}, {'current': 2500.872, 'min': 0.0, 'max': 0.0}, {'current': 2500.063, 'min': 0.0, 'max': 0.0}, {'current': 2505.075, 'min': 0.0, 'max': 0.0}, {'current': 2503.623, 'min': 0.0, 'max': 0.0}, {'current': 2504.85, 'min': 0.0, 'max': 0.0}, {'current': 2504.509, 'min': 0.0, 'max': 0.0}, {'current': 2504.271, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6454200744629}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:44:01,791 INFO    HandlerThread:1879 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:44:01,791 INFO    HandlerThread:1879 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:44:01,791 DEBUG   HandlerThread:1879 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:44:01,819 DEBUG   HandlerThread:1879 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:44:01,819 DEBUG   HandlerThread:1879 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:44:02,534 DEBUG   HandlerThread:1879 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:44:02,581 INFO    HandlerThread:1879 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:44:02,585 DEBUG   SenderThread:1879 [sender.py:send():336] send: files
+2023-03-09 23:44:02,585 INFO    SenderThread:1879 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:44:02,600 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:44:02,616 DEBUG   SenderThread:1879 [sender.py:send():336] send: telemetry
+2023-03-09 23:44:02,626 DEBUG   SenderThread:1879 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:44:02,790 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/conda-environment.yaml
+2023-03-09 23:44:02,803 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/wandb-metadata.json
+2023-03-09 23:44:02,819 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/requirements.txt
+2023-03-09 23:44:03,640 DEBUG   SenderThread:1879 [sender.py:send():336] send: config
+2023-03-09 23:44:03,732 INFO    wandb-upload_0:1879 [upload_job.py:push():138] Uploaded file /tmp/tmp01qe7zccwandb/qvio2pk7-wandb-metadata.json
+2023-03-09 23:44:03,793 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log
+2023-03-09 23:44:05,810 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/output.log
+2023-03-09 23:44:07,644 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:12,645 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:17,593 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:17,593 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:17,646 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:22,595 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:22,652 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:23,826 INFO    Thread-13 :1879 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/files/config.yaml
+2023-03-09 23:44:27,596 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:28,126 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:32,598 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:33,126 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:37,599 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:38,127 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:42,601 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:43,128 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:47,602 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:48,128 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:52,603 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:53,129 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:44:57,604 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:44:58,130 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:01,745 DEBUG   SystemMonitor:1879 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:45:01,750 DEBUG   SenderThread:1879 [sender.py:send():336] send: stats
+2023-03-09 23:45:02,606 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:03,751 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:07,607 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:08,752 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:12,608 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:13,752 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:17,609 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:18,753 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:22,610 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:23,754 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:27,612 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:28,755 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:31,752 DEBUG   SenderThread:1879 [sender.py:send():336] send: stats
+2023-03-09 23:45:32,613 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:34,754 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:37,614 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:39,755 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:42,617 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:44,755 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:47,618 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:49,756 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:52,619 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:54,757 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:45:57,620 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:45:59,757 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:46:01,755 DEBUG   SenderThread:1879 [sender.py:send():336] send: stats
+2023-03-09 23:46:02,621 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:46:05,757 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:46:07,623 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:46:10,758 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:46:12,624 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:46:15,758 DEBUG   HandlerThread:1879 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:46:15,881 ERROR   memory    :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77)
+2023-03-09 23:46:15,881 ERROR   memory    :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77)
+2023-03-09 23:46:15,955 ERROR   cpu       :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77)
+2023-03-09 23:46:15,957 ERROR   cpu       :1879 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=77)
+2023-03-09 23:46:16,690 WARNING StreamThr :1879 [internal.py:is_dead():416] Internal process exiting, parent pid 77 disappeared
+2023-03-09 23:46:16,690 ERROR   StreamThr :1879 [internal.py:wandb_internal():153] Internal process shutdown.
+2023-03-09 23:46:16,744 INFO    MainThread:1879 [internal.py:handle_exit():77] Internal process exited

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log

@@ -0,0 +1,28 @@
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Configure stats pid to 77
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:43:51,519 INFO    MainThread:77 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug.log
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_234351-w9y25r7k/logs/debug-internal.log
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:init():602] starting backend
+2023-03-09 23:43:51,520 INFO    MainThread:77 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:43:51,531 INFO    MainThread:77 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:43:51,532 INFO    MainThread:77 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:43:51,540 INFO    MainThread:77 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:43:51,541 INFO    MainThread:77 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:43:56,652 INFO    MainThread:77 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:44:01,678 INFO    MainThread:77 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:44:01,678 INFO    MainThread:77 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:44:02,591 INFO    MainThread:77 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:44:02,592 INFO    MainThread:77 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:44:02,592 INFO    MainThread:77 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:44:02,592 INFO    MainThread:77 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:44:02,593 INFO    MainThread:77 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:44:02,594 INFO    MainThread:77 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml

@@ -0,0 +1,39 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: false
+    python_version: 3.7.11
+    start_time: 1678377055.792902
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log

@@ -0,0 +1,8 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Images scaling:  0.5
+Epoch 1/5:   0%|                                                                                                                                                               | 0/5088 [00:00<?, ?img/s]

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:51:02.055192",
+    "startedAt": "2023-03-09T15:50:55.747682",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "n1d6fa00775e48a18f80acc1355070be-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2499.8833999999993,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.977,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.997,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.988,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.917,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.97,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.943,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.54,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2495.346,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.962,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.004,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.959,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.301,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.965,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.368,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.009,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.994,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.888,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.939,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.988,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.823,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.983,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.913,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.978,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.926,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.036,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.982,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.974,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.973,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.013,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2496.799,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.978,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.89,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.002,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.023,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.004,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.98,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.012,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.983,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.039,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.007,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.962,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.03,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.971,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.002,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.03,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.067,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.04,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.001,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.011,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.025,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.999,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.003,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.024,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.914,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.006,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.99,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.056,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.979,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.11,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.137,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.01,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.998,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.993,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.6466979980469
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug-internal.log

@@ -0,0 +1,71 @@
+2023-03-09 23:50:55,798 INFO    StreamThr :4276 [internal.py:wandb_internal():90] W&B internal server running at pid: 4276, started at: 2023-03-09 23:50:55.796150
+2023-03-09 23:50:55,800 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:50:55,801 INFO    WriterThread:4276 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/run-ay46c760.wandb
+2023-03-09 23:50:55,806 DEBUG   SenderThread:4276 [sender.py:send():336] send: header
+2023-03-09 23:50:55,806 DEBUG   SenderThread:4276 [sender.py:send():336] send: run
+2023-03-09 23:50:55,827 INFO    SenderThread:4276 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/ay46c760
+2023-03-09 23:50:56,963 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:50:56,964 INFO    SenderThread:4276 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files
+2023-03-09 23:50:56,964 INFO    SenderThread:4276 [sender.py:_start_run_threads():1081] run started: ay46c760 with start time 1678377055.792902
+2023-03-09 23:50:56,964 DEBUG   SenderThread:4276 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:50:57,005 INFO    SenderThread:4276 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:50:57,006 DEBUG   SenderThread:4276 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:50:57,967 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-summary.json
+2023-03-09 23:51:01,964 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:02,007 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:02,027 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:51:02,034 DEBUG   HandlerThread:4276 [system_info.py:__init__():31] System info init
+2023-03-09 23:51:02,034 DEBUG   HandlerThread:4276 [system_info.py:__init__():46] System info init done
+2023-03-09 23:51:02,034 INFO    HandlerThread:4276 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:51:02,034 INFO    SystemMonitor:4276 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:51:02,034 INFO    HandlerThread:4276 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:51:02,035 INFO    SystemMonitor:4276 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:51:02,036 INFO    SystemMonitor:4276 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:51:02,036 INFO    SystemMonitor:4276 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:51:02,037 INFO    SystemMonitor:4276 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:51:02,037 INFO    SystemMonitor:4276 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:51:02,055 DEBUG   HandlerThread:4276 [system_info.py:probe():195] Probing system
+2023-03-09 23:51:02,081 DEBUG   HandlerThread:4276 [git.py:repo():40] git repository is invalid
+2023-03-09 23:51:02,081 DEBUG   HandlerThread:4276 [system_info.py:probe():240] Probing system done
+2023-03-09 23:51:02,081 DEBUG   HandlerThread:4276 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:51:02.055192', 'startedAt': '2023-03-09T15:50:55.747682', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2499.8833999999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.977, 'min': 0.0, 'max': 0.0}, {'current': 2499.997, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.917, 'min': 0.0, 'max': 0.0}, {'current': 2499.97, 'min': 0.0, 'max': 0.0}, {'current': 2499.943, 'min': 0.0, 'max': 0.0}, {'current': 2499.54, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2495.346, 'min': 0.0, 'max': 0.0}, {'current': 2499.962, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.959, 'min': 0.0, 'max': 0.0}, {'current': 2499.301, 'min': 0.0, 'max': 0.0}, {'current': 2499.965, 'min': 0.0, 'max': 0.0}, {'current': 2500.368, 'min': 0.0, 'max': 0.0}, {'current': 2500.009, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.994, 'min': 0.0, 'max': 0.0}, {'current': 2499.888, 'min': 0.0, 'max': 0.0}, {'current': 2499.939, 'min': 0.0, 'max': 0.0}, {'current': 2499.988, 'min': 0.0, 'max': 0.0}, {'current': 2499.823, 'min': 0.0, 'max': 0.0}, {'current': 2499.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}, {'current': 2499.913, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.926, 'min': 0.0, 'max': 0.0}, {'current': 2500.036, 'min': 0.0, 'max': 0.0}, {'current': 2499.982, 'min': 0.0, 'max': 0.0}, {'current': 2499.974, 'min': 0.0, 'max': 0.0}, {'current': 2499.973, 'min': 0.0, 'max': 0.0}, {'current': 2500.013, 'min': 0.0, 'max': 0.0}, {'current': 2496.799, 'min': 0.0, 'max': 0.0}, {'current': 2499.978, 'min': 0.0, 'max': 0.0}, {'current': 2499.89, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2500.023, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.004, 'min': 0.0, 'max': 0.0}, {'current': 2499.98, 'min': 0.0, 'max': 0.0}, {'current': 2500.012, 'min': 0.0, 'max': 0.0}, {'current': 2499.983, 'min': 0.0, 'max': 0.0}, {'current': 2500.039, 'min': 0.0, 'max': 0.0}, {'current': 2500.007, 'min': 0.0, 'max': 0.0}, {'current': 2499.962, 'min': 0.0, 'max': 0.0}, {'current': 2500.03, 'min': 0.0, 'max': 0.0}, {'current': 2499.971, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.002, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2500.03, 'min': 0.0, 'max': 0.0}, {'current': 2500.067, 'min': 0.0, 'max': 0.0}, {'current': 2500.04, 'min': 0.0, 'max': 0.0}, {'current': 2500.001, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.011, 'min': 0.0, 'max': 0.0}, {'current': 2500.025, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2499.999, 'min': 0.0, 'max': 0.0}, {'current': 2500.003, 'min': 0.0, 'max': 0.0}, {'current': 2500.024, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2499.914, 'min': 0.0, 'max': 0.0}, {'current': 2500.006, 'min': 0.0, 'max': 0.0}, {'current': 2499.99, 'min': 0.0, 'max': 0.0}, {'current': 2500.056, 'min': 0.0, 'max': 0.0}, {'current': 2499.979, 'min': 0.0, 'max': 0.0}, {'current': 2500.11, 'min': 0.0, 'max': 0.0}, {'current': 2500.137, 'min': 0.0, 'max': 0.0}, {'current': 2500.01, 'min': 0.0, 'max': 0.0}, {'current': 2499.998, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2499.993, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.6466979980469}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:51:02,081 INFO    HandlerThread:4276 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:51:02,081 INFO    HandlerThread:4276 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:51:02,081 DEBUG   HandlerThread:4276 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:51:02,151 DEBUG   HandlerThread:4276 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:51:02,151 DEBUG   HandlerThread:4276 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:51:02,871 DEBUG   HandlerThread:4276 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:51:02,931 INFO    HandlerThread:4276 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:51:02,934 DEBUG   SenderThread:4276 [sender.py:send():336] send: files
+2023-03-09 23:51:02,934 INFO    SenderThread:4276 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:51:02,943 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:51:02,959 DEBUG   SenderThread:4276 [sender.py:send():336] send: telemetry
+2023-03-09 23:51:02,969 DEBUG   SenderThread:4276 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:51:03,108 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/conda-environment.yaml
+2023-03-09 23:51:03,131 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/requirements.txt
+2023-03-09 23:51:03,144 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/wandb-metadata.json
+2023-03-09 23:51:04,016 DEBUG   SenderThread:4276 [sender.py:send():336] send: config
+2023-03-09 23:51:04,093 INFO    wandb-upload_0:4276 [upload_job.py:push():138] Uploaded file /tmp/tmpnt2sknetwandb/70huozag-wandb-metadata.json
+2023-03-09 23:51:04,100 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log
+2023-03-09 23:51:06,147 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/output.log
+2023-03-09 23:51:07,026 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:12,027 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:17,028 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:17,942 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:17,942 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:22,029 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:22,943 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:27,035 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:27,945 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:28,167 INFO    Thread-13 :4276 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/files/config.yaml
+2023-03-09 23:51:32,560 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:51:32,946 DEBUG   HandlerThread:4276 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:51:36,120 ERROR   memory    :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543)
+2023-03-09 23:51:36,121 ERROR   memory    :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543)
+2023-03-09 23:51:36,159 ERROR   cpu       :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543)
+2023-03-09 23:51:36,162 ERROR   cpu       :4276 [interfaces.py:monitor():141] Failed to sample metric: psutil.NoSuchProcess process no longer exists (pid=2543)
+2023-03-09 23:51:36,843 WARNING StreamThr :4276 [internal.py:is_dead():416] Internal process exiting, parent pid 2543 disappeared
+2023-03-09 23:51:36,843 ERROR   StreamThr :4276 [internal.py:wandb_internal():153] Internal process shutdown.
+2023-03-09 23:51:36,946 INFO    HandlerThread:4276 [handler.py:finish():842] shutting down handler
+2023-03-09 23:51:37,565 INFO    SenderThread:4276 [sender.py:finish():1504] shutting down sender
+2023-03-09 23:51:37,565 INFO    WriterThread:4276 [datastore.py:close():298] close: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/run-ay46c760.wandb
+2023-03-09 23:51:38,072 INFO    MainThread:4276 [internal.py:handle_exit():77] Internal process exited

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug.log

@@ -0,0 +1,27 @@
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Configure stats pid to 2543
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:50:55,775 INFO    MainThread:2543 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug.log
+2023-03-09 23:50:55,776 INFO    MainThread:2543 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235055-ay46c760/logs/debug-internal.log
+2023-03-09 23:50:55,776 INFO    MainThread:2543 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:50:55,776 INFO    MainThread:2543 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:50:55,776 INFO    MainThread:2543 [wandb_init.py:init():602] starting backend
+2023-03-09 23:50:55,776 INFO    MainThread:2543 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:50:55,790 INFO    MainThread:2543 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:50:55,792 INFO    MainThread:2543 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:50:55,796 INFO    MainThread:2543 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:50:55,797 INFO    MainThread:2543 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:50:56,962 INFO    MainThread:2543 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:51:02,007 INFO    MainThread:2543 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:51:02,007 INFO    MainThread:2543 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:51:02,940 INFO    MainThread:2543 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:51:02,941 INFO    MainThread:2543 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:51:02,941 INFO    MainThread:2543 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:51:02,941 INFO    MainThread:2543 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:51:02,942 INFO    MainThread:2543 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:51:02,943 INFO    MainThread:2543 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml

@@ -0,0 +1,39 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: false
+    python_version: 3.7.11
+    start_time: 1678377318.650455
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log

@@ -0,0 +1,7 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Images scaling:  0.5

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt

@@ -0,0 +1,136 @@
+appdirs==1.4.4
+argon2-cffi-bindings==21.2.0
+argon2-cffi==21.3.0
+asttokens==2.0.8
+astunparse==1.6.3
+attrs==21.4.0
+backcall==0.2.0
+beautifulsoup4==4.10.0
+bleach==5.0.0
+brotlipy==0.7.0
+certifi==2021.10.8
+cffi==1.14.6
+chardet==4.0.0
+click==8.1.3
+conda-build==3.21.5
+conda-package-handling==1.7.3
+conda==4.10.3
+cryptography==35.0.0
+cycler==0.11.0
+debugpy==1.6.0
+decorator==5.1.0
+defusedxml==0.7.1
+dnspython==2.1.0
+docker-pycreds==0.4.0
+easydict==1.10
+entrypoints==0.4
+fastjsonschema==2.15.3
+filelock==3.3.1
+fonttools==4.38.0
+gitdb==4.0.10
+gitpython==3.1.31
+glob2==0.7
+idna==2.10
+imageio==2.25.0
+importlib-metadata==4.11.3
+importlib-resources==5.7.1
+ipykernel==6.13.0
+ipython-genutils==0.2.0
+ipython==7.29.0
+ipywidgets==8.0.4
+jedi==0.18.0
+jinja2==3.1.1
+json5==0.9.6
+jsonschema==4.4.0
+jupyter-client==7.3.0
+jupyter-core==4.10.0
+jupyterlab-pygments==0.2.2
+jupyterlab-server==1.2.0
+jupyterlab-widgets==3.0.5
+jupyterlab==2.2.5
+kiwisolver==1.4.4
+libarchive-c==2.9
+markupsafe==2.0.1
+matplotlib-inline==0.1.2
+matplotlib==3.5.3
+mindspore-cuda11-dev==2.0.0.dev20221108
+mindspore-dev==2.0.0.dev20230109
+minkowskiengine==0.5.4
+mistune==0.8.4
+mkl-fft==1.3.1
+mkl-random==1.2.2
+mkl-service==2.4.0
+ms-adapter==0.1.0
+msadapter==0.0.1a0
+nbclient==0.6.0
+nbconvert==6.5.0
+nbformat==5.3.0
+nest-asyncio==1.5.5
+networkx==2.6.3
+ninja==1.11.1
+notebook==6.4.11
+numpy==1.21.2
+olefile==0.46
+open3d-python==0.7.0.0
+opencv-python==4.6.0.66
+packaging==21.3
+pandas==1.3.5
+pandocfilters==1.5.0
+parso==0.8.2
+pathtools==0.1.2
+pexpect==4.8.0
+pickleshare==0.7.5
+pillow==8.4.0
+pip==21.0.1
+pkginfo==1.7.1
+prometheus-client==0.14.1
+prompt-toolkit==3.0.20
+protobuf==3.20.3
+psutil==5.8.0
+ptyprocess==0.7.0
+pycosat==0.6.3
+pycparser==2.20
+pygments==2.10.0
+pyopenssl==20.0.1
+pyparsing==3.0.8
+pypng==0.20220715.0
+pyrsistent==0.18.1
+pysocks==1.7.1
+python-dateutil==2.8.2
+python-etcd==0.4.5
+pytz==2021.3
+pywavelets==1.3.0
+pyyaml==6.0
+pyzmq==22.3.0
+requests==2.25.1
+ruamel-yaml-conda==0.15.100
+scikit-image==0.19.3
+scipy==1.7.3
+send2trash==1.8.0
+sentry-sdk==1.16.0
+setproctitle==1.3.2
+setuptools==58.0.4
+six==1.16.0
+smmap==5.0.0
+soupsieve==2.2.1
+tensorboardx==2.6
+terminado==0.13.3
+tifffile==2021.11.2
+tinycss2==1.1.1
+torch==1.10.0
+torchac==0.9.3
+torchelastic==0.2.0
+torchtext==0.11.0
+torchvision==0.11.1
+tornado==6.1
+tqdm==4.61.2
+traitlets==5.1.0
+typing-extensions==3.10.0.2
+urllib3==1.26.14
+wandb==0.13.11
+wcwidth==0.2.5
+webencodings==0.5.1
+wheel==0.36.2
+widgetsnbextension==4.0.5
+xlrd==1.2.0
+zipp==3.8.0

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json

@@ -0,0 +1,443 @@
+{
+    "os": "Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid",
+    "python": "3.7.11",
+    "heartbeatAt": "2023-03-09T15:55:25.128694",
+    "startedAt": "2023-03-09T15:55:18.607606",
+    "docker": null,
+    "cuda": null,
+    "args": [],
+    "state": "running",
+    "program": "train.py",
+    "codePath": "train.py",
+    "host": "n1d6fa00775e48a18f80acc1355070be-task0-0",
+    "username": "root",
+    "executable": "/opt/conda/bin/python",
+    "cpu_count": 40,
+    "cpu_count_logical": 80,
+    "cpu_freq": {
+        "current": 2502.0496374999993,
+        "min": 0.0,
+        "max": 0.0
+    },
+    "cpu_freq_per_core": [
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.767,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.475,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.398,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.156,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.385,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.678,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.643,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.163,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.395,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.117,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.927,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2497.953,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.177,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.15,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.575,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.649,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.6,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.068,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.553,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.434,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.396,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.163,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.428,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.283,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.769,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.975,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.573,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.008,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.441,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.995,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.077,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.185,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.401,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.465,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.575,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2499.949,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.373,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.085,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.672,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2498.784,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.976,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.28,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.92,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.817,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.072,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.767,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.946,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.257,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.292,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2502.734,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.093,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.904,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.615,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.676,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2503.732,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.387,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.376,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.485,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.054,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.539,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.0,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2507.286,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.724,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2508.643,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2501.17,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.69,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.158,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.016,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.234,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.759,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.361,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.269,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2500.461,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2505.742,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2504.565,
+            "min": 0.0,
+            "max": 0.0
+        },
+        {
+            "current": 2506.111,
+            "min": 0.0,
+            "max": 0.0
+        }
+    ],
+    "disk": {
+        "total": 878.6640281677246,
+        "used": 283.64724349975586
+    },
+    "gpu": "Tesla T4",
+    "gpu_count": 2,
+    "gpu_devices": [
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        },
+        {
+            "name": "Tesla T4",
+            "memory_total": 15843721216
+        }
+    ],
+    "memory": {
+        "total": 376.5794219970703
+    }
+}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json

@@ -0,0 +1 @@
+{}

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log

@@ -0,0 +1,133 @@
+2023-03-09 23:55:18,660 INFO    StreamThr :6152 [internal.py:wandb_internal():90] W&B internal server running at pid: 6152, started at: 2023-03-09 23:55:18.657868
+2023-03-09 23:55:18,661 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status
+2023-03-09 23:55:18,663 INFO    WriterThread:6152 [datastore.py:open_for_write():85] open: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/run-wqdqlwjv.wandb
+2023-03-09 23:55:18,671 DEBUG   SenderThread:6152 [sender.py:send():336] send: header
+2023-03-09 23:55:18,671 DEBUG   SenderThread:6152 [sender.py:send():336] send: run
+2023-03-09 23:55:18,690 INFO    SenderThread:6152 [sender.py:_maybe_setup_resume():723] checking resume status for None/U-Net/wqdqlwjv
+2023-03-09 23:55:19,987 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: check_version
+2023-03-09 23:55:19,991 INFO    SenderThread:6152 [dir_watcher.py:__init__():219] watching files in: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files
+2023-03-09 23:55:19,991 INFO    SenderThread:6152 [sender.py:_start_run_threads():1081] run started: wqdqlwjv with start time 1678377318.650455
+2023-03-09 23:55:19,991 DEBUG   SenderThread:6152 [sender.py:send_request():363] send_request: summary_record
+2023-03-09 23:55:20,015 INFO    SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
+2023-03-09 23:55:20,016 DEBUG   SenderThread:6152 [sender.py:send_request():363] send_request: check_version
+2023-03-09 23:55:20,998 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-summary.json
+2023-03-09 23:55:24,987 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:25,017 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:25,076 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: run_start
+2023-03-09 23:55:25,081 DEBUG   HandlerThread:6152 [system_info.py:__init__():31] System info init
+2023-03-09 23:55:25,081 DEBUG   HandlerThread:6152 [system_info.py:__init__():46] System info init done
+2023-03-09 23:55:25,081 INFO    HandlerThread:6152 [system_monitor.py:start():183] Starting system monitor
+2023-03-09 23:55:25,082 INFO    SystemMonitor:6152 [system_monitor.py:_start():147] Starting system asset monitoring threads
+2023-03-09 23:55:25,082 INFO    HandlerThread:6152 [system_monitor.py:probe():204] Collecting system info
+2023-03-09 23:55:25,082 INFO    SystemMonitor:6152 [interfaces.py:start():187] Started cpu monitoring
+2023-03-09 23:55:25,083 INFO    SystemMonitor:6152 [interfaces.py:start():187] Started disk monitoring
+2023-03-09 23:55:25,083 INFO    SystemMonitor:6152 [interfaces.py:start():187] Started gpu monitoring
+2023-03-09 23:55:25,084 INFO    SystemMonitor:6152 [interfaces.py:start():187] Started memory monitoring
+2023-03-09 23:55:25,085 INFO    SystemMonitor:6152 [interfaces.py:start():187] Started network monitoring
+2023-03-09 23:55:25,128 DEBUG   HandlerThread:6152 [system_info.py:probe():195] Probing system
+2023-03-09 23:55:25,168 DEBUG   HandlerThread:6152 [git.py:repo():40] git repository is invalid
+2023-03-09 23:55:25,169 DEBUG   HandlerThread:6152 [system_info.py:probe():240] Probing system done
+2023-03-09 23:55:25,169 DEBUG   HandlerThread:6152 [system_monitor.py:probe():213] {'os': 'Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid', 'python': '3.7.11', 'heartbeatAt': '2023-03-09T15:55:25.128694', 'startedAt': '2023-03-09T15:55:18.607606', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'train.py', 'codePath': 'train.py', 'host': 'n1d6fa00775e48a18f80acc1355070be-task0-0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 40, 'cpu_count_logical': 80, 'cpu_freq': {'current': 2502.0496374999993, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2500.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.475, 'min': 0.0, 'max': 0.0}, {'current': 2500.398, 'min': 0.0, 'max': 0.0}, {'current': 2503.156, 'min': 0.0, 'max': 0.0}, {'current': 2500.385, 'min': 0.0, 'max': 0.0}, {'current': 2501.678, 'min': 0.0, 'max': 0.0}, {'current': 2499.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.163, 'min': 0.0, 'max': 0.0}, {'current': 2500.395, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2506.117, 'min': 0.0, 'max': 0.0}, {'current': 2499.927, 'min': 0.0, 'max': 0.0}, {'current': 2497.953, 'min': 0.0, 'max': 0.0}, {'current': 2499.177, 'min': 0.0, 'max': 0.0}, {'current': 2501.15, 'min': 0.0, 'max': 0.0}, {'current': 2503.575, 'min': 0.0, 'max': 0.0}, {'current': 2501.649, 'min': 0.0, 'max': 0.0}, {'current': 2500.6, 'min': 0.0, 'max': 0.0}, {'current': 2501.068, 'min': 0.0, 'max': 0.0}, {'current': 2501.553, 'min': 0.0, 'max': 0.0}, {'current': 2500.434, 'min': 0.0, 'max': 0.0}, {'current': 2500.396, 'min': 0.0, 'max': 0.0}, {'current': 2499.163, 'min': 0.0, 'max': 0.0}, {'current': 2501.428, 'min': 0.0, 'max': 0.0}, {'current': 2501.283, 'min': 0.0, 'max': 0.0}, {'current': 2501.769, 'min': 0.0, 'max': 0.0}, {'current': 2499.975, 'min': 0.0, 'max': 0.0}, {'current': 2499.573, 'min': 0.0, 'max': 0.0}, {'current': 2500.008, 'min': 0.0, 'max': 0.0}, {'current': 2500.441, 'min': 0.0, 'max': 0.0}, {'current': 2499.995, 'min': 0.0, 'max': 0.0}, {'current': 2500.077, 'min': 0.0, 'max': 0.0}, {'current': 2499.185, 'min': 0.0, 'max': 0.0}, {'current': 2500.401, 'min': 0.0, 'max': 0.0}, {'current': 2500.465, 'min': 0.0, 'max': 0.0}, {'current': 2499.575, 'min': 0.0, 'max': 0.0}, {'current': 2499.949, 'min': 0.0, 'max': 0.0}, {'current': 2501.373, 'min': 0.0, 'max': 0.0}, {'current': 2501.085, 'min': 0.0, 'max': 0.0}, {'current': 2500.672, 'min': 0.0, 'max': 0.0}, {'current': 2498.784, 'min': 0.0, 'max': 0.0}, {'current': 2504.976, 'min': 0.0, 'max': 0.0}, {'current': 2505.28, 'min': 0.0, 'max': 0.0}, {'current': 2503.92, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2503.817, 'min': 0.0, 'max': 0.0}, {'current': 2507.072, 'min': 0.0, 'max': 0.0}, {'current': 2504.767, 'min': 0.0, 'max': 0.0}, {'current': 2504.946, 'min': 0.0, 'max': 0.0}, {'current': 2503.257, 'min': 0.0, 'max': 0.0}, {'current': 2500.292, 'min': 0.0, 'max': 0.0}, {'current': 2502.734, 'min': 0.0, 'max': 0.0}, {'current': 2507.093, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2505.904, 'min': 0.0, 'max': 0.0}, {'current': 2505.615, 'min': 0.0, 'max': 0.0}, {'current': 2503.676, 'min': 0.0, 'max': 0.0}, {'current': 2503.732, 'min': 0.0, 'max': 0.0}, {'current': 2505.387, 'min': 0.0, 'max': 0.0}, {'current': 2500.376, 'min': 0.0, 'max': 0.0}, {'current': 2501.485, 'min': 0.0, 'max': 0.0}, {'current': 2506.054, 'min': 0.0, 'max': 0.0}, {'current': 2506.539, 'min': 0.0, 'max': 0.0}, {'current': 2500.0, 'min': 0.0, 'max': 0.0}, {'current': 2507.286, 'min': 0.0, 'max': 0.0}, {'current': 2500.724, 'min': 0.0, 'max': 0.0}, {'current': 2508.643, 'min': 0.0, 'max': 0.0}, {'current': 2501.17, 'min': 0.0, 'max': 0.0}, {'current': 2500.69, 'min': 0.0, 'max': 0.0}, {'current': 2505.158, 'min': 0.0, 'max': 0.0}, {'current': 2500.016, 'min': 0.0, 'max': 0.0}, {'current': 2500.234, 'min': 0.0, 'max': 0.0}, {'current': 2500.759, 'min': 0.0, 'max': 0.0}, {'current': 2504.361, 'min': 0.0, 'max': 0.0}, {'current': 2500.269, 'min': 0.0, 'max': 0.0}, {'current': 2500.461, 'min': 0.0, 'max': 0.0}, {'current': 2505.742, 'min': 0.0, 'max': 0.0}, {'current': 2504.565, 'min': 0.0, 'max': 0.0}, {'current': 2506.111, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 878.6640281677246, 'used': 283.64724349975586}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 15843721216}, {'name': 'Tesla T4', 'memory_total': 15843721216}], 'memory': {'total': 376.5794219970703}}
+2023-03-09 23:55:25,169 INFO    HandlerThread:6152 [system_monitor.py:probe():214] Finished collecting system info
+2023-03-09 23:55:25,169 INFO    HandlerThread:6152 [system_monitor.py:probe():217] Publishing system info
+2023-03-09 23:55:25,169 DEBUG   HandlerThread:6152 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
+2023-03-09 23:55:25,187 DEBUG   HandlerThread:6152 [system_info.py:_save_pip():67] Saving pip packages done
+2023-03-09 23:55:25,187 DEBUG   HandlerThread:6152 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
+2023-03-09 23:55:25,907 DEBUG   HandlerThread:6152 [system_info.py:_save_conda():86] Saving conda packages done
+2023-03-09 23:55:25,960 INFO    HandlerThread:6152 [system_monitor.py:probe():219] Finished publishing system info
+2023-03-09 23:55:25,963 DEBUG   SenderThread:6152 [sender.py:send():336] send: files
+2023-03-09 23:55:25,964 INFO    SenderThread:6152 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
+2023-03-09 23:55:25,999 DEBUG   SenderThread:6152 [sender.py:send():336] send: telemetry
+2023-03-09 23:55:25,999 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: stop_status
+2023-03-09 23:55:26,008 DEBUG   SenderThread:6152 [sender.py:send():336] send: config
+2023-03-09 23:55:26,055 DEBUG   SenderThread:6152 [sender.py:send_request():363] send_request: stop_status
+2023-03-09 23:55:26,143 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/requirements.txt
+2023-03-09 23:55:26,165 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/wandb-metadata.json
+2023-03-09 23:55:26,171 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/conda-environment.yaml
+2023-03-09 23:55:27,136 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_created():278] file/dir created: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log
+2023-03-09 23:55:27,210 INFO    wandb-upload_0:6152 [upload_job.py:push():138] Uploaded file /tmp/tmpcpr9x05xwandb/qhhlrewz-wandb-metadata.json
+2023-03-09 23:55:29,138 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/output.log
+2023-03-09 23:55:30,921 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:35,922 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:40,923 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:40,972 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:40,973 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:45,923 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:45,974 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:50,929 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:55:50,991 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:52,159 INFO    Thread-13 :6152 [dir_watcher.py:_on_file_modified():295] file/dir modified: /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/files/config.yaml
+2023-03-09 23:55:55,977 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:55:56,728 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:00,979 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:01,728 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:05,980 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:06,729 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:10,982 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:11,730 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:15,983 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:16,731 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:20,984 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:21,731 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:25,085 DEBUG   SystemMonitor:6152 [system_monitor.py:_start():161] Starting system metrics aggregation loop
+2023-03-09 23:56:25,090 DEBUG   SenderThread:6152 [sender.py:send():336] send: stats
+2023-03-09 23:56:25,986 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:27,092 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:30,987 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:32,092 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:35,988 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:37,093 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:40,990 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:42,094 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:45,991 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:47,094 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:50,992 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:52,095 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:56:55,093 DEBUG   SenderThread:6152 [sender.py:send():336] send: stats
+2023-03-09 23:56:55,994 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:56:58,094 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:00,995 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:03,095 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:05,996 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:08,096 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:10,997 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:13,096 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:15,998 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:18,097 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:21,000 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:23,098 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:25,095 DEBUG   SenderThread:6152 [sender.py:send():336] send: stats
+2023-03-09 23:57:26,001 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:29,097 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:31,003 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:34,098 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:36,004 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:39,099 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:41,005 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:44,099 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:46,007 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:49,100 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:51,008 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:57:54,101 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:57:55,098 DEBUG   SenderThread:6152 [sender.py:send():336] send: stats
+2023-03-09 23:57:56,009 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:00,100 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:01,010 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:05,101 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:06,012 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:10,101 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:11,013 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:15,102 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:16,014 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:20,103 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:21,015 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:25,101 DEBUG   SenderThread:6152 [sender.py:send():336] send: stats
+2023-03-09 23:58:26,018 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:26,103 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:31,018 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:31,104 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:36,019 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:36,105 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:41,021 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: keepalive
+2023-03-09 23:58:41,108 DEBUG   HandlerThread:6152 [handler.py:handle_request():144] handle_request: status_report
+2023-03-09 23:58:45,110 INFO    MainThread:6152 [internal.py:handle_exit():77] Internal process exited

official/cv/unet2d/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log

@@ -0,0 +1,27 @@
+2023-03-09 23:55:18,636 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Configure stats pid to 4410
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from /code/UNet-master/2dunet-adapter/wandb/settings
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': 'train.py', 'program': 'train.py'}
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_setup.py:_flush():76] Applying login settings: {'anonymous': 'must'}
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:_log_setup():506] Logging user logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug.log
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:_log_setup():507] Logging internal logs to /code/UNet-master/2dunet-adapter/wandb/run-20230309_235518-wqdqlwjv/logs/debug-internal.log
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:init():546] calling init triggers
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:init():553] wandb.init called with sweep_config: {}
+config: {}
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:init():602] starting backend
+2023-03-09 23:55:18,637 INFO    MainThread:4410 [wandb_init.py:init():606] setting up manager
+2023-03-09 23:55:18,648 INFO    MainThread:4410 [backend.py:_multiprocessing_setup():108] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
+2023-03-09 23:55:18,650 INFO    MainThread:4410 [wandb_init.py:init():613] backend started and connected
+2023-03-09 23:55:18,654 INFO    MainThread:4410 [wandb_init.py:init():701] updated telemetry
+2023-03-09 23:55:18,655 INFO    MainThread:4410 [wandb_init.py:init():741] communicating run to backend with 60.0 second timeout
+2023-03-09 23:55:19,986 INFO    MainThread:4410 [wandb_run.py:_on_init():2131] communicating current version
+2023-03-09 23:55:25,018 INFO    MainThread:4410 [wandb_run.py:_on_init():2140] got version response 
+2023-03-09 23:55:25,018 INFO    MainThread:4410 [wandb_init.py:init():789] starting run threads in backend
+2023-03-09 23:55:25,970 INFO    MainThread:4410 [wandb_run.py:_console_start():2112] atexit reg
+2023-03-09 23:55:25,971 INFO    MainThread:4410 [wandb_run.py:_redirect():1967] redirect: SettingsConsole.WRAP_RAW
+2023-03-09 23:55:25,971 INFO    MainThread:4410 [wandb_run.py:_redirect():2032] Wrapping output streams.
+2023-03-09 23:55:25,971 INFO    MainThread:4410 [wandb_run.py:_redirect():2057] Redirects installed.
+2023-03-09 23:55:25,972 INFO    MainThread:4410 [wandb_init.py:init():831] run started, returning control to user process
+2023-03-09 23:55:25,972 INFO    MainThread:4410 [wandb_run.py:_config_callback():1249] config_cb None None {'epochs': 5, 'batch_size': 1, 'learning_rate': 1e-05, 'save_checkpoint': True, 'img_scale': 0.5}

official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/config.yaml

@@ -0,0 +1,39 @@
+wandb_version: 1
+
+_wandb:
+  desc: null
+  value:
+    cli_version: 0.13.11
+    framework: torch
+    is_jupyter_run: false
+    is_kaggle_kernel: false
+    python_version: 3.7.11
+    start_time: 1678377605.403295
+    t:
+      1:
+      - 1
+      - 55
+      2:
+      - 1
+      - 55
+      3:
+      - 23
+      4: 3.7.11
+      5: 0.13.11
+      8:
+      - 5
+batch_size:
+  desc: null
+  value: 1
+epochs:
+  desc: null
+  value: 5
+img_scale:
+  desc: null
+  value: 0.5
+learning_rate:
+  desc: null
+  value: 1.0e-05
+save_checkpoint:
+  desc: null
+  value: true

official/cv/unet2d/2dunet-adapter/wandb/run-20230310_000005-yo5e3390/files/output.log

@@ -0,0 +1,8 @@
+INFO: Starting training:
+        Epochs:          5
+        Batch size:      1
+        Learning rate:   1e-05
+        Training size:   5088
+        Checkpoints:     True
+        Images scaling:  0.5
+Epoch 1/5:   0%|                                                                                                                                                               | 0/5088 [00:00<?, ?img/s]