Publicar

06 - BACKUP AUTOMATIZADO E OTIMIZADO COM RCLONE E PYTHON

Doi: 10.47094/978-65-284-0683-8/80-87

PALAVRAS CHAVE: Inteligência artificial - Aplicações médicas. Redes neurais (Ciência da computação) - Aplicações médicas. Sistemas de detecção de objetos (YOLO).

KEYWORDS: Artificial intelligence - Medical applications. Neural networks (Computer science) - Medical applications. Object detection systems (YOLO).

ABSTRACT:
1. THE RISK LANDSCAPE AND THE NEED FOR BACKUP
Training deep neural networks, such as models from the YOLO family, requires significant processing time and generates numerous files during execution: weight checkpoints (`best.pt` and `last.pt`), confusion matrices, precision-recall curves, validation images, and tabular logs (`results.csv`). Power outages or host machine failures during this process can corrupt local files, resulting in the loss of hours or even days of processing work.
Manual uploads or standard synchronization methods for cloud services (such as Google Drive) face two practical obstacles:
1. Volume of small files: Prediction and log folders contain thousands of files, each only a few kilobytes in size. Cloud services impose limits on transactions per second (TPS) and may reject connections that initiate too many simultaneous requests.
2. Checkpoint file size: `best.pt` and `last.pt` files can range from tens to hundreds of megabytes. If a connection is interrupted, the system must be able to resume the transfer automatically without corrupting the file.

To overcome these limitations, Rclone serves as a suitable solution for automating the upload process.

Autores

  • Adriano Carvalho Costa

  • Adriano Soares de Oliveira Bailão

  • Andrea Barboza Proto Sardi

  • Gabriela Almeida Marques

  • Gidelia Araújo Ferreira de Melo

  • Gustavo Henrique Joazeiro de Almeida

  • Juliana Fortunato

  • Leandro Manuel Pereira Sanches

  • Lorena Santana Arantes

  • Matheus Barp Pierozan

  • Ronaldo Camilo dos Santos

  • Sergio Souza Novak

Download