June 23, 2022

Migrate High Volume Data in Google Drive

    When A share a folder to B, the file belong to whoever upload the file and not easy to transfer ownership when A and B are in different organizations it also applies to any org account versus personal Google Drive.

    For moving big files over 20GB to B, B can right click on the file and make a copy, copy to somewhere else, new file will belong to B. For small files, we can use google Colab with rsync command to sync up folder recursively. Each account has daily 750 GB data transfer amount. It generates files with 0 Byte size when transfer quota used up without stopping the rsync. It confuse me since the size of the folders are not identical but rsync stop copying file since file existed.

Useful rsync commends:
recursive rsync, showing progress and not overwrite existed files

!rsync --ignore-existing -ra --progress '/content/drive/MyDrive//FolderA/'
'/content/drive/MyDrive//FolderB'

Find and delete empty files

!find "/content/drive/MyDrive/Backup_Local/SYR/" -size 0  | xargs rm

Find the size for the folder

!du -sh '/content/drive/MyDrive/Backup/' | sort -n -r 

Useful link

https://ourtechroom.com/tech/copy-shared-google-drive-files-folder-to-my-drive/ 

No comments:

Post a Comment