Skip to content
Connect

Getting data from an Azure blob container

Connecting to an Azure blob container

Section titled “Connecting to an Azure blob container”

To get data from an Azure blob container, you need to execute connect get az.

Depending on the type of authentication you need to either use account name, account key and blob container name:

  • --az-container then the blob container name
  • --az-account-name then the AZ account name
  • --az-account-key then the AZ account key

Or you can use connection string:

  • --az-connection-string then the connection string

Or use Shared Access Signature (SAS) URL:

  • --az-sas-url then the SAS URL

Getting single file from an Azure blob container

Section titled “Getting single file from an Azure blob container”

To get a single file from an Azure blob container you should use

  • --file followed by the file name to be downloaded

Following command will get file1.zip file from an Azure blob container my-az-blob using account name and account key for authentication.

Terminal window
$ connect get az --file file1.zip --az-container my-az-blob --az-account-name myaccount --az-account-key 1O8MbYTgY8Z/KD7Ip181dar7zvWfpTyzGIVdX1WsCmBSRj+XL1oXTkcMWri2ejhfmcA7/ZAFO6YL+ASt/GRBow==
2025/06/05 13:02:44 INFO Getting files from az://my-az-blob/
2025/06/05 13:02:44 INFO Files to get: count=1
[file1.zip] 10.00 MiB / 10.00 MiB done

Getting multiple files from an Azure blob container

Section titled “Getting multiple files from an Azure blob container”

If you’d like to get multiple files from an Azure blob container you need to specify either file mask or file regular expression for the files.

  • --file-mask a file mask
  • --file-regex regular expression

This will download all files that match a certain mask or pattern. You can also provide an optional source folder that contains the files to be downloaded:

  • -s, --src-folder folder containing the files to download

If you don’t provide the source folder files will be downloaded from “root folder” of the Azure blob container.

Following command will get all .zip files from data folder to an Azure blob container my-az-blob using Shared Access Signature (SAS) URL to authenticate.

Terminal window
$ connect get az --file-mask "*.zip" -s data --az-sas-url "https://myaccount.blob.core.windows.net/my-az-blob?sp=racwdli&st=2024-09-11T14:53:45Z&se=2024-09-25T22:53:45Z&spr=https&sv=2022-11-02&sr=c&sig=WZl3RWGl6pPHKUVyGlWB0ZxYjcmIK%2FbCkdjglw2wku0%3D"
2025/06/05 14:17:17 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:17:17 INFO Files to get: count=3
[file1.zip] 10.00 MiB / 10.00 MiB done
[file2.zip] 10.00 MiB / 10.00 MiB done
[file3.zip] 10.00 MiB / 10.00 MiB done

You can also provide an optional destination folder where the files will be downloaded to:

  • -d, --dst-folder target folder where to download files

If you don’t provide the destination folder files will be downloaded to the current folder.

In below example all .zip files from data folder of Azure blob container my-az-blob will be downloaded to local folder using Shared Access Signature (SAS) URL for authentication.

Terminal window
$ connect get az --file-mask "*.zip" -s data -d local --az-sas-url "https://myaccount.blob.core.windows.net/my-az-blob?sp=racwdli&st=2024-09-11T14:53:45Z&se=2024-09-25T22:53:45Z&spr=https&sv=2022-11-02&sr=c&sig=WZl3RWGl6pPHKUVyGlWB0ZxYjcmIK%2FbCkdjglw2wku0%3D"
2025/06/05 14:18:15 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:18:15 INFO Files to get: count=3
[file1.zip] 10.00 MiB / 10.00 MiB done
[file2.zip] 10.00 MiB / 10.00 MiB done
[file3.zip] 10.00 MiB / 10.00 MiB done

Getting files from an Azure blob container in batch mode

Section titled “Getting files from an Azure blob container in batch mode”

When using the application in batch or non-interactive mode, such as in a script or scheduler, you should disable the progress bar by using

  • --batch to get files in batch mode

Following command will get all .zip files from the data folder of Azure blob container my-az-blob and place them in the local folder, using Shared Access Signature (SAS) URL to authenticate.

Terminal window
$ connect get az --file-mask "*.zip" -s data -d local --batch --az-sas-url "https://myaccount.blob.core.windows.net/my-az-blob?sp=racwdli&st=2024-09-11T14:53:45Z&se=2024-09-25T22:53:45Z&spr=https&sv=2022-11-02&sr=c&sig=WZl3RWGl6pPHKUVyGlWB0ZxYjcmIK%2FbCkdjglw2wku0%3D"
2025/06/05 14:19:00 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:19:00 INFO Files to get: count=3
2025/06/05 14:19:00 INFO Transferring file=data/file1.zip targetFolder=local
2025/06/05 14:19:00 INFO File transferred successfully bytesTransferred=10485760 duration=96ms transferRate=103.19MB/s file=data/file1.zip targetFolder=local
2025/06/05 14:19:00 INFO Transferring file=data/file2.zip targetFolder=local
2025/06/05 14:19:00 INFO File transferred successfully bytesTransferred=10485760 duration=146ms transferRate=68.07MB/s file=data/file2.zip targetFolder=local
2025/06/05 14:19:00 INFO Transferring file=data/file3.zip targetFolder=local
2025/06/05 14:19:01 INFO File transferred successfully bytesTransferred=10485760 duration=120ms transferRate=83.00MB/s file=data/file3.zip targetFolder=local

Getting files from an Azure blob container using scheduler

Section titled “Getting files from an Azure blob container using scheduler”

When running the application from a scheduler, it’s recommended to ensure only one instance operates on a specific folder. It’s possible to configure the application to create a flag file at startup and delete it upon completion. If another instance tries to start while one is already running, it will detect the flag file and exit. This approach ensures that only one instance of the application runs at any given time. To do that use

  • --flag path to the flag file

Following command will get all .zip files from the data folder of Azure blob container my-az-blob and place them in the local folder using /tmp/flag1 as a flag file and account name and account key for authentication. Mind that we are using batch mode as it’s required for the application to run correctly from script or scheduler.

Terminal window
$ connect get az --flag /tmp/flag1 --batch --file-mask "*.zip" -s data -d local --az-container my-az-blob --az-account-name myaccount --az-account-key 1O8MbYTgY8Z/KD7Ip181dar7zvWfpTyzGIVdX1WsCmBSRj+XL1oXTkcMWri2ejhfmcA7/ZAFO6YL+ASt/GRBow==
2025/06/05 14:20:39 INFO Using flag file: /tmp/flag1
2025/06/05 14:20:39 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:20:39 INFO Files to get: count=3
2025/06/05 14:20:39 INFO Transferring file=data/file1.zip targetFolder=local
2025/06/05 14:20:39 INFO File transferred successfully bytesTransferred=10485760 duration=171ms transferRate=58.19MB/s file=data/file1.zip targetFolder=local
2025/06/05 14:20:39 INFO Transferring file=data/file2.zip targetFolder=local
2025/06/05 14:20:40 INFO File transferred successfully bytesTransferred=10485760 duration=210ms transferRate=47.53MB/s file=data/file2.zip targetFolder=local
2025/06/05 14:20:40 INFO Transferring file=data/file3.zip targetFolder=local
2025/06/05 14:20:40 INFO File transferred successfully bytesTransferred=10485760 duration=132ms transferRate=75.67MB/s file=data/file3.zip targetFolder=local
2025/06/05 14:20:40 INFO Removing flag file: /tmp/flag1

Getting files from an Azure blob container in parallel

Section titled “Getting files from an Azure blob container in parallel”

To speed up file download you can fetch files simultaneously. Application will then get files using multiple sessions. To specify number of concurrent GCS sessions use:

  • --parallel number of Azure sessions

Following command will get all .zip files from the data folder of Azure blob container my-az-blob using account name and account key for authentication and 3 parallel sessions.

Terminal window
$ connect get az --file-mask "*.zip" -s data --batch --parallel 3 --az-container my-az-blob --az-account-name myaccount --az-account-key 1O8MbYTgY8Z/KD7Ip181dar7zvWfpTyzGIVdX1WsCmBSRj+XL1oXTkcMWri2ejhfmcA7/ZAFO6YL+ASt/GRBow==
2025/06/05 14:21:38 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:21:38 INFO Files to get: count=3
2025/06/05 14:21:38 INFO Transferring file=data/file3.zip targetFolder=
2025/06/05 14:21:38 INFO Transferring file=data/file2.zip targetFolder=
2025/06/05 14:21:38 INFO Transferring file=data/file1.zip targetFolder=
2025/06/05 14:21:38 INFO File transferred successfully bytesTransferred=10485760 duration=103ms transferRate=96.27MB/s file=data/file3.zip targetFolder=
2025/06/05 14:21:38 INFO File transferred successfully bytesTransferred=10485760 duration=109ms transferRate=91.31MB/s file=data/file2.zip targetFolder=
2025/06/05 14:21:38 INFO File transferred successfully bytesTransferred=10485760 duration=117ms transferRate=85.00MB/s file=data/file1.zip targetFolder=

Getting files from an Azure blob container in sequence

Section titled “Getting files from an Azure blob container in sequence”

If you don’t specify the --parallel option then files will be downloaded using single Azure session in the same order as they showed up on Azure blob container (oldest files first).

Action after file is downloaded from an Azure blob container

Section titled “Action after file is downloaded from an Azure blob container”

After file is successfully downloaded to local system it’s possible to remove it from source or move it to a different folder at the source system.

  • --delete delete a file after it’s downloaded
  • --move-folder target folder to move the file after it’s downloaded

Following command will get all .zip files from data folder of Azure blob container my-s3-bucket using account name and account key for authentication, 3 parallel sessions and store them in local folder. Then files will be moved to archive folder at in the Azure blob container.

Terminal window
$ connect get az --file-mask "*.zip" -s data -d local --batch --parallel 3 --move-folder archive --az-container my-az-blob --az-account-name myaccount --az-account-key 1O8MbYTgY8Z/KD7Ip181dar7zvWfpTyzGIVdX1WsCmBSRj+XL1oXTkcMWri2ejhfmcA7/ZAFO6YL+ASt/GRBow==
2025/06/05 14:22:51 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:22:51 INFO Files to get: count=3
2025/06/05 14:22:51 INFO Transferring file=data/file2.zip targetFolder=local
2025/06/05 14:22:51 INFO Transferring file=data/file1.zip targetFolder=local
2025/06/05 14:22:51 INFO Transferring file=data/file3.zip targetFolder=local
2025/06/05 14:22:52 INFO File moved to folder folder=archive file=data/file3.zip
2025/06/05 14:22:52 INFO File transferred successfully bytesTransferred=10485760 duration=103ms transferRate=96.86MB/s file=data/file3.zip targetFolder=local
2025/06/05 14:22:52 INFO File moved to folder folder=archive file=data/file1.zip
2025/06/05 14:22:52 INFO File moved to folder folder=archive file=data/file2.zip
2025/06/05 14:22:52 INFO File transferred successfully bytesTransferred=10485760 duration=167ms transferRate=59.68MB/s file=data/file1.zip targetFolder=local
2025/06/05 14:22:52 INFO File transferred successfully bytesTransferred=10485760 duration=183ms transferRate=54.40MB/s file=data/file2.zip targetFolder=local

Following command will get all .zip files from data folder of Azure blob container my-az-blob using account name and account key for authentication, 3 parallel sessions and then removes the files from the Azure blob container.

Terminal window
$ connect get az --file-mask "*.zip" -s data --batch --parallel 3 --delete --az-container my-az-blob --az-account-name myaccount --az-account-key 1O8MbYTgY8Z/KD7Ip181dar7zvWfpTyzGIVdX1WsCmBSRj+XL1oXTkcMWri2ejhfmcA7/ZAFO6YL+ASt/GRBow==
2025/06/05 14:24:48 INFO Getting files from az://my-az-blob/data/
2025/06/05 14:24:48 INFO Files to get: count=3
2025/06/05 14:24:48 INFO Transferring file=data/file2.zip targetFolder=
2025/06/05 14:24:48 INFO Transferring file=data/file1.zip targetFolder=
2025/06/05 14:24:48 INFO Transferring file=data/file3.zip targetFolder=
2025/06/05 14:24:48 INFO File deleted file=data/file1.zip
2025/06/05 14:24:48 INFO File transferred successfully bytesTransferred=10485760 duration=95ms transferRate=105.15MB/s file=data/file1.zip targetFolder=
2025/06/05 14:24:48 INFO File deleted file=data/file2.zip
2025/06/05 14:24:48 INFO File transferred successfully bytesTransferred=10485760 duration=154ms transferRate=64.55MB/s file=data/file2.zip targetFolder=
2025/06/05 14:24:49 INFO File deleted file=data/file3.zip
2025/06/05 14:24:49 INFO File transferred successfully bytesTransferred=10485760 duration=193ms transferRate=51.66MB/s file=data/file3.zip targetFolder=

All Azure Get Options

Section titled “All Azure Get Options”
Terminal window
Usage:
  connect get az [command] [flags]


Flags:
      --az-account-key string         AZ account key
      --az-account-name string        AZ account name
      --az-connection-string string   AZ connection string
      --az-container string           AZ container name
      --az-sas-url string             AZ SAS URL


Global Flags:
      --batch                     No progress bars
      --delete                    Try to delete files after successful get
  -d, --dst-folder string         Destination folder for retrieved files
      --file string               File name to be get
      --file-mask string          File mask to filter files
      --file-regex string         File regex to filter files
      --flag string               Flag file name
      --from-mail string          From mail used to send notifications (only in batch mode)
      --godebug                   Turns on debug mode
      --help                      Prints help for the command
      --log-format string         Log output format: text|json (default "text")
      --mail-format string        Mail format [text|html] (only in batch mode) (default "plain")
      --move-folder string        Folder to move files after successful get
      --no-color                  Do not use colors in logs
      --parallel uint             Number of sessions used to get files (default 1)
      --quiet                     Makes no output
  -s, --src-folder string         Folder to look for files
      --to-mail-failure strings   Email list to send failure notification (only in batch mode)
      --to-mail-success strings   Email list to send success notification (only in batch mode)

Errors

Section titled “Errors”

Please check here for information on the possible errors.