Merge pull request #677 from Azure/dev

10.3.0 Release

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [[email protected]](mailto:[email protected]) with questions or concerns

ChangeLog.md

@@ -1,6 +1,129 @@
 
 # Change Log
 
+## Version 10.3.0
+
+### Breaking changes
+
+1. The `*` character is no longer supported as a wildcard in URLs, except for the two exceptions
+   noted below. It remains supported in local file paths.
+   1. The first execption is that `/*` is still allowed at the very end of the "path" section of a
+      URL. This is illustrated by the difference between these two source URLs:
+      `https://account/container/virtual?SAS` and 
+      `https://account/container/virtualDir/*?SAS`.  The former copies the virtual directory
+      `virtualDir` by creating a folder of that name at the destination.  The latter copies the
+      _contents_ of `virtual` dir directly into the target without creating a folder named
+      "virtualDir".'
+   1. The second exception is when you are transferring multiple _whole_ containers (or S3 buckets). You can
+      use * as a wildcard in the container or bucket name. 
+1. The `--include` and `--exclude` parameters have been replaced by `--include-pattern` and
+   `--exclude-pattern` (for filenames) and `--include-path` and `--exclude-path` (for paths,
+   including directory and filenames).
+   The new parameters have behaviour that is better defined in complex situations (such as
+   recursion).  The `*` wildcard is supported in the pattern parameters, but _not_ in the path ones.
+1. There have been two breaking changes to the JSON output that is produced if you request
+   JSON-formatted output. The `sync` command's output in JSON has changed for consistency reasons,
+   and the final message type, for `copy` and `sync` has changed its name from `Exit` to `EndOfJob`.
+   Tools using the JSON output format to integrate AzCopy should be aware.
+1. If downloading to "null" on Windows the target must now be named "NUL", according to standard
+   Windows conventions.  "/dev/null" remains correct on Linux. (This feature can be used to test
+   throughput or check MD5s without saving the downloaded data.) 
+1. The file format of the (still undocumented) `--list-of-files` parameter is changed.  (It remains
+   undocmented because, for simplicity, users are
+   encouraged to use the new `--include-pattern` and `--include-path` parameters instead.)
+
+### New features
+
+1. `sync` is supported from Blob Storage to Blob Storage, and from Azure Files to Azure Files.
+1. `copy` is supported from Azure Files to Azure Files, and from Blob Storage to Azure Files.
+1. Percent complete is displayed as each job runs.
+1. VHD files are auto-detected as page blobs.
+1. A new benchmark mode allows quick and easy performance benchmarking of your network connection to
+   Blob Storage. Run AzCopy with the paramaters `bench --help` for details.  This feature is in
+   Preview status.
+1. The location for AzCopy's "plan" files can be specified with the environment variable
+   `AZCOPY_JOB_PLAN_LOCATION`. (If you move the plan files and also move the log files using the existing
+   `AZCOPY_LOG_LOCATION`, then AzCopy will not store anything under your home directory on Linux and
+   MacOS.  On Windows AzCopy will keep just one small encrypted file under `c:\users\<username>\.azcopy`)
+1. Log files and plan files can be cleaned up to save disk space, using AzCopy's new `jobs rm` and
+   `jobs clean` commands.
+1. When listing jobs with `jobs show`, the status of each job is included in the output.   
+1. The `--overwrite` parameter now supports the value of "prompt" to prompt the user on a
+   file-by-file basis. (The old values of true and false are also supported.)   
+1. The environment variable `AZCOPY_CONCURRENCY_VALUE` can now be set to "AUTO". This is expected to be
+    useful for customers with small networks, or those running AzCopy on
+   moderately-powered machines and transfer blobs between accounts.  This feature is in preview status.
+1. When uploading from Windows, files can be filtered by Windows-specific file attributes (such as
+   "Archive", "Hidden" etc)
+1. Memory usage can be controlled by setting the new environment variable `AZCOPY_BUFFER_GB`.
+   Decimal values are supported. Actual usage will be the value specified, plus some overhead. 
+1. An extra integrity check has been added: the length of the
+   completed desination file is checked against that of the source.
+1. When downloading, AzCopy can automatically decompress blobs (or Azure Files) that have a
+   `Content-Encoding` of `gzip` or `deflate`. To enable this behaviour, supply the `--decompress`
+   parameter.
+1. The number of disk files accessed concurrently can be controlled with the new
+   `AZCOPY_CONCURRENT_FILES` environment variable. This is an advanced setting, which generally
+   should not be modified.  It does not affect the number of HTTP connections, which is still
+   controlled by `AZCOPY_CONCURRENCY_VALUE`.
+1. The values of key environment variables are listed at the start of the log file.
+1. An official Windows 32-bit build is now released, in addition to the usual 64-bit builds for
+   Linux, Mac and Windows.
+1. If you need to refer a literal `*` in the name of a blob or Azure Files file, e.g. for a blob
+   named "\*", escape the `*` using standard URL escaping. To do this, replace the `*` with the following
+   character sequence: %2A 
+
+### Bug fixes
+
+1. When an AzCopy job is cancelled with CTRL-C, any partially-updated files are now deleted from
+   the destination. Previous releases of AzCopy v10 would just immediately exit, leaving destination files
+   potentially containing an unknown mix of old and new data. E.g. if uploading a new version of a file
+   over top of an old version, cancellation could result in the file being left with some parts
+   containing old data, and some containing new data. This issue affected downloads to local disk and
+   uploads to Azure Files, ADLS Gen 2, page blobs and append blobs. The bug did not affect transfers to block
+   blobs.
+1. If a transfer to a brand-new block blob is cancelled before it completes, the uncommitted blocks are now cleaned up
+   immediately. Previous versions would leave them, for automatic garbage collection to delete 7 days later.
+1. Long pathnames (over 260 characters) are now supported everywhere on Windows, including on UNC
+   shares.
+1. Safety is improved in the rare cases where two source files correspond to just one destination file. This can happen
+   when transferring to a case-insensitive destination, when the new `--decompress` flag removes an extension but
+   there's already a file without the extension, and in very rare cases related to escaping of filenames with illegal
+   characters. The bug fix ensures that the single resulting file contains data from only _one_ of the source files.
+1. When supplying a `--content-type` on the command line it's no longer necessary to also specify
+   `--no-guess-mime-type`.
+1. There is now no hard-coded limit on the number of files that can be processed by the `sync`
+   command.  The number that can be processed (without paging of memory to disk) depends only on the
+   amount of RAM available.
+1. Transfer of sparse page blobs has been improved, so that for many sparse page blobs only the
+   populated pages will transferred.  The one exception is blobs which have had a very high number
+   of updates, but which still have significant sparse sections.  Those blobs may not be
+   transferred optimally in this release. Handling of such blobs will be improved in a future release.
+1. Accessing root of drive (e.g. `d:\`) no longer causes an error.
+1. On slow networks, there are no longer excessive log messages sent to the Event Log (Windows) and
+   SysLog (Linux).
+1. If AzCopy can't check whether it's up to date, it will no longer hang. (Previously, it could hang
+   if its version check URL, https://aka.ms/azcopyv10-version-metadata, was unreachable due to
+   network routing restrictions.)  
+1. High concurrency values are supported (e.g. over 1000 connections). While these values are seldom
+   needed, they are occasionally useful - e.g. for service-to-service transfer of files around 1 MB
+   in size.
+1. Files skipped due to "overwrite=false" are no longer logged as "failed".
+1. Logging is more concise at the default log level.
+1. Error message text, returned by Blob and File services, is now included in the log.
+1. A log file is created for copy jobs even when there was nothing to copy.
+1. In the log, UPLOAD SUCCESSFUL messages now include the name of the successful file.
+1. Clear error messages are given to show that AzCopy does not currently support Customer-Provided
+   Encryption Keys.
+1. On Windows, downloading a filename with characters not supported by the operating system will
+   result in those characters being URL-encoded to construct a Windows-compatible filename. The
+   encoding process is reversed if the file is uploaded.
+1. Uploading a single file to ADLS Gen 2 works now.
+1. The `remove` command no longer hangs when removing blobs that have snapshots. Instead it will fail to 
+   delete them, and report the failures clearly.
+1. Jobs downloading from ADLS Gen 2 that result in no scheduled transfers will no longer hang.
+
+
 ## Version 10.2.1
 
 ### Bug fix

azbfs/path_extensions.go

@@ -0,0 +1,31 @@
+package azbfs
+
+import (
+	"encoding/base64"
+	"time"
+)
+
+func (p Path) LastModifiedTime() time.Time {
+	if p.LastModified == nil {
+		return time.Time{}
+	}
+
+	t, err := time.Parse(time.RFC1123, *p.LastModified)
+	if err != nil {
+		return time.Time{}
+	}
+
+	return t
+}
+
+func (p Path) ContentMD5() []byte {
+	if p.ContentMD5Base64 == nil {
+		return nil
+	}
+
+	md5, err := base64.StdEncoding.DecodeString(*p.ContentMD5Base64)
+	if err != nil {
+		return nil
+	}
+	return md5
+}

azbfs/url_service.go

@@ -1,25 +1,30 @@
 package azbfs
 
 import (
+	"context"
 	"net/url"
 
 	"github.com/Azure/azure-pipeline-go/pipeline"
 )
 
 // A ServiceURL represents a URL to the Azure Storage File service allowing you to manipulate file shares.
 type ServiceURL struct {
-	client managementClient
+	client filesystemClient
 }
 
 // NewServiceURL creates a ServiceURL object using the specified URL and request policy pipeline.
 func NewServiceURL(url url.URL, p pipeline.Pipeline) ServiceURL {
 	if p == nil {
 		panic("p can't be nil")
 	}
-	client := newManagementClient(url, p)
+	client := filesystemClient{newManagementClient(url, p)}
 	return ServiceURL{client: client}
 }
 
+func (s ServiceURL) ListFilesystemsSegment(ctx context.Context, marker *string) (*FilesystemList, error) {
+	return s.client.List(ctx, nil, marker, nil, nil, nil, nil)
+}
+
 // URL returns the URL endpoint used by the ServiceURL object.
 func (s ServiceURL) URL() url.URL {
 	return s.client.URL()

azbfs/zc_pipeline.go

@@ -40,7 +40,7 @@ func NewPipeline(c Credential, o PipelineOptions) pipeline.Pipeline {
 	}
 	f = append(f,
 		pipeline.MethodFactoryMarker(), // indicates at what stage in the pipeline the method factory is invoked
-		NewRequestLogPolicyFactory(o.RequestLog))
+		NewRequestLogPolicyFactory_Deprecated(o.RequestLog))
 
 	return pipeline.NewPipeline(f, pipeline.Options{HTTPSender: nil, Log: o.Log})
 }

azbfs/zc_policy_request_log.go

@@ -33,8 +33,9 @@ func (o RequestLogOptions) defaults() RequestLogOptions {
 	return o
 }
 
-// NewRequestLogPolicyFactory creates a RequestLogPolicyFactory object configured using the specified options.
-func NewRequestLogPolicyFactory(o RequestLogOptions) pipeline.Factory {
+// NewRequestLogPolicyFactory_Deprecated creates a RequestLogPolicyFactory object configured using the specified options.
+// Deprecated because we are moving to centralize everything on the one logging policy in STE
+func NewRequestLogPolicyFactory_Deprecated(o RequestLogOptions) pipeline.Factory {
 	o = o.defaults() // Force defaults to be calculated
 	return pipeline.FactoryFunc(func(next pipeline.Policy, po *pipeline.PolicyOptions) pipeline.PolicyFunc {
 		// These variables are per-policy; shared by multiple calls to Do

azure-pipelines.yml

@@ -1,14 +1,14 @@
 trigger:
   branches:
     include:
-      - master
       - dev
+      - master
 
 pr:
   branches:
     include:
-      - master
       - dev
+      - master
 
 jobs:
   - job: Linux_and_Windows_Builds
@@ -42,4 +42,75 @@ jobs:
 
       - task: PublishBuildArtifacts@1
         displayName: 'Publish Artifacts'
-        condition: succeededOrFailed()
+        condition: succeededOrFailed()
+
+  - job: Test_On_Ubuntu
+    variables:
+      isMutexSet: 'false'
+    # allow maximum build time, in case we have build congestion
+    timeoutInMinutes: 360
+    pool:
+      vmImage: 'ubuntu-16.04'
+    steps:
+      - task: UsePythonVersion@0
+        name: 'Set_up_Python'
+        inputs:
+          versionSpec: '3.7'
+      - task: GoTool@0
+        name: 'Set_up_Golang'
+        inputs:
+          version: '1.12'
+      - script: |
+          pip install azure-storage-blob==12.0.0b3
+          # the recent release 1.0.0b4 has a breaking change
+          pip install azure-core==1.0.0b3
+          # acquire the mutex before running live tests to avoid conflicts
+          python ./tool_distributed_mutex.py lock "$(MUTEX_URL)"
+          # set the variable to indicate that the mutex was actually acquired
+          echo '##vso[task.setvariable variable=isMutexSet]true'
+        name: 'Acquire_the_distributed_mutex'
+      - script: |
+          # run unit test and build executable
+          # the set -e line is needed so that the unit tests failure would cause the job to fail properly
+          # -timeout 25m as our tests typically run between 10-15m and I'd rather have some headroom.
+          # "-check.v" (must be after package list) outputs timings
+          set -e
+          go test -timeout 25m -race -short -cover ./cmd ./common ./ste ./azbfs "-check.v" 
+          GOARCH=amd64 GOOS=linux go build -o azcopy_linux_amd64
+        name: 'Run_unit_tests'
+        env:
+          ACCOUNT_NAME: $(ACCOUNT_NAME)
+          ACCOUNT_KEY: $(ACCOUNT_KEY)
+          AWS_ACCESS_KEY_ID: $(AWS_ACCESS_KEY_ID)
+          AWS_SECRET_ACCESS_KEY: $(AWS_SECRET_ACCESS_KEY)
+      - script: |
+          go build -o test-validator ./testSuite/
+          mkdir test-temp
+          export AZCOPY_EXECUTABLE_PATH=$(pwd)/azcopy_linux_amd64
+          export TEST_SUITE_EXECUTABLE_LOCATION=$(pwd)/test-validator
+          export TEST_DIRECTORY_PATH=$(pwd)/test-temp
+          python ./testSuite/scripts/run.py
+        name: 'Run_smoke_tests'
+        env:
+          ACCOUNT_NAME: $(ACCOUNT_NAME)
+          ACCOUNT_KEY: $(ACCOUNT_KEY)
+          AWS_ACCESS_KEY_ID: $(AWS_ACCESS_KEY_ID)
+          AWS_SECRET_ACCESS_KEY: $(AWS_SECRET_ACCESS_KEY)
+          AZCOPY_OAUTH_TOKEN_INFO: $(AZCOPY_OAUTH_TOKEN_INFO)
+          CONTAINER_OAUTH_URL: $(CONTAINER_OAUTH_URL)
+          CONTAINER_OAUTH_VALIDATE_SAS_URL: $(CONTAINER_OAUTH_VALIDATE_SAS_URL)
+          CONTAINER_SAS_URL: $(CONTAINER_SAS_URL)
+          FILESYSTEM_SAS_URL: $(FILESYSTEM_SAS_URL)
+          FILESYSTEM_URL: $(FILESYSTEM_URL)
+          OAUTH_AAD_ENDPOINT: $(OAUTH_AAD_ENDPOINT)
+          OAUTH_TENANT_ID: $(OAUTH_TENANT_ID)
+          PREMIUM_CONTAINER_SAS_URL: $(PREMIUM_CONTAINER_SAS_URL)
+          S2S_DST_BLOB_ACCOUNT_SAS_URL: $(S2S_DST_BLOB_ACCOUNT_SAS_URL)
+          S2S_SRC_BLOB_ACCOUNT_SAS_URL: $(S2S_SRC_BLOB_ACCOUNT_SAS_URL)
+          S2S_SRC_FILE_ACCOUNT_SAS_URL: $(S2S_SRC_FILE_ACCOUNT_SAS_URL)
+          S2S_SRC_S3_SERVICE_URL: $(S2S_SRC_S3_SERVICE_URL)
+          SHARE_SAS_URL: $(SHARE_SAS_URL)
+      - script: python ./tool_distributed_mutex.py unlock "$(MUTEX_URL)"
+        name: 'Release_the_distributed_mutex'
+        # this runs even if the job was canceled (only if the mutex was acquired by this job)
+        condition: and(always(), eq(variables['isMutexSet'], 'true'))