Merge pull request #308 from serilog/dev

Release 5.5.1

CHANGES.md

@@ -1,3 +1,9 @@
+# 5.5.1
+* Fixed issue #300 Support DateTime2 data type for TimeStamp column (thanks to @stedel for the contribution).
+* Added Sourcelink support and publish symbols to nuget.org which allows consumers of the package to debug into the sink code.
+* Added troubleshooting tip in README with workaround for SqlClient issue regarding missing Microsoft.Data.SqlClient.Sni.dll in .NET Framework apps and updated `samples\AppConfigDemo` accordingly.
+* Added information in README about batched sink SqlBulkCopy behavior according to issue #209.
+
 # 5.5.0
 * Implemented enhancement #208: use Microsoft.Data.SqliClient for all platforms except net452 to enable Column Encryption (thanks to @mungk for the contribution).
 * Fixed issue #290 MissingMethodException with .NET Standard 2.0.

README.md

@@ -436,14 +436,19 @@ This column stores the event level (Error, Information, etc.). For backwards-com
 
 ### TimeStamp
 
-This column stores the time the log event was sent to Serilog as a SQL `datetime` (default) or `datetimeoffset` type. If `datetimeoffset` should be used, this can be configured as follows.
+This column stores the time the log event was sent to Serilog as a SQL `datetime` (default), `datetime2` or `datetimeoffset` type. If `datetime2` or `datetimeoffset` should be used, this can be configured as follows.
 
 ```csharp
 var columnOptions = new ColumnOptions();
 columnOptions.TimeStamp.DataType = SqlDbType.DateTimeOffset;
 ```
 
-Please be aware that you have to configure the sink for `datetimeoffset` if the used logging database table has a `TimeStamp` column of type `datetimeoffset`. On the other hand you must not configure for `datetimeoffset` if the `TimeStamp` column is of type `datetime`. Failing to configure the data type accordingly can result in log table entries with wrong timezone offsets or no log entries being created at all due to exceptions during logging.
+```csharp
+var columnOptions = new ColumnOptions();
+columnOptions.TimeStamp.DataType = SqlDbType.DateTime2;
+```
+
+Please be aware that you have to configure the sink for `datetimeoffset` if the used logging database table has a `TimeStamp` column of type `datetimeoffset`. If the underlying database uses `datetime2` for the `TimeStamp` column, the sink must be configured to use `datetime2`. On the other hand you must not configure for `datetimeoffset` if the `TimeStamp` column is of type `datetime` or `datetime2`. Failing to configure the data type accordingly can result in log table entries with wrong timezone offsets or no log entries being created at all due to exceptions during logging.
 
 While TimeStamp may appear to be a good candidate as a clustered primary key, even relatively low-volume logging can emit identical timestamps forcing SQL Server to add a "uniqueifier" value behind the scenes (effectively an auto-incrementing identity-like integer). For frequent timestamp range-searching and sorting, a non-clustered index is better.
 
@@ -699,6 +704,11 @@ Any Serilog application should _always_ call `Log.CloseAndFlush` before shutting
 AppDomain.CurrentDomain.ProcessExit += (s, e) => Log.CloseAndFlush();
 ```
 
+### Consider batched sink SqlBulkCopy behavior
+
+If you initialize the sink with `WriteTo` then it uses a batched sink semantics. This means that it does not directly issue an SQL command to the database for each log call, but it collectes log events in a buffer and later asynchronously writes a bulk of them to the database using [SqlBulkCopy](https://docs.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlbulkcopy?view=dotnet-plat-ext-3.1). **If SqlBulkCopy fails to write a single row of the batch to the database, the whole batch will be lost.** Unfortunately it is not easily possible (and probably only with a significant performance impact) to find out what lines of the batch caused problems. Therefore the sink cannot easily retry the operation with the problem lines removed. Typical problems can be that data (like the log message) exceeds the field length in the database or fields which cannot be null are null in the log event. Keep this in mind when using the batched version of the sink and avoid log events to be created with data that is invalid according to your database schema. Use a wrapper class or Serilog Enrichers to validate and correct the log event data before it gets written to the database.
+
+
 ### Test outside of Visual Studio
 
 When you exit an application running in debug mode under Visual Studio, normal shutdown processes may be interrupted. Visual Studio issues a nearly-instant process kill command when it decides you're done debugging. This is a particularly common problem with ASP.NET and ASP.NET Core applications, in which Visual Studio instantly terminates the application as soon as the browser is closed. Even `finally` blocks usually fail to execute. If you aren't seeing your last few events written, try testing your application outside of Visual Studio.
@@ -711,6 +721,10 @@ If you're reading about a feature that doesn't seem to work, check whether you'r
 
 Please check your NuGet references and confirm you are specifically referencing _Serilog.Sinks.MSSqlServer_. In the early days of .NET Core, there was a popular Core-specific fork of this sink, but the documentation and NuGet project URLs pointed here. Today the package is marked deprecated, but we continue to see some confusion around this.
 
+### .NET Framework apps must reference Microsoft.Data.SqlClient
+
+If you are using the sink in a .NET Framework app, make sure to add a nuget package reference to Microsoft.Data.SqlClient in your app project. This is necessary due to a bug in SqlClient which can lead to exceptions about missing Microsoft assemblies. Details can be found in [issue 283](https://github.com/serilog/serilog-sinks-mssqlserver/issues/283#issuecomment-664397489) and [issue 208](https://github.com/serilog/serilog-sinks-mssqlserver/issues/208#issuecomment-664503566).
+
 ## Querying Property Data
 
 Extracting and querying the property column directly can be helpful when looking for specific log sequences. SQL Server has query syntax supporting columns that store either XML or JSON data.

appveyor.yml

@@ -9,6 +9,7 @@ build_script:
 test: off
 artifacts:
 - path: artifacts/Serilog.*.nupkg
+- path: artifacts/Serilog.*.snupkg
 deploy:
 - provider: NuGet
   api_key:
@@ -18,7 +19,9 @@ deploy:
 - provider: GitHub
   auth_token:
     secure: p4LpVhBKxGS5WqucHxFQ5c7C8cP74kbNB0Z8k9Oxx/PMaDQ1+ibmoexNqVU5ZlmX
-  artifact: /Serilog.*\.nupkg/
+  artifacts:
+    /Serilog.*\.nupkg/
+    /Serilog.*\.snupkg/
   tag: v$(appveyor_build_version)
   on:
     branch: master

sample/AppConfigDemo/AppConfigDemo.csproj

@@ -12,6 +12,8 @@
     <FileAlignment>512</FileAlignment>
     <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
     <Deterministic>true</Deterministic>
+    <NuGetPackageImportStamp>
+    </NuGetPackageImportStamp>
   </PropertyGroup>
   <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
     <PlatformTarget>AnyCPU</PlatformTarget>
@@ -33,12 +35,36 @@
     <WarningLevel>4</WarningLevel>
   </PropertyGroup>
   <ItemGroup>
+    <Reference Include="Microsoft.Data.SqlClient, Version=1.13.20136.2, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.Data.SqlClient.1.1.3\lib\net46\Microsoft.Data.SqlClient.dll</HintPath>
+    </Reference>
     <Reference Include="Microsoft.Extensions.Configuration.Abstractions, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60, processorArchitecture=MSIL">
       <HintPath>..\..\packages\Microsoft.Extensions.Configuration.Abstractions.3.1.4\lib\netstandard2.0\Microsoft.Extensions.Configuration.Abstractions.dll</HintPath>
     </Reference>
     <Reference Include="Microsoft.Extensions.Primitives, Version=3.1.4.0, Culture=neutral, PublicKeyToken=adb9793829ddae60, processorArchitecture=MSIL">
       <HintPath>..\..\packages\Microsoft.Extensions.Primitives.3.1.4\lib\netstandard2.0\Microsoft.Extensions.Primitives.dll</HintPath>
     </Reference>
+    <Reference Include="Microsoft.Identity.Client, Version=3.0.8.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.Identity.Client.3.0.8\lib\net45\Microsoft.Identity.Client.dll</HintPath>
+    </Reference>
+    <Reference Include="Microsoft.IdentityModel.JsonWebTokens, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.IdentityModel.JsonWebTokens.5.5.0\lib\net461\Microsoft.IdentityModel.JsonWebTokens.dll</HintPath>
+    </Reference>
+    <Reference Include="Microsoft.IdentityModel.Logging, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.IdentityModel.Logging.5.5.0\lib\net461\Microsoft.IdentityModel.Logging.dll</HintPath>
+    </Reference>
+    <Reference Include="Microsoft.IdentityModel.Protocols, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.IdentityModel.Protocols.5.5.0\lib\net461\Microsoft.IdentityModel.Protocols.dll</HintPath>
+    </Reference>
+    <Reference Include="Microsoft.IdentityModel.Protocols.OpenIdConnect, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.IdentityModel.Protocols.OpenIdConnect.5.5.0\lib\net461\Microsoft.IdentityModel.Protocols.OpenIdConnect.dll</HintPath>
+    </Reference>
+    <Reference Include="Microsoft.IdentityModel.Tokens, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Microsoft.IdentityModel.Tokens.5.5.0\lib\net461\Microsoft.IdentityModel.Tokens.dll</HintPath>
+    </Reference>
+    <Reference Include="Newtonsoft.Json, Version=10.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\Newtonsoft.Json.10.0.1\lib\net45\Newtonsoft.Json.dll</HintPath>
+    </Reference>
     <Reference Include="Serilog, Version=2.0.0.0, Culture=neutral, PublicKeyToken=24c2f752a8e58a10, processorArchitecture=MSIL">
       <HintPath>..\..\packages\Serilog.2.9.0\lib\net46\Serilog.dll</HintPath>
     </Reference>
@@ -48,6 +74,16 @@
     </Reference>
     <Reference Include="System.Configuration" />
     <Reference Include="System.Core" />
+    <Reference Include="System.Data.Common, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\System.Data.Common.4.3.0\lib\net451\System.Data.Common.dll</HintPath>
+      <Private>True</Private>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="System.Drawing" />
+    <Reference Include="System.IdentityModel" />
+    <Reference Include="System.IdentityModel.Tokens.Jwt, Version=5.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL">
+      <HintPath>..\..\packages\System.IdentityModel.Tokens.Jwt.5.5.0\lib\net461\System.IdentityModel.Tokens.Jwt.dll</HintPath>
+    </Reference>
     <Reference Include="System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51, processorArchitecture=MSIL">
       <HintPath>..\..\packages\System.Memory.4.5.4\lib\net461\System.Memory.dll</HintPath>
     </Reference>
@@ -58,6 +94,7 @@
     <Reference Include="System.Runtime.CompilerServices.Unsafe, Version=4.0.6.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
       <HintPath>..\..\packages\System.Runtime.CompilerServices.Unsafe.4.7.1\lib\net461\System.Runtime.CompilerServices.Unsafe.dll</HintPath>
     </Reference>
+    <Reference Include="System.Windows.Forms" />
     <Reference Include="System.Xml.Linq" />
     <Reference Include="System.Data.DataSetExtensions" />
     <Reference Include="Microsoft.CSharp" />
@@ -82,4 +119,11 @@
     <Folder Include="Properties\" />
   </ItemGroup>
   <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
+  <Import Project="..\..\packages\Microsoft.Data.SqlClient.SNI.1.1.0\build\net46\Microsoft.Data.SqlClient.SNI.targets" Condition="Exists('..\..\packages\Microsoft.Data.SqlClient.SNI.1.1.0\build\net46\Microsoft.Data.SqlClient.SNI.targets')" />
+  <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
+    <PropertyGroup>
+      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
+    </PropertyGroup>
+    <Error Condition="!Exists('..\..\packages\Microsoft.Data.SqlClient.SNI.1.1.0\build\net46\Microsoft.Data.SqlClient.SNI.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\..\packages\Microsoft.Data.SqlClient.SNI.1.1.0\build\net46\Microsoft.Data.SqlClient.SNI.targets'))" />
+  </Target>
 </Project>

sample/AppConfigDemo/packages.config

@@ -1,10 +1,21 @@
 <?xml version="1.0" encoding="utf-8"?>
 <packages>
+  <package id="Microsoft.Data.SqlClient" version="1.1.3" targetFramework="net461" />
+  <package id="Microsoft.Data.SqlClient.SNI" version="1.1.0" targetFramework="net461" />
   <package id="Microsoft.Extensions.Configuration.Abstractions" version="3.1.4" targetFramework="net461" />
   <package id="Microsoft.Extensions.Primitives" version="3.1.4" targetFramework="net461" />
+  <package id="Microsoft.Identity.Client" version="3.0.8" targetFramework="net461" />
+  <package id="Microsoft.IdentityModel.JsonWebTokens" version="5.5.0" targetFramework="net461" />
+  <package id="Microsoft.IdentityModel.Logging" version="5.5.0" targetFramework="net461" />
+  <package id="Microsoft.IdentityModel.Protocols" version="5.5.0" targetFramework="net461" />
+  <package id="Microsoft.IdentityModel.Protocols.OpenIdConnect" version="5.5.0" targetFramework="net461" />
+  <package id="Microsoft.IdentityModel.Tokens" version="5.5.0" targetFramework="net461" />
+  <package id="Newtonsoft.Json" version="10.0.1" targetFramework="net461" />
   <package id="Serilog" version="2.9.0" targetFramework="net461" />
   <package id="Serilog.Settings.AppSettings" version="2.2.2" targetFramework="net461" />
   <package id="System.Buffers" version="4.5.1" targetFramework="net461" />
+  <package id="System.Data.Common" version="4.3.0" targetFramework="net461" />
+  <package id="System.IdentityModel.Tokens.Jwt" version="5.5.0" targetFramework="net461" />
   <package id="System.Memory" version="4.5.4" targetFramework="net461" />
   <package id="System.Numerics.Vectors" version="4.5.0" targetFramework="net461" />
   <package id="System.Runtime.CompilerServices.Unsafe" version="4.7.1" targetFramework="net461" />

src/Serilog.Sinks.MSSqlServer/Serilog.Sinks.MSSqlServer.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <Description>A Serilog sink that writes events to Microsoft SQL Server</Description>
-    <VersionPrefix>5.5.0</VersionPrefix>
+    <VersionPrefix>5.5.1</VersionPrefix>
     <Authors>Michiel van Oudheusden;Christian Kadluba;Serilog Contributors</Authors>
     <TargetFrameworks>netstandard2.0;net452;net461;net472;netcoreapp2.0;netcoreapp2.2</TargetFrameworks>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
@@ -18,6 +18,10 @@
     <PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
     <RepositoryUrl>https://github.com/serilog/serilog-sinks-mssqlserver</RepositoryUrl>
     <RepositoryType>git</RepositoryType>
+    <PublishRepositoryUrl>true</PublishRepositoryUrl>
+    <EmbedUntrackedSources>true</EmbedUntrackedSources>
+    <IncludeSymbols>true</IncludeSymbols>
+    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
     <RuntimeIdentifiers>win</RuntimeIdentifiers>
     <GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
     <GenerateAssemblyDescriptionAttribute>false</GenerateAssemblyDescriptionAttribute>
@@ -26,6 +30,7 @@
 
   <ItemGroup>
     <!-- Whenever these are updated, also update versions in packages.config for net452 -->
+    <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="All"/>
     <PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="3.0.0">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>

src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptions.cs

@@ -20,15 +20,15 @@ public TimeStampColumnOptions() : base()
             }
 
             /// <summary>
-            /// The TimeStamp column only supports the DateTime and DateTimeOffset data types.
+            /// The TimeStamp column only supports the DateTime, DateTime2 and DateTimeOffset data types.
             /// </summary>
             public new SqlDbType DataType
             {
                 get => base.DataType;
                 set
                 {
-                    if (value != SqlDbType.DateTime && value != SqlDbType.DateTimeOffset)
-                        throw new ArgumentException("The Standard Column \"TimeStamp\" only supports the DateTime and DateTimeOffset formats.");
+                    if (value != SqlDbType.DateTime && value != SqlDbType.DateTimeOffset && value != SqlDbType.DateTime2)
+                        throw new ArgumentException("The Standard Column \"TimeStamp\" only supports the DateTime, DateTime2 and DateTimeOffset formats.");
                     base.DataType = value;
                 }
             }

src/Serilog.Sinks.MSSqlServer/Sinks/MSSqlServer/Output/JsonLogEventFormatter.cs

@@ -124,7 +124,7 @@ private void WriteTimeStampIfPresent(LogEvent logEvent, TextWriter output, ref s
             output.Write(precedingDelimiter);
             precedingDelimiter = _commaDelimiter;
             var colData = WritePropertyName(logEvent, output, StandardColumn.TimeStamp);
-            var value = _columnOptions.TimeStamp.DataType == SqlDbType.DateTime
+            var value = (_columnOptions.TimeStamp.DataType == SqlDbType.DateTime || _columnOptions.TimeStamp.DataType == SqlDbType.DateTime2)
                 ? ((DateTime)colData.Value).ToString("o", CultureInfo.InvariantCulture)
                 : ((DateTimeOffset)colData.Value).ToString("o", CultureInfo.InvariantCulture);
             JsonValueFormatter.WriteQuotedJsonString(value, output);

test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/ColumnOptions/TimeStampColumnOptionsTests.cs

@@ -40,5 +40,16 @@ public void CannotSetDataTypeNVarChar()
             // Act and assert - should throw
             Assert.Throws<ArgumentException>(() => options.TimeStamp.DataType = SqlDbType.NVarChar);
         }
+
+        [Trait("Feature", "#300")]
+        [Fact]
+        public void CanSetDataTypeDateTime2()
+        {
+            // Arrange
+            var options = new Serilog.Sinks.MSSqlServer.ColumnOptions();
+
+            // Act - should not throw
+            options.TimeStamp.DataType = SqlDbType.DateTime2;
+        }
     }
 }

test/Serilog.Sinks.MSSqlServer.Tests/Sinks/MSSqlServer/Output/JsonLogEventFormatterTests.cs

@@ -1,4 +1,4 @@
-using System;
+using System;
 using System.Collections.Generic;
 using System.Data;
 using System.IO;
@@ -90,6 +90,27 @@ public void FormatTimeStampColumnTypeDateTimeRendersCorrectTimeStamp()
             Assert.Equal(expectedResult, renderResult);
         }
 
+        [Fact]
+        [Trait("Feature", "#300")]
+        public void FormatTimeStampColumnTypeDateTime2RendersCorrectTimeStamp()
+        {
+            // Arrange
+            const string expectedResult = "{\"TimeStamp\":\"2020-07-01T09:41:10.1230000\",\"Level\":\"Information\",\"Message\":\"\",\"MessageTemplate\":\"Test message template\"}";
+            _testColumnOptions.TimeStamp.DataType = SqlDbType.DateTime2;
+            var testLogEvent = CreateTestLogEvent(new DateTimeOffset(2020, 7, 1, 9, 41, 10, 123, TimeSpan.Zero));
+
+            // Act
+            string renderResult;
+            using (var outputWriter = new StringWriter())
+            {
+                _sut.Format(testLogEvent, outputWriter);
+                renderResult = outputWriter.ToString();
+            }
+
+            // Assert
+            Assert.Equal(expectedResult, renderResult);
+        }
+
         [Fact]
         public void FormatWithPropertiesRendersCorrectProperties()
         {