Sunday, June 28, 2015

SQL Server 2016 - Truncate at Partition level

Enhancements to TRUNCATE in SQL Server 2016:

SQL Server 2016 introduced truncation at partition level without logging at the individual rows.
This is kind of having WHERE clause to DELETE statement.

The new TRUNCATE statement in SQL Server 2016 is:

TRUNCATE TABLE dbo.PartitionedTable
WITH (PARTITIONS (2, 4, 6 TO 8))

Above statement will truncate all the partitions 2, 4, 6, 7 and 8.



Labels: , , , ,

SQL Server 2016 - Query Store

Query Store in SQL Server:
Query store is a new component in SQL Server that captures Queries, Query Plans, Runtime Statistics, etc. inside the database.

It is available at database level and we can enforce the SQL Server query processor to execute the queries in a specific manner by using forcing plans.

Query store collects query texts and all relevant properties, as well as query plan choices and performance metrics.

It will provide the details regarding fire queries, query plans that it is used to execute the queries which will help in trouble shooting the performance issues.

Find below some of the things that we can do easily with Query Store:
1. Conduct system-wide or database-level analysis and troubleshooting
2. Access the full history of query execution
3. Quickly pinpoint the most expensive queries
4. Get all queries whose performance regressed over time
5. Easily force a better plan from history

Typical Performance trouble shooting that we can follow through Query store is:


We can easily enable Query Store by right clicking on the database and selecting the properties:

After you enable this option then we can see Query Store folder on expanding the database:

Click on any of the options available on expanding QueryStore, you will see the window as shown below:

Select the specific query, query plan and then select the option of ForcePlan OR UnForcePaln plan for that particular query.

Labels: , , , ,

Friday, June 26, 2015

SQL Server 2016 - Temporal Tables

Temporal Tables:
Temporal tables are nothing but maintaining historical information for the rows by the SQL server itself for the given table.
Instead of implementing SCD for the tables by the developers, if you create the table as Temporal Table then SQL Server automatically will take care of maitaining the history for the changing data.

To Create the temporal tables we should use below syntax (Here i have created Department table):
CREATE TABLE dbo.Department
(
DepartmentID int NOT NULL IDENTITY(1,1) PRIMARY KEY CLUSTERED,
DepartmentName varchar(50) NOT NULL,
ManagerID int NULL,

ValidFrom datetime2 GENERATED ALWAYS AS ROW START NOT NULL DEFAULT CAST('1900-01-01' AS DATETIME2) ,
ValidTo datetime2 GENERATED ALWAYS AS ROW END NOT NULL DEFAULT CAST('9999-12-31' AS DATETIME2) ,

PERIOD FOR SYSTEM_TIME
(
StartDate,
EndDate
)
)
WITH ( SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.DepartmentHistory) );
GO

1. Two additional Start & End date Audit columns of datetime2 datatype for capturing the validity of records. You can use any meaningful column name here, we will use StartDate and EndDate column names

2. Both the column names have to be specified in PERIOD FOR SYSTEM_TIME (StartDate, EndDate) clause along with the columns.

3. Specify WITH (SYSTEM_VERSIONING = ON) option at the end of the CREATE TABLE statement with optional (HISTORY_TABLE = <<History_Table_Name>>) option.
Once you create the table it will automatically create two tables internally
one for current data and other for the historical data.


When you are querying the table, you can mention the table name that you have mentioned in Create table statement that will automatically call historical table internally and will return the results.

If we want to see only historical records then we can query the historical table that we have mentioned in our create table script.

If there are no updates to the data then no rows will be available in the historical table.

Limitation of Temporal Tables:
1. We can't query Temporal tables using Linked Server
2. History table should not have constraints (PK, FK, Table or Column constraints).
3. INSERT and UPDATE statements cannot reference the SYSTEM_TIME period columns.
4. TRUNCATE TABLE is not supported till SYSTEM_VERSIONING is ON and we can truncate if we Switch OFF this setting
5. Direct modification of the data in a history table is not permitted.
6. INSTEAD OF triggers are not permitted on either the tables.
7. Usage of Replication technologies is limited.

For more information refer Temporal tables Part 1 and Part 2

Labels: , , , , ,

Monday, June 22, 2015

New features in SQL 2016


Below are the new features introduced in SQL 2016:
  1. Always Encrypted
  2. Polybase
  3. Stretch Database
  4. In-Memory OLTP
  5. Columnstore indexes
  6. Dynamic Data Masking
  7. Multiple TempDB files
  8. Live Query Statistics
  9. JSON Support

Always Encrypted is designed to allow encrypted data to always be encrypted but still allow SQL Server to work with the data.

It allows you to access the data available in Hadoop through T-SQL commands.

It allows you to access the latest data from local server and historical information from Azure DB.
We can use this feature whenver we have less space in our local server to store the data.

There are many enhancements to In-Memory OLTP in SQL 2016 such as increase in size of the table, alteration to the existing tables etc..

Columnstore Indexes:
It was introduced in SQL Server 2012 and enhanced in 2014 and 2016.
Columnstore index stores the data in terms of columns instead of rows.

Find below the enhancements that were made in SQL 2016:
1. A Table will still have one NonClustered ColumnStore Index, but this will be updatable so we can also update table
2. Now you can create a Filtered NonClustered ColumnStore Index by specifying a WHERE clause to it.
3. A Clustered ColumnStore Index can have one or more NonClustered Indexes.
4. Clustered ColumnStore Index can now be Unique indirectly as you can create a Primary Key Constraint on a Heap table (and also Foreign Key constraints).
5. MS allowed us to create column store index on memory optimized tables but you need to adhere to following rules:
  1. It should be created while we are creating memory optimized table itself
  2. It should include all the columns available in the table
  3. No filtered index allowed that means we need to include all the rows 
Dynamic Data Masking:
It is about masking of the data available in columns.
There are three ways to mask the data:
a. default()
b. email()
c. partial()
all the above functions will provide you the customizations to the masked data.
For more information refer Dynamic Masking in SQL 2016

Multiple TempDB files:
We can have more than tempdb data files in SQL Server 2016.
By default number of data files will be equal to the number of cores available in CPU OR 8 which ever is minimum, it will be set to that.
The primary file for tempdb will be temp.mdf. Others will be names as tempdb_mssql_#.ndf where # represents the unique identification number for each data file.

We need to mention the tempdb file count when we are rebuilding system databases as it won't persist. We cab provide the count of tempdb files with SQLTEMPDBFILECOUNT parameter. If it is not provided it will take the default value.

SQL Server 2016 provides you with the live statistics of currently active queries.

JSON Support:
SQL 2016 introduced new clause FOR JSON to convert the resulted data into JSON format.
For more details refer FOR JSON in SQL 2016

To know about the improvements in SSIS in SQL 2016 go through improvements in SSIS in SQL 2016

Labels: , , , ,

Live Query Statistics in SQL 2016

Live Query Statistics:

SQL Server 2016 provides you the live statistics of a running query and you can enable this option by simply clicking the highlighted area.


Once you enable this option and run the query, you will the result set area as shown below:




By seeing this we can easily understand how much our query executed and which part of the query is taking more time to execute.

Labels: , , , ,

Enhancements to In-Memory OLTP in SQL 2016

In-Memory OLTP:
It is introduced in SQL Server 2014 and they enhanced many things in 2016 release.

Find below the list of enhancements that have been made in SQL Server 2016:
Feature/Limit
SQL Server 2014
SQL Server 2014
Maximum size of table
256 GB
2 TB
LOB (varbinary(max), [n]varchar(max))
Not Supported
Supported
Transparent Data Encryption (TDE)
Not Supported
Supported
ALTER PROCEDURE / sp_recompile
Not Supported
Supported
Nested natively compiled procedure
Not Supported
Supported
Natively-compiled scalar UDFs
Not Supported
Supported
ALTER TABLE
Not Supported
Supported (Only Offline)
Indexes on NULLable columns
Not Supported
Supported
Foreign Keys
Not Supported
Supported
Check/Unique Constraints
Not Supported
Supported
Parallelism
Not Supported
Supported
OUTER JOIN, OR, NOT, UNION, DISTINCT, EXISTS, IN
Not Supported
Supported

ALTER TABLE is an offline operation, by using this we can do following things:
a. Adding columns
b. Dropping columns
c. Indexes
d. Constraints

We can also change the bucket count values with REBUILD option but it needs 2x memory.

Labels: , , ,

Stretch database feature in SQL Server 2016

Stretch Database:
It allows you to store a portion of a table in the Azure SQL Database.

Stretch Database feature offer local server performance for latest data and cloud storage for old data without any change to the application.
Users are mostly interested in latest data, with this feature latest data will be available in your local server where as the historical information will be moved to Azure.
When you enable stretch database, it creates a second database that is hosted in Azure. Then when you mark a table as “stretch”, SQL Server will automatically start moving its data into the cloud.
Currently only “archive table” mode is enabled, which assumes that it is working on a history table and moves all of the rows.
The “archive row” mode, which hasn’t been released yet, will use a WHERE clause to determine which rows to archive.
Common scenarios include rows that are more than a year old or have a flag indicating that the row is no longer live (e.g. completed orders).
No need to change anything in application : The SQL to query a stretch table is exactly the same as the SQL needed to query for a normal table. The query execution engine will automatically take care of distributing the query between the local and Azure-based server. This means you can enable stretch on a database without any changes to the applications using it.
Needs changes in backup and restore mechanism: Normal backups only include the locally hosted data. A full backup, including data located in the stretch database, will require a different procedure.
Limitations:
Below column types are not supported:
  1. filestream
  2. timestamp
  3. sql_variant
  4. XML
  5. geometry
  6. geography
  7. hierarchyid
  8. CLR user-defined types (UDTs)
Below features also not supported:

  1. Column Set
  2. Computed Columns
  3. Check constraints
  4. Foreign key constraints that reference the table
  5. Default constraints
  6. XML indexes
  7. Full text indexes
  8. Spatial indexes
  9. Clustered columnstore indexes
  10. Indexed views that reference the table
    For more information on how to enable the Stretch database feature refer SQL Server 2016 stretch database feature

Labels: , , , ,

Polybase in SQL Server 2016

Polybase allows you to access the data stored in Hadoop by using T-SQL.
Sqoop Limitations
PolyBase provides the ability to integrate a Hadoop cluster with SQL Server, which will allow you to query the data in a Hadoop Cluster from SQL Server.
 
While the Apache environment provided the Sqoop application to integrate Hadoop with other relational databases, it wasn’t really enough. With Sqoop, the data is actually moved from the Hadoop cluster into SQL Server, or the relational database of your choice. This is problematic because you needed to know before you ran Sqoop that you had enough room within your database to hold all the data.
Polybase – Hadoop Integration with SQL Server
Unlike Sqoop, PolyBase does not load data into SQL Server. Instead it provides SQL Server with the ability to query Hadoop while leaving the data in the HDFS clusters.
As Hadoop is schema-on-read, within SQL server you generate the schema to apply to your data stored in Hadoop. After the table schema is known, PolyBase provides the ability to then query data outside of SQL Server from within SQL Server. 

Using PolyBase it is possible to integrate data from two completely different file systems, providing freedom to store the data in either place.

For more details regarding how we can pull the data from Hadoop to SQL Server, refer Polybase in SQL 2016. 

Labels: , , ,

Always Encrypted feature in SQL Server 2016

Always Encrypted is designed to allow encrypted data to always be encrypted but still allow SQL Server to work with the data.

  1. Data is encrypted at all times and will only be decrypted once it reaches the application:
    In the diagram above you see that the data for one or more columns of a table is stored in an encrypted state. When SQL Server acts on this data locally it acts only on the encrypted version. It never decrypts it and so it’s encrypted in memory as well as on the wire as it transits the network (or Internet) on the way to the client. SQL Server treats the encrypted data as if it were the raw field. Only at the point where the data reaches the client is it decrypted for use in your applications. This makes the encrypted data nearly impervious to man-in-the-middle attacks or file based decryption on the server.

  2. Encryption keys are not stored on the server
    SQL Server will not store the keys to decrypt the data it stores in Always On Encrypted fields.

    You have to register on the server but the certificate will be available on the clients and the actual certificate is not accessible on the server. The client can store the encryption keys currently in the local certificate store and in time in Azure Key Vaults or Hardware Security Modules. One thing to consider is that SQL Server DBAs will not be able to view any of the encrypted data during migrations, imports, etc.
    Other work around will be to install the client certificate locally on the server but that would negate much of the security of Always On Security by giving a hacked server access to the encrypted data.
  3. Always on Encrypted columns support only equality operators only
    As the data is encrypted in SQL Server only column equality is supported. That means you can use equality in WHERE, JOIN, and GROUP BY clauses but you can’t use LIKE or other aggregation or pattern matching features like SUM, SUBSTRING, etc.

  4. You need to upgrade your client software to .NET 4.6
    As the encryption and decryption is done at the client, your clients will need to upgrade to the new version of the .NET framework which is 4.6. Version 1 currently only supports SQL Server Client driver but ODBC and JDBC drivers will be coming at a later date.

  5. This is not TDE (Transparent Data Encryption)TDE is a feature of SQL Server that encrypts the data files themselves on the server. This is basically encrypting your entire database. While this provides security for stored or “at rest” data, once the file is decrypted by SQL Server the data remains decrypted in plain text in memory and when it’s sent across the wires.
    Always On Encryption will encrypt the data in memory and on the wire but due to the performance hit on the client it’s not recommended to secure every field in a table or database as TDE does. So you need to consider this when you decide which security feature to implement.
  6. Encrypted columns take significantly more space
    While columns defined with Always On Encryption specify the size of the original decrypted data they actually store the encrypted value which can be much larger. This we need to consider while deciding on the number of columns to be encrypted in your table.

Labels: , ,

New in SQL Server 2016 - SSIS

Below are the new capabilities introduced in SQL Server 2016 (Integration Services):
  1. Always On support
  2. Deploy Packages to Integration Services Server
  3. Project Upgrade

Always On Support:
Always On Support is a disaster-recovery solution that provides an enterprise-level alternative to database mirroring.
In SQL Server 2016, SSIS introduces new capability in order to provide high-availability for the SSISDB database and its contents (projects, packages, execution logs, etc.).
You can add the SSISDB database to an AlwaysOn Availability Group. When a failover occurs, one of the secondary nodes automatically becomes the new primary node.

For more information please refer AlwaysOn for SSIS Catalog (SSISDB).

Incremental Package Deployment:
It will allows you to deploy one or more packages without deploying the whole project.
You can incrementally deploy packages using:
a. Deployment Wizard
b. SSMS (uses Deployment Wizard)
c. Stored procedures
d. Management Object Model (MOM) API
For more information please refer Deploy Packages to Integration Services Server.

Project Upgrade:
When you upgrade your SSIS projects from previous to the current version, the project-level connection managers will continue to work as usual and the package layout/annotations are retained.



Labels: , , ,