PowerShell – Copy SQL Server Tables – Structure, Data & Indexes To Another Database/Instance

Image

 

UPDATE Mar 25, 2021:

I have created an updated version of the script with enhancements and sample usage that is available at my GitHub:

https://github.com/SQLJana/PowerShell/blob/c6cbe1d5b4f97026d1b00df59403dcd1ed367065/Copy-SQLTable.ps1

Requirement:

Recently I got a request from a user that he wanted to copy a specific set of tables and their indexes into a new database to ship to the vendor for analysis. The problem was that the DB had thousands of tables (8,748 to be precise). Hunting and pecking for specific tables from that is possible but tedious. Even if I managed to do that, I still have to manually script out the indexes and run them in target as the native “Import/Export Wizard” does not do indexes. It only copies the table structure and data! I am not a big fan of point and click anyway.

Solution: Copy-SQLTable

PowerShell is my tool of choice and it took me a couple of hours to code/refine/test and blog. So, here is what I did to script the tables and their indexes and then copy the data over. It is pretty self-explanatory. I am sure a lot of you will find this very useful. I know that I will use it a lot.

The function comments:

PowerShell code gets manged by wordpress when it is in between block comments! I have separated out the code comments here so that the code block which is below this does not get messed up.

#
#######################
#
.SYNOPSIS
    Copies tables from source to target SQL instance. Optionally creates indexes

.DESCRIPTION
    It is usually a pain to copy with the UI wizard to copy data for specific tables especially
        if it included copying the indexes too! This function eases the pain!

.INPUTS
    Source and target information and table list to copy

.OUTPUTS
    None

.EXAMPLE 

        [string] $sourceInstance = 'MySourceServer\SourceInst'
        [string] $sourceDB = 'MySourceDB'
        [string] $destInstance = $sourceInstance
        [string] $destDB = 'MyTargetDB_ForVendor'
        [bool] $dropTargetTableIfExists = $true
        [bool] $copyIndexes = $true
        [bool] $copyData = $true
        [string[]] $tables = @('dbo.T_STA_DWH_SEC_INDUSTRY_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_INDUSTRY',
                    'dbo.T_STA_DWH_SEC_CALC_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_CALC',
                    'dbo.T_REF_MAS_CLASS_DESC',
                    'dbo.T_REF_MAS_CLASS_MAP',
                    'dbo.T_REF_MAS_CLASSIFICATION_CATEGORY',
                    'dbo.T_DYN_MAS_SAP_ACC_ENTRY',
                    'dbo.T_STA_DWH_SECURITIES_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_CLASS',
                    'dbo.T_STA_MAS_SECURITIES',
                    'dbo.T_REF_MAS_COUNTRY',
                    'dbo.T_REF_MAS_CURRENCY',
                    'dbo.T_STA_BPS_SEC',
                    'dbo.T_STA_BPS_ULT',
                    'dbo.T_STA_DQM_SEC_CALC_CONTROL_RPT',
                    'dbo.T_STA_DQM_SEC_INDUSTRY_CONTROL_RPT',
                    'dbo.T_STA_DQM_SECURITY_CONTROL_RPT',
                    'dbo.T_STA_MAS_ISSUER')

        Copy-SQLTable `
            -SourceInstance $sourceInstance `
            -SourceDB $sourceDB `
            -DestInstance $destInstance `
            -DestDB $destDB `
            -DropTargetTableIfExists: $dropTargetTableIfExists `
            -CopyIndexes: $copyIndexes `
            -CopyData: $copyData `
            -Tables $tables `
            -Verbose
.NOTES 

Version History
    v1.0  - Jun 12, 2017. Jana Sattainathan [Twitter: @SQLJana] [Blog: sqljana.wordpress.com]

.LINK
    sqljana.wordpress.com
#
#

The function code:


function Copy-SQLTable
{
    [CmdletBinding()]
    param( 

        [Parameter(Mandatory=$true)]
        [string] $SourceInstance,

        [Parameter(Mandatory=$true)]
        [string] $SourceDB,        

        [Parameter(Mandatory=$true)]
        [string] $DestInstance,

        [Parameter(Mandatory=$true)]
        [string] $DestDB,

        [Parameter(Mandatory=$false)]
        [switch] $DropTargetTableIfExists = $false,

        [Parameter(Mandatory=$false)]
        [switch] $CopyIndexes = $true,

        [Parameter(Mandatory=$false)]
        [switch] $CopyConstraints = $false,

        [Parameter(Mandatory=$false)]
        [switch] $CopyData = $true,

        [Parameter(Mandatory=$true)]
        [string[]] $Tables,

        [Parameter(Mandatory=$false)]
        [int] $BulkCopyBatchSize = 10000,

        [Parameter(Mandatory=$false)]
        [int] $BulkCopyTimeout = 600   #10 minutes

    )

    [string] $fn = $MyInvocation.MyCommand
    [string] $stepName = "Begin [$fn]"  

    [string] $sourceConnString = "Data Source=$SourceInstance;Initial Catalog=$SourceDB;Integrated Security=True;"
    [string] $destConnString = "Data Source=$DestInstance;Initial Catalog=$DestDB;Integrated Security=True;"
    [int] $counter = 0

    try
    {    

        $stepName = "[$fn]: Import SQLPS module and initialize source connection"
        #---------------------------------------------------------------
        Write-Verbose $stepName

        Import-Module 'SQLPS'
        $sourceServer = New-Object Microsoft.SqlServer.Management.Smo.Server $SourceInstance
        $sourceDatabase = $sourceServer.Databases[$SourceDB]
        $sourceConn  = New-Object System.Data.SqlClient.SQLConnection($sourceConnString)
        $sourceConn.Open()

        foreach($table in $sourceDatabase.Tables)
        {
            $tableName = $table.Name
            $schemaName = $table.Schema
            $tableAndSchema = "$schemaName.$tableName"

            if ($Tables.Contains($tableAndSchema))
            {
                $counter = $counter + 1
                Write-Progress -Activity "Copy progress:" `
                            -PercentComplete ([int](100 * $counter / $Tables.Count)) `
                            -CurrentOperation ("Completed {0}% of the tables" -f ([int](100 * $counter / $Tables.Count))) `
                            -Status ("Working on table: [{0}]" -f $tableAndSchema) `
                            -Id 1

                Write-Verbose "[$fn]: ---------------------------------------------------------------"
                $stepName = "[$fn]: About to copy table [$tableAndSchema]"
                Write-Verbose $stepName
                Write-Verbose "[$fn]: ---------------------------------------------------------------"

                $stepName = "[$fn]: Create schema [$schemaName] in target if it does not exist"
                #---------------------------------------------------------------
                Write-Verbose $stepName

                $schemaScript = "IF NOT EXISTS (SELECT * FROM sys.schemas WHERE name = '$schemaName')
                                    BEGIN
                                        EXEC('CREATE SCHEMA $schemaName')
                                    END"

                Invoke-Sqlcmd `
                            -ServerInstance $DestInstance `
                            -Database $DestDB `
                            -Query $schemaScript

                if ($DropTargetTableIfExists -eq $true)
                {
                    Write-Verbose "[$fn]: Drop table [$tableName] in target if it exists"
                    #---------------------------------------------------------------
                    Write-Verbose $stepName

                    $schemaScript = "IF EXISTS (SELECT 1 WHERE OBJECT_ID('$tableAndSchema') IS NOT NULL)
                                        BEGIN
                                            EXEC('DROP TABLE $tableAndSchema')
                                        END"

                    Invoke-Sqlcmd `
                                -ServerInstance $DestInstance `
                                -Database $DestDB `
                                -Query $schemaScript
                }

                $stepName = "[$fn]: Scripting default scripting options - default"
                #----------------------------
                $scriptingCreateOptions = New-Object Microsoft.SqlServer.Management.Smo.ScriptingOptions
                Write-Verbose $stepName

                $scriptingCreateOptions.ExtendedProperties = $true; # Script Extended Properties

                #$scriptingCreateOptions.DriAllConstraints = $true   # to include referential constraints in the script
                #$scriptingCreateOptions.NoCollation = $false; # Use default collation
                #$scriptingCreateOptions.SchemaQualify = $true; # Qualify objects with schema names
                #$scriptingCreateOptions.ScriptSchema = $true; # Script schema
                #$scriptingCreateOptions.IncludeDatabaseContext = $true;
                #$scriptingCreateOptions.EnforceScriptingOptions = $true;
                #$scriptingCreateOptions.Indexes= $true # Yup, these would be nice
                #$scriptingCreateOptions.Triggers= $true # This should be included when scripting a database                

                $stepName = "[$fn]: Create constraints"
                #---------------------------------------------------------------
                Write-Verbose $stepName

                #Copy constraints
                if ($CopyConstraints -eq $true)
                {
                    $scriptingCreateOptions.DRIAll= $true     #All the constraints
                }
                else
                {
                    $scriptingCreateOptions.DRIAll= $false
                }

                $stepName = "[$fn]: Get the source table script for [$tableName] and create in target"
                #---------------------------------------------------------------
                Write-Verbose $stepName

                $Tablescript = ($table.Script($scriptingCreateOptions) | Out-String)

                Invoke-Sqlcmd `
                            -ServerInstance $DestInstance `
                            -Database $DestDB `
                            -Query $Tablescript

                #Only copy if needed. There may be a need to just copy table structures!
                if ($CopyData -eq $true)
                {
                    $stepName = "[$fn]: Get data reader for source table"
                    #---------------------------------------------------------------
                    Write-Verbose $stepName

                    $sql = "SELECT * FROM $tableAndSchema"
                    $sqlCommand = New-Object system.Data.SqlClient.SqlCommand($sql, $sourceConn)
                    [System.Data.SqlClient.SqlDataReader] $sqlReader = $sqlCommand.ExecuteReader()

                    $stepName = "[$fn]: Copy data from source to destination for table"
                    #---------------------------------------------------------------
                    Write-Verbose $stepName

                    $bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($destConnString, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
                    $bulkCopy.DestinationTableName = $tableAndSchema
                    $bulkCopy.BulkCopyTimeOut = $BulkCopyTimeout
                    $bulkCopy.BatchSize = $BulkCopyBatchSize
                    $bulkCopy.WriteToServer($sqlReader)
                    $sqlReader.Close()
                    $bulkCopy.Close()
                }

                #Do the index creations after the data load! That is the smarter thing to do.
                if ($CopyIndexes -eq $true)
                {
                    $stepName = "[$fn]: Create indexes for [$tableName] in target"
                    #---------------------------------------------------------------
                    Write-Verbose $stepName

                    foreach($index in $table.Indexes )
                    {
                        Write-Verbose "Creating index [$($index.Name)] for [$tableName]"

                        $indexScript = ($index.script() | Out-String)

                        Invoke-Sqlcmd `
                            -ServerInstance $DestInstance `
                            -Database $DestDB `
                            -Query $indexScript
                    }
                }

            }
        }

        Write-Verbose 'Cleanup'
        #---------------------------------------------------------------

        $sourceConn.Close()

    }
    catch
    {
        [Exception]$ex = $_.Exception
        Throw "Unable to copy table(s). Error in step: `"{0}]`" `n{1}" -f `
                        $stepName, $ex.Message
    }
    finally
    {
        #Return value if any
    }
}

Usage:

Below is a sample usage example where I try to copy a few tables from a source instance database to another database on the same instance (used as target).

        [string] $sourceInstance = 'MySourceServer\SourceInst'
        [string] $sourceDB = 'MySourceDB'
        [string] $destInstance = $sourceInstance
        [string] $destDB = 'MyTargetDB_ForVendor'
        [bool] $dropTargetTableIfExists = $true
        [bool] $copyIndexes = $true
        [bool] $copyData = $true
        [string[]] $tables = @('dbo.T_STA_DWH_SEC_INDUSTRY_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_INDUSTRY',
                    'dbo.T_STA_DWH_SEC_CALC_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_CALC',
                    'dbo.T_REF_MAS_CLASS_DESC',
                    'dbo.T_REF_MAS_CLASS_MAP',
                    'dbo.T_REF_MAS_CLASSIFICATION_CATEGORY',
                    'dbo.T_DYN_MAS_SAP_ACC_ENTRY',
                    'dbo.T_STA_DWH_SECURITIES_RPT_DQM',
                    'dbo.T_STA_MAS_SEC_CLASS',
                    'dbo.T_STA_MAS_SECURITIES',
                    'dbo.T_REF_MAS_COUNTRY',
                    'dbo.T_REF_MAS_CURRENCY',
                    'dbo.T_STA_BPS_SEC',
                    'dbo.T_STA_BPS_ULT',
                    'dbo.T_STA_DQM_SEC_CALC_CONTROL_RPT',
                    'dbo.T_STA_DQM_SEC_INDUSTRY_CONTROL_RPT',
                    'dbo.T_STA_DQM_SECURITY_CONTROL_RPT',
                    'dbo.T_STA_MAS_ISSUER')

        Copy-SQLTable `
            -SourceInstance $sourceInstance `
            -SourceDB $sourceDB `
            -DestInstance $destInstance `
            -DestDB $destDB `
            -DropTargetTableIfExists: $dropTargetTableIfExists `
            -CopyIndexes: $copyIndexes `
            -CopyData: $copyData `
            -Tables $tables `
            -Verbose

Below, you will see a screenshot of the function in action:
Copy-SQLTable

Features:

* Modular and generic to copy 1 or more tables
* Copy just table structures
* Create schemas in target based on the source table list
* Copy indexes (optionally)
* Copy data (optionally)
* Drop and recreate tables

Limitations

There are some limitations that I can think of but it should still be very usable for copying small/medium sized tables.

* No support for SQL username/password (only integrated security)
* No parallelism (serial copy)
* No progress information at the table row level for copy
* Table names have to be in the form schema.table without square brackets!

Code is not as good as I would like it to be but it works and in my tests, I was able to copy over tables with a few million rows without issues. Also, I am sure that a lot of bells and whistles can be added to this one.

If you have comments or suggestions, please comment below.

40 thoughts on “PowerShell – Copy SQL Server Tables – Structure, Data & Indexes To Another Database/Instance

    1. Thanks Don. I am well. Hope you are too. Hope to see you this Thursday.

      I will follow up with the contribution to dbatools. Thanks for the link.

      1. I looked at that cmdlet first but it only copies data between preexisting tables. It seems they have the intention to create a copy-dbatableschema at some point which I thought your solution might be able to contribute to.

  1. What a great find! I’m so happy I found this because I was struggling to programmatically do this without using an intermediate step which bcp requires!

  2. Hello,
    Thanks for the tips !
    But in your example, instead of having a list in “hard” on the script, how could i implement a list of table that could be listed in a txt file ? ( sorry i’m a beginner in PS )

    1. Hi Max. In PowerShell, reading from a text file into an array variable is quite simple
      [string[]]$tables = Get-Content -Path ‘C:\USER\Documents\TableList.txt’

      Then, you just pass the above array to my function just as shown in the example in the section “Usage:”. In fact, all you need to do is replace the line in “Usage” that says “[string[]] $tables = @(‘dbo…” with this above line that reads from text file.

      1. Hello Sir
        Thank you so much for your answer
        it was really helpful
        But maybe i can ask you some help again.
        I’m working on a projet to export on sql file the table
        I could find this “https://gist.github.com/vincpa/1755925” which is working on the DBserver “locally” very well
        But if i’m using the script remotely on my laptop ( of course i have added the connexion to sql server on the script), the script is unable to write the datas. i have checked the variables and all seems ok.
        He is able to create the folders, not the files.
        Any ideas?

  3. Hi Jana,
    Really great function thank you. Saves so much time when compared to using SQL Import/Export wizard or writing T-SQL to do the same.
    I would second Donal’s comment that it will be a valued contribution to dbatools.io
    Cheers
    Kane

    1. Hi Kane,
      Thank you for the feedback and the kind words. I complete agree that I should make it part of dbatools but I have not had the time to make it happen as this has to evolve to be more generic with additional error handling to fit the dbatools standard. I am fully open to anyone taking this code and making it a part of dbatools.

      Thanks, Jana

  4. Thank you, great script – To add status you just need two lines in the bulkcopy definition (this using verbose, i.e. add -verbose to see):
    $bulkCopy.NotifyAfter=$NotifySize
    $bulkCopy.Add_SQlRowscopied( {Write-Verbose “$($args[1].RowsCopied) rows copied”} )

    Nice little addition; you can alternatively give a progress bar instead of Write-Verbose with Write-Progress but to be useful you need to count rows first.

    Thanks for the script and ideas,

    Steve

    1. Hi, I added that syntax to get how many rows got copied but it doesnt show any thing. Below is the part of script where I updated. Is this not correct?

      $bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($destConnString, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity)
      $bulkCopy.DestinationTableName = $tableAndSchema
      $bulkCopy.BulkCopyTimeOut = $BulkCopyTimeout
      $bulkCopy.BatchSize = $BulkCopyBatchSize
      $bulkCopy.NotifyAfter=$NotifySize
      $bulkCopy.Add_SQlRowscopied( {Write-Verbose “$($args[1].RowsCopied) rows copied”} )
      $bulkCopy.WriteToServer($sqlReader)
      $sqlReader.Close()
      $bulkCopy.Close()

    2. Hi Steve, You are welcome and thanks for the suggestion. I finally got a chance to add the row copy progress as both Verbose output and progress output. Please check out the latest code on GitHub (link referenced at the top of this post).

  5. Hi Jana,
    I find your query very helpful. However, I need to sync schema between tables. The number of tables may vary. How to modify your script to sync all tables in a database without having to manually adding table names to an array. I would like the array to have name from sys.tables.

    I will greatly appreciate your quick response

    Thanks.

    1. Hi Bhaktapur,

      If you are redoing all tables, you could just do a backup of the source database and restore the target database :-).

      Alternatively, if you have to get all the tables you could do something like this:

      [string[]] $tables = Invoke-Sqlcmd -ServerInstance “MySQLInstance” -Database “MyDB” -Query “select SCHEMA_NAME(schema_id)+’.name’ as SchemaDotTable from sys.tables” -OutputAs DataRow | select -ExpandProperty SchemaDotTable

      The rest of the code in the “Usage” could remain. The above code is just making an array of all the tables by querying the source instance. You would have to replace with appropriate values for instance and database names.

      1. Hi Jana,
        Thank you for your reply. I did the following and it worked:
        $table = Invoke-SQLCmd -query “SELECT ‘dbo’+’.’ + name AS TableName FROM sys.tables WHERE type = ‘U’ AND name NOT IN (‘Calls’,’CallSubmissions’,’CallSubmissionEmails’,’sysdiagrams’)” -Server $sourceInstance -Database $sourceDB | foreach{$_.TableName}
        [string[]] $tables = @($table)…

        However, I noticed that the function drops the existing table and creates it in the destination server, but it does not create (copies) the table constraint. For example, in one of the table in source I have default constraint which did not get copied to destination.

        How can the function be tweaked to copy table constraints as well – mostly concerned with default constraint.

        Will greatly appreciate your help.

        Thank You.

      2. I have updated the function to include a new parameter CopyConstraints. Also, if you do not want the table to be recreated set the DropTargetTableIfExists appropriately. You can also tweak the other creation options (for example triggers) as necessary.

        Copy-SQLTable `
        -SourceInstance ‘SourceServer\Instance’ `
        -SourceDB ‘SourceDB’ `
        -DestInstance ‘TargetServer\Instance’ `
        -DestDB ‘TargetDB’ `
        -DropTargetTableIfExists: $true `
        -CopyIndexes: $true `
        -CopyConstraints: $true `
        -Tables @(‘dbo.Table_1’)

  6. Thanks!! its very nice script. Though it fulfils my requirement but as a curiosity if target table already exist and I don’t want to drop that instead rename the existing table with MMDDYYYY and then create the new table with same schema and copy the data. What should I change in script to achieve that?

    Thanks again for providing nice script

    1. Hi Takur, You are welcome. I would create a separate script that would first rename the qualifying tables and then run this script. However, I have created additional functionality in my script to skip existing tables if needed. Available on GitHub at:
      Hi Sharma, I have created an updated version of the script available on my GitHub at the below location. It has the row count information and other additional features.

      https://github.com/SQLJana/PowerShell/blob/c6cbe1d5b4f97026d1b00df59403dcd1ed367065/Copy-SQLTable.ps1

      1. Thanks for the new script. I do still has few question if you can help —
        Its not working for copy Constraints, not sure if I am doing something wrong but below is
        what I am using —
        [bool] $CopyConstraints = $true
        [bool] $copyIndexes = $true
        .
        .
        .
        -CopyIndexes: $copyIndexes `
        -CopyConstraints: $true `

        Also would like to check if there is any option to show the progress of copied row live (like we see in export import wizard) Am I asking too much

      2. Sorry for the late response Thakur. Your call looks good to me. Are you referring to Check Constraints or other constraints like Foreign Key / PK constraints? The parameter you are referring to is just for Check Constraints if I am not mistaken.

        As for the row level progress, I will try to add it in the next iteration (not sure when 🙂 ).

  7. Hi Jana,
    Actually I did test for both, check constraint as well as FK / PK constraints and in both case its not scripted and applied to target tables.

    1. ok Thakur, you win! Thanks to your persistence, I have fixed the issue with Constraints and also added the copy progress for rows both as Verbose output and as a progress bar. Please do note that when CopyConstraints is set, Check constraints, PK and FK will all be copied over. If the FK tables dont exist already, you will see errors.

      As usual, the script is updated on GitHub (link referenced at the top of this post) and the code on this blog post continues to be the old one.

  8. Thank you Jana. It works perfectly. Please let me know when you done with modification of renaming tables instead dropping if exist. Asking that to get help in scenarios where we have to modify table data and need to have backup in existing database and server.

    1. Great. I am glad it works as expected. The table rename is too specific to be included as part of tbis. You should create a separate routine to rename ahead of the copy.

      1. Hi Jana,

        I do have small request other than this. Do you have any powershell script to stop and restart SQL services dynamically? I mean it should check for default or named instances and get the services names dynamically and stop and start?

        Thank you
        Thakur

  9. Hi Jana, How to use this script to copy tables while renaming them at the same time? I have hundreds of tables I need to copy in the same database, i need to add “_deleted” suffix to table name and copy data from existing tables to new renamed tables.

  10. Hi @SQLJANA,

    Thanks for the provided script. It worked for me.
    However, I was wondering how the script could be tweaked to become a MySQL version, where the source tables are just saved or located in a folder like: C:\path\to\table\*.sql. And then, utilizing the script to upload/import those tables to the programData – C:\ProgramData\MySQL\MySQL Server 8.0\Data\xxx.
    With that, it would save the stress of using the MySQL workbench UI to import the individual tables respectively

  11. Hi Jana

    while copying source table where clause is not working

    like

    select * from tablename with(nolock) where date = ‘date’

    1. Sorry Bala. Currently the way the function is designed, it reads the meta-data of the source table to create it on the target if needed with constraints, indexes etc. You should consider using dbatools – dbatools docs | Copy-DbaDbTableData

      From the dbatools documentation:

      Example: 7
      C:> $params = @{

      SqlInstance = ‘server1’
      Destination = ‘server1’
      Database = ‘AdventureWorks2017’
      DestinationDatabase = ‘AdventureWorks2017’
      DestinationTable = ‘[AdventureWorks2017].[Person].[EmailPromotion]’
      BatchSize = 10000
      Table = ‘[OtherDb].[Person].[Person]’
      Query = “SELECT * FROM [OtherDb].[Person].[Person] where EmailPromotion = 1”
      }

      PS C:> Copy-DbaDbTableData @params
      Copies data returned from the query on server1 into the AdventureWorks2017 on server1, using a 3-part name for the DestinationTable parameter. Copy is processed in BatchSize of 10000 rows.
      See the Query param documentation for more details.

Leave a comment