Versioning config data in D365/Power Apps - beyond the CMT - part 2

In part 1 of this series , we learned how to avoid some key limitations by using PowerShell instead of the Configuration Migration Tool. We created a simple system to import and export records to JSON files.

However, there are still some more important limitations in the CMT which are still present in that solution.

Oh no! More limitations?

Clive Oldridge kindly pointed out the main limitation in a comment:

This is true! both the CMT and our solution so far are additive only - they will only ever create or update records, but not delete them when they are no longer part of the data we a importing. This means our target environment will end up with extra records left behind in the “heap” Clive is describing and potentially causing mischief.

Where we left off

Before we work out how to resolve this, as a reminder, here’s an example export script for two tables:

$ErrorActionPreference="Stop"

install-module -scope CurrentUser Rnwood.Dataverse.Data.Powershell
$connection = get-dataverseconnection -url https://org60220130.crm11.dynamics.com/ -interactive

Get-DataverseRecord -connection $connection -TableName myorg_categoryl1 | Set-DataverseRecordsFolder data/myorg_categoryl1

Get-DataverseRecord -connection $connection -TableName myorg_categoryl2 | Set-DataverseRecordsFolder data/myorg_categoryl2

And here’s the matching import script assuming myorg_categoryl2 records have a relationship to depend on myorg_categoryl1 records:

$ErrorActionPreference="Stop"

install-module -scope CurrentUser Rnwood.Dataverse.Data.Powershell
$connection = get-dataverseconnection -url https://org60220130.crm11.dynamics.com/ -interactive

# Import data to each table in dependency order

Get-DataverseRecordsFolder data\myorg_categoryl1 | Set-DataverseRecord -Connection $connection -Verbose

Get-DataverseRecordsFolder data\myorg_categoryl2 | Set-DataverseRecord -Connection $connection -Verbose

Our strategy

1. Whenever we export the data, we will make our script compare the records that were exported with the last set that was exported. Any records we don’t find must have been deleted.

2. We will log the deletions by moving the relevant files in a deletions subfolder.

3. When we import the data, we will use the files in deletions to delete any matching records (if they still exist).

Logging deletions in the export script

We just need to add -withdeletions to our Set-DataverseRecordsFolder commands to trigger the magic comparison.

Get-DataverseRecord -connection $connection -TableName myorg_categoryl1 | 
   Set-DataverseRecordsFolder data/myorg_categoryl1 -withdeletions

Get-DataverseRecord -connection $connection -TableName myorg_categoryl2 |
   Set-DataverseRecordsFolder data/myorg_categoryl2 -withdeletions

Now whenever this is run, you’ll see the deletions subfolder of data/myorg_category1 updated based on any records that have been deleted in your source when compared to the files before it ran.

Processing deletions in the import script

Now we need to update our import script to make it process the deletions:

# Import data to each table in dependency order

Get-DataverseRecordsFolder data\myorg_categoryl1 | 
   Set-DataverseRecord -Connection $connection -Verbose

Get-DataverseRecordsFolder data\myorg_categoryl2 | 
   Set-DataverseRecord -Connection $connection -Verbose

# Now process the deletions in reverse dependency order

Get-DataverseRecordsFolder data\myorg_categoryl2 -deletions | 
   Remove-DataverseRecord -Connection $c -Verbose -IfExists

Get-DataverseRecordsFolder data\myorg_categoryl1 -deletions | 
   Remove-DataverseRecord -Connection $c -Verbose -IfExists

We need to add another section to the script with a line for each required table:

• The -deletions makes Get-DataverseRecordsFolder read just the deletions.

• These are then piped using |to Remove-DataverseRecord to delete them. The -IfExists there ensures that any records that never or no longer exist (like when this is run again in future) don’t cause an error.

• We need to put the lines in reverse dependency order. This makes sure that myorg_categoryl2 records that might reference the l1 records will be deleted first. (If it were the other way around, the remaining l2 records might block the deletion of l1 records that need to be deleted)

Other strategies

This isn’t the only strategy for identifying which records should be deleted. We could instead just list the records in the target environment that don’t match the records we are importing. Now you’re all PS experts, I’m going to leave this up to you if this is what you want.

That’s all folks!

What other limitations and issues can you think of? Share your questions and experience over at LinkedIn!