Move attachments from file type field from one environment to another in dynamics 365

How do we move attachments in the file type field from one environment to another?

Limitations:
There is no support in the configuration migration tool to move attachments in the file type field from one Dynamics 365 environment to another.

Proposed Solution :

Writing a console app that fetches the attachments and moves them to target Dynamics 365 environment

Idea :

A console app that can query the attachments from the source environment and then uploads them to the target environment

Ability to have filters while querying the attachments so not all are imported into the target instance.

Scenario :
In this example, static documents (pdf) are stored on the entity itself in a file type field that can be reused across other entities, with no need to retrieve attachments from SharePoint / blob, they form the configuration data in the system which needs to be moved as part of deployments.

console app code :

credentials are stored in app.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.2" />
    </startup>
  <connectionStrings>
    <add name="SourceConnection" connectionString="AuthType=ClientSecret;Url=https://org1.crm4.dynamics.com;ClientId=#clientid#;ClientSecret=#clientsecret#" />
    <add name="TargetConnection" connectionString="AuthType=ClientSecret;Url=https://org2.crm4.dynamics.com;ClientId=#clientid#;ClientSecret=#clientsecret#" />
  </connectionStrings>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Runtime.CompilerServices.Unsafe" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-6.0.0.0" newVersion="6.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Text.Json" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-6.0.0.7" newVersion="6.0.0.7" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-13.0.0.0" newVersion="13.0.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

public class CopyAttachments
    {
        public static IOrganizationService _sourceservice;
        public static IOrganizationService _targetservice;
        public static OrganizationServiceContext _sourcecontext;
        public static OrganizationServiceContext _targetcontext;
        public static string sourceEntityname = "sh_masterdocument";
        public static string targetEntityname = "sh_masterdocument";
        public static string filetypefieldname = "sh_documentattachment";
        public static void Main(string[] args)
        {


            var sourceconnectionstring = ConfigurationManager.ConnectionStrings["SourceConnection"].ConnectionString;
            _sourceservice = GetServiceProxy(sourceconnectionstring);
            _sourcecontext = new OrganizationServiceContext(_sourceservice);

            var targetconnectionstring = ConfigurationManager.ConnectionStrings["TargetConnection"].ConnectionString;
            _targetservice = GetServiceProxy(targetconnectionstring);
            _targetcontext = new OrganizationServiceContext(_targetservice);

             GetAttachments(_sourceservice);
        }

        /// <summary>
        /// Sets up the connections
        /// </summary>
        /// <param name="connectionstring">takes the connection string object</param>
        /// <returns></returns>
       public static IOrganizationService GetServiceProxy(string connectionstring)
        {
            CrmServiceClient conn = new CrmServiceClient(connectionstring);
            if (conn.OrganizationServiceProxy != null)
                return conn.OrganizationServiceProxy;
            else
                return conn.OrganizationWebProxyClient;
        }

        /// <summary>
        /// Gets the source entity where fileattachment needs to be fetched 
        /// </summary>
        /// <param name="_sourceservice">takes the organization service object</param>
        private static void GetAttachments(IOrganizationService _sourceservice)
        {
            string queryMasterDocuments = GetAttachmentDataFromSourceEntity();

            EntityCollection result = _sourceservice.RetrieveMultiple(new FetchExpression(queryMasterDocuments));
            if (result.Entities.Count > 0)
            {
                foreach (var item in result.Entities)
                {
                    DownloadFile(_sourceservice, sourceEntityname, item.Id, filetypefieldname);
                }
            }
        }

        /// <summary>
        /// Takes the fetchxml query that will return the source entity where file type field is present
        /// </summary>
        /// <returns>retunrs the Guids of the source records</returns>
        private static string GetAttachmentDataFromSourceEntity()
        {
            return @"<fetch top='50'>
                      <entity name='sh_masterdocument'>
                      <attribute name='sh_masterdocumentid' />
                      <filter>
                      <condition attribute='sh_masterdocumentid' operator='ne' value='00000000-12e5-ed11-a7c7-000d3a227603' />
                      </filter>
                      </entity>
                      </fetch>";
        }
        /// <summary>
        /// Downloads the file from source entity, it is assumed the source and target master records have same guids.
        /// </summary>
        /// <param name="_sourceservice">takes the source service connection</param>
        /// <param name="entityName">takes the source entityname</param>
        /// <param name="recordGuid">takes the source recordGuid</param>
        /// <param name="fileAttributeName">takes the attributename of the file type field</param>
        private static void DownloadFile(IOrganizationService _sourceservice, string entityName, Guid recordGuid,
            string fileAttributeName)
        {
            var initializeFileBlocksDownloadRequest = new InitializeFileBlocksDownloadRequest
            {
                Target = new EntityReference(entityName, recordGuid),
                FileAttributeName = fileAttributeName
            };

            var initializeFileBlocksDownloadResponse = (InitializeFileBlocksDownloadResponse)
                _sourceservice.Execute(initializeFileBlocksDownloadRequest);

            DownloadBlockRequest downloadBlockRequest = new DownloadBlockRequest
            {
                FileContinuationToken = initializeFileBlocksDownloadResponse.FileContinuationToken
            };

            var downloadBlockResponse = (DownloadBlockResponse)_sourceservice.Execute(downloadBlockRequest);

            // store the file in file bytes
            byte[] fileBytes = downloadBlockResponse.Data;
            EntityReference entityReference = new EntityReference(targetEntityname, recordGuid);
            string fileMimeType = "application/pdf";
            bool uploadResult = UploadFile(entityReference, fileAttributeName, initializeFileBlocksDownloadResponse.FileName, _targetservice, fileBytes, fileMimeType);
            Console.WriteLine(initializeFileBlocksDownloadResponse.FileName, uploadResult);
        }

        /// <summary>
        /// uploads the file from source to target
        /// </summary>
        /// <param name="entityReference">target entity reference</param>
        /// <param name="attributeName">target attribute name of file type</param>
        /// <param name="fileName"> file name</param>
        /// <param name="service">target service connection</param>
        /// <param name="fileBytes">file data</param>
        /// <param name="mimeType">mime type of the file</param>
        /// <returns>returns true or false</returns>
        public static bool UploadFile(EntityReference entityReference, string attributeName, string fileName, IOrganizationService service, byte[] fileBytes, string mimeType)
        {
            var blockIds = new List<string>();
            var initializeFileUploadRequest = new InitializeFileBlocksUploadRequest
            {
                FileAttributeName = attributeName,
                Target = entityReference,
                FileName = fileName
            };
            var fileUploadResponse = (InitializeFileBlocksUploadResponse)service.Execute(initializeFileUploadRequest);
            var blockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(Guid.NewGuid().ToString()));
            blockIds.Add(blockId);

            var blockRequest = new UploadBlockRequest()
            {
                FileContinuationToken = fileUploadResponse.FileContinuationToken,
                BlockId = blockId,
                BlockData = fileBytes
            };

            _ = (UploadBlockResponse)service.Execute(blockRequest);

            var commitRequest = new CommitFileBlocksUploadRequest()
            {
                BlockList = blockIds.ToArray(),
                FileContinuationToken = fileUploadResponse.FileContinuationToken,
                FileName = initializeFileUploadRequest.FileName,
                MimeType = mimeType,
            };

            var commitFileBlocksUploadResponse =
                (CommitFileBlocksUploadResponse)service.Execute(commitRequest);
            return commitFileBlocksUploadResponse.FileId != Guid.Empty;
        }
    }

Improvements :

This can be part of the Azure pipelines and the .exe can be executed in the power shell task in the pipeline

The config can be dynamically replaced with environment-specific connection details

The connections in local can also be fetched using key vault without the need for app. config

References :

https://learn.microsoft.com/en-us/power-apps/developer/data-platform/file-column-data?tabs=sdk

Note:
The source code can be found in the link shared under references, this example only illustrates using what is available and fine-tuning it to a particular scenario and does not claim to have written the source code.

Related :

https://beyondd365.dev/update-attribute-metadata-file-size-of-file-type-field-in-dynamics-365