New Sync Event Log and the Intune Backoff/Retry Mechanisms

by Rudy Ooms | Aug 28, 2024 | Blog

Table of Contents

2

3

Earlier this week, I discussed the discovery of a new log file, AppWorkload.log, from the Intune Management Extension, highlighting its importance when we need to troubleshoot some failing app deployments.

In this blog, we’ll examine the latest developments by diving into another significant update: Microsoft’s new Sync event log and the backoff/retry feature. This new Sync log and Retry Intune Sync mechanism are designed to improve communication and sync with Intune, especially in environments with network issues!

Introduction to the new Retry Schedule

In almost every environment, devices need to sync periodically with Intune to receive the latest policies and report their status.

This communication/sync is pretty important for maintaining security, compliance, and functionality. However, network conditions are not always optimal. For example, connectivity issues, DNS failures, or service unavailability can disrupt the sync process.

To manage such disruptions gracefully, the OMA-DM (Open Mobile Alliance Device Management) client got a nice update. In the latest 27686.1000 Windows Canary Build, Microsoft implemented a “backoff” mechanism.

The Backoff/Retry in OMA-DM Client

This backoff mechanism is a controlled retry logic designed to handle temporary communication failures between the device and Intune. When the OMA-DM client encounters a failure, such as a DNS resolution error or a network timeout, it doesn’t immediately retry.

Instead, it waits before trying to reconnect, with the delay increasing after each failure. This prevents network overload and gives time to resolve issues. In large environments, where thousands of devices connect to Intune at once, this backoff mechanism helps maintain stability.

How the Backoff Mechanism is Triggered

The backoff mechanism kicks in when the OMA-DM client runs into certain errors, like a DNS resolution failure. This happens when the device can’t resolve the server’s hostname, often due to network issues or DNS problems. The moment a DNS error occurs, the corresponding error, 0x80072ee7, will be logged in the new Sync event log.

This 201 event will trigger the backoff process to manage the sync retries more effectively.

Detailed Examination of the BackOff Mechanism

Let’s examine how this back-off mechanism works, step by step.

Sync Operation Begins:

The Backoff process kicks in when the device attempts to sync with the server. This process is managed by the SendDataToServer function in the omadmclient.exe.
The SendDataToServer function is responsible for sending the device’s data to the server for synchronization.

DNS Error Occurs:

During the sync process, a DNS resolution failure can occur. This is a common network-related error represented by the error code 0x80072EE7 (ERROR_INTERNET_NAME_NOT_RESOLVED).

The SendDataToServer function detects this error and triggers the backoff mechanism.

Logging the DNS Error in the Sync Event lo:

When the DNS error is detected, it is logged in the new Sync event log (DeviceManagement-Enterprise-Diagnostics-ProviderSync). The event log captures the 0x80072ee7 error message and notes that a backoff retry will be attempted.

Registry Key Check

Before the retry scheduled task is created, it checks specific registry keys to determine the backoff time.

InitialBackoffTime

This registry key specifies the initial delay (in milliseconds) before the first retry attempt after detecting a failure. The backoff mechanism uses this value to determine how long to wait before trying to sync again.

MaxBackoffTime:

This registry key defines the maximum amount of time (in milliseconds) that the system will wait between retries. If multiple failures occur, the backoff time may increase exponentially up to this maximum value

Creating the Backoff Retry Task:

Once the DNS error is logged and the current oma-dm setings are checked, the system prepares to retry the sync operation. To manage this retry, a scheduled task is created using the Schedule Back Off Retry Task function.

This function creates a task in the Windows Task Scheduler (Microsoft\Windows\EnterpriseMgmt\SessionRetry.). This schedule will attempt to retry the sync after a specified delay.

we will notice that procmon also mentions the retry schedule for incomplete session getting created

the retry schedule created for incomplete session showing up in the task scheduler

Queueing the Retry Task:

The task is queued in the Windows Task Scheduler and waits for the specified backoff time before retrying the sync operation.

This task is critical to ensuring that the device does not continuously retry too quickly, which could overload the server or cause further issues.

Retry Execution:

After the backoff time has elapsed, the scheduled task executes the sync operation again.
During each retry, the registry may be updated with details such as Retry Delay, reflecting the delay calculated for the next retry. This allows the system to keep track of how long it has waited between retries.

The backoff times (InitialBackoffTime and MaxBackoffTime) guide how long the system waits before retrying. These values are accessed but not updated during the process
The retry will be logged inside the new Sync event log:

The Retry Task

· The Retry schedule created for incomplete session uses the Device Enroller with some specific parameters.

The task uses parameters such as the InitiationID (a GUID that uniquely identifies the session) to track the retry session.

Task Deletion:

Once the retry session succeeded and the sync is successful again, the queued retry scheduled task is automatically deleted to prevent it from running unnecessarily.

This cleanup ensures that only necessary tasks are present in the Task Scheduler, maintaining system efficiency.

Logging the Successful Sync in the new Sync Event log:

The new Sync event log is updated to reflect the successful completion of the sync operation.

This log entry indicates that the backoff mechanism worked as intended and that the device is now synchronized.

This detailed process ensures that devices can recover from network-related issues like DNS resolution failures and eventually complete their sync operations, maintaining compliance and up-to-date configurations. Let’s zoom into the registry details and the event log a bit more.

The backoff mechanism registry keys

The backoff mechanism relies heavily on registry settings to determine the behavior of retry attempts. These registry keys are stored under specific paths within the Windows Registry, which the OMA-DM client reads and writes to during its operation. Key registry values include:

InitialBackOffTime: This DWORD value sets the initial delay before the first retry attempt. The value is typically represented in milliseconds, with 0x00007530 equating to 30,000 milliseconds or 30 seconds. This time is crucial as it prevents immediate and repeated retry attempts that could further strain the network.
MaxBackOffTime: Another DWORD value that sets the upper limit on the backoff time. For instance, 0x0001d4c0 represents 120,000 milliseconds or 120 seconds. This cap ensures that even after multiple failed attempts, the retry mechanism doesn’t delay indefinitely, allowing for periodic retrying to check if the network condition has improved.
ConnRetryFreq: This value controls how frequently retries should occur after the initial backoff time, providing a balance between giving enough time for potential network recovery and ensuring that retries are not too sparse to cause unnecessary delays.
ServerLastAccessTime, ServerLastFailureTime, ServerLastSuccessTime: These string values store timestamps of the last successful server access, the last failed attempt, and the last successful attempt, respectively. These are critical for diagnostics and understanding the communication patterns between the client and server.

These values are usually managed programmatically by the OMA-DM client and provide the flexibility to fine-tune the retry behavior depending on specific deployment scenarios.

Let’s take a look at the new Sync Event log

Throughout the backoff and retry process, detailed logs are generated and can be found in the Windows Event Viewer under DeviceManagement-Enterprise-Diagnostics-Providersync.

Thew new sync event log showing all the information we need when we need to troubleshoot sync issues

Notably, the new Sync event log plays a critical role in improving visibility into sync activities, making it easier to distinguish between different types of events. These logs capture events such as:

The creation of a retry task
The scheduling of the retry session
The success or failure of retry attempts
Detailed information in the Sync event log, which provides insights into synchronization processes and related issues
The eventual outcome of the session, whether it succeeded or ultimately failed

By splitting the event logs, it becomes more evident which log is responsible for what, thereby allowing administrators to more effectively diagnose and resolve sync and policies issues. These logs can be pretty important for administrators who need to troubleshoot issues related to device management and understand why certain devices might be falling behind in policies!

Conclusion

The backoff feature within the OMA-DM client is a new mechanism designed to handle the realities of networked environments, where connections are not always stable or reliable (AKA totally broken).

By managing retry attempts and leveraging the Windows Task Scheduler, the OMA-DM client ensures that devices remain in communication with Intune without overwhelming the network or the Intune service!