Azure Monitor Agent: The future is not here (yet).

W365 Resolution, August 2024:

I am happy to announce that as of late August 2024, Azure Monitor does support Windows 365 Cloud PCs.

Source: https://learn.microsoft.com/en-us/windows-365/enterprise/whats-new#week-of-august-26-2024-service-release-2408

For future reference, I don’t believe I was ever contacted to let me know that my feature request had an ETA or was completed. Either way, I am just happy that it is done, but it’s odd to me that I did not receive any notification of this given I was the person who brought it to attention as far as I know. I suppose I’ll have to keep this in mind for any other feature requests.

I know there were some other bumps and concerns I had with the AMA at the time of writing, but frankly it’s been around a year and a half since I first played with AMA and I’m sure much has changed since then. I hope to get some time to play with its current state and to evaluate what it can do, as I’m sure it’s improved in many ways over that time.

Hopefully, I will have a new article on the topic soon!

Original article:

Pretty bold title, huh?

If you hadn’t guessed already, I have some pretty strong feelings to express here. In short, this article is going to be discussing why the old Log Analytics agent was lacking and, explain how close the Azure Monitor Agent (as of 5/30/23) is to becoming an incredible solution – but why it isn’t yet. This is a prelude to my second revision of the Log Analytics learning series, as why this solution isn’t a real option is a large part of that intro.

I hope I don’t come off too harshly here. In all reality, I was simply excited for this solution and what it could mean for the community, only to have it crumble to dust in my hands. That’s no fun and, unfortunately, the reasons why are real forehead smackers from my perspective.

Before we get too far, I want to add that the information I am sharing here is both straight from Microsoft documentation (which I will link to) as well as discussion with support regarding AMA and Windows 365 (which I will do my best to call out when information came from them). That said, some of it is my opinion.

I will be covering three main topics.

• The Log Analytics Agent
• The Azure Monitor Agent (AMA)
  The good, the bad, and the ugly.
• The Azure Arc Agent
• The Future (Updated 6/13/23)
  
  

Log Analytics Agent:

The Log Analytics Agent is the old solution as far as dedicated client OS agents go for Log Analytics. It will be retired August 31, 2024, and Microsoft recommends you move over to the AMA.

This article isn’t meant to cover the outgoing solution but, I will quickly say that this tool can’t… (Some of these are straight out of the limitations here or elsewhere in the same document)

• Can’t send data to Azure Monitor Metrics, Azure Storage, or Azure Event Hubs.
• Difficult to configure unique monitoring definitions for individual agents.
• Difficult to manage at scale because each virtual machine has a unique configuration.
• Can’t collect Windows Event Logs from the Security log. I believe other more niche logs were affected by this too.
• Can’t filter what Windows Event Logs it collects – if you point it to a log it will just pull all which is costly.
• No customization to static data elements or ability to collect files for items like device/application inventory.

If you really want a mostly Indepth comparison, look here.

The Azure Monitor Agent:

As mentioned, the AMA is what is going to replace the Log Analytics Agent. In fairness, that retirement isn’t for over another year so there is plenty of time to improve the AMA. And, for what it’s worth, there are many good things about it today.

AMA – The Good:

Unlike the Log Analytics Agent, the AMA can…

• Collect ALL Windows Event Log events, including the Security log.
• Filter event logs using XPath queries – this lets you control what events you do and don’t pick up more granualry
• Uses DCRs (Data Collection Rules) which can further parse things like events and further filter what isn’t possible via XPath alone. You don’t pay to ingest events that get tossed by the DCR.

See here: Collect events and performance counters from virtual machines with Azure Monitor Agent – Azure Monitor | Microsoft Learn

These three alone could/would have spelled the end of PowerShell based log analytics for Windows events, especially as this data is also live. Indeed, I have some tests already going and running via this solution.

Additionally, the AMA has static text file-based log collection allowing you to ingest the raw contents of a file. I attempted to get this to pick up device inventory information by writing it to a file however, every time I attempted to even setup their example with their example data it brought the AMA agent on the machine to it’s knees and knocked out all collection that was already working on the device. I wanted to continue to look into this but, further on we will see what killed my momentum outright.

AMA also supports Event Hubs (I believe) which is another nail in the coffin for PowerShell based Log Analytics, at least in terms of how the data is uploaded.

AMA – The Bad:

The biggest bumps (not show stopers) to this solution would be the large effort required to restructure and engineer the workbooks and queries as the data is not ingested with the same level of detail (for events), nor in the same format, nor in the same tables. Very annoyingly in my opinion, when you create a rule to ingest events, you don’t point it to a table but rather a Log Analytics workspace where it then dumps all “event” type data into a single Table. No more separating Windows Update events to table X, and Power events to table Y. I was hoping this would be controllable per DCR and still hope that will be a feature one day. Still, we have event IDs and can build queries to narrow those down and display only the appropriate data.

AMA – The Ugly:

There are two major roadblocks to this solution. Both of them somehow manage to revolve around the fact the Azure Monitor Agent was built really tight knit with Azure VMs. At first glance that doesn’t sound like an unreasonable or bad thing but, allow me to detail it a little and open your eyes.

1. Deploying the AMA to physical devices

Set up the Azure Monitor agent on Windows client devices – Azure Monitor | Microsoft Learn

Our first crash comes when we talk about the current state of deployment to physical devices. I admit it’s been about a month since I played with this, to many other advancements have had my attention, so my memory is rough. That said, quickly looking through the documentation I am recalling what a nightmare this is.

First, just to share the knowledge, deploying to physical devices requires the use of an MSI instead of the PowerShell commandlets used by Azure VMs. To be clear, that’s not an issue at all to me. Installing an MSI is simple enough.

However, the limitations section quickly highlights some fun problems.

• The Data Collection rules can only target the Azure AD tenant scope, i.e. all DCRs associated to the tenant (via Monitored Object) will apply to all Windows client machines within that tenant with the agent installed using this client installer. Granular targeting using DCRs is not supported for Windows client devices yet
  
  Or in other words, when you configure a DCR to collect XYZ, you then have to scope that to ALL physical devices. You currently cannot scope it to half, or a group, or anything like that. It is all or nothing. More on this in a moment.
  
  
• The agent installed using the Windows client installer is designed mainly for Windows desktops or workstations that are always connected. While the agent can be installed via this method on laptops, it is not optimized for battery consumption and network limitations on a laptop.
  
  This isn’t really a show-stopper, it’s not going to stop you from putting it on laptops, but I don’t like it. Especially because this means Microsoft just doesn’t have a recommended mobile solution as best I can find. Sure, they will point you to Intune (and do in a couple articles), but there is a reason I have elaborate 11 parts series on how to how to use Log Analytics to achieve the necessities Intune can’t – Intune is not a realistic solution for much of this data, certainly not at all for Windows Event Log events.

Circling back, what’s this all about with needing to scope a DCR to all or nothing as far as physical devices? Well, this is because (as of current) to scope a DCR to physical devices you have to create a monitored object in Azure. You then scope the DCRs (Data Collection Rules – the things you build to tell it what to collect) to the monitored object. Making that more fun is that you must do most of this via PowerShell. I obviously don’t hate PowerShell but it wasn’t exactly the most friendly process to go through.

You might glance at that article and have the same thought I did – why not just make more than one monitored object and associate devices to one or the other to control assignment? Or, only associate the devices you want to monitor to the monitored object.

Simple, it’s because you don’t do either of those things. When you create the Monitored Object, it automatically picks up all your devices that have the agent installed. You never run a command to associate device X to Monitored Object Y. Nor can you have more than one Monitored Object. Fun!

2. Deploying the AMA to Windows 365 Devices

Okay, so physical devices are a bit of a mess right now, but there is plenty of time to improve that! So, in the meantime, surely we can really push our virtual azure devices further, especially given how “tight knit” AMA is with Azure – right?

Nope!

Sure, you can make AMA work great with AVD (I imagine), but it entirely DOES NOT SUPPORT Windows 365 from an architectural perspective. Why? How?!?

Since Windows 365 devices are not physical devices, you are supposed to deploy the Agent Extension. There are two main ways to do this.

First – You can do this via the DCR itself by simply checking the device under the resources tab – Azure will automatically show you the compatible devices. But if you try this, your Cloud PCs won’t show up…

Second – You can also install it via PowerShell and use that to tie devices to the DCRs. Kind of a pain, now we have to think about deploying scripts or something like that, but not the end of the world. So, let’s take a look at one of the commands.

Set-AzVMExtension -Name AzureMonitorWindowsAgent -ExtensionType AzureMonitorWindowsAgent -Publisher Microsoft.Azure.Monitor -ResourceGroupName <resource-group-name> -VMName <virtual-machine-name> -Location <location> -TypeHandlerVersion <version-number> -EnableAutomaticUpgrade $true

If you know anything about Windows 365, you should already see the problem. If not, let me call it out a bit more clearly. What resource group in your Azure Tenant do your Cloud PCs live?

The answer is they don’t. Yes, Windows 365 devices are Azure joined devices. Yes, they are Azure VMs. But, they don’t live in your tenant.

As shown and explained clearly in the architecture documentation, Windows 365 devices live inside Microsoft’s Azure tenant.

For example, I primarily deal with this kind of setup.

As such, I can indeed go find the Cloud PCs network adapter inside our Tenant – and that’s it. The actual VM itself lives inside Microsoft’s tenant.

What about that MSI we used on physical devices? Nope. Microsoft has built Azure VMs such that they know they are Azure VMs. Attempts to run the MSI on those devices gives you a nice little error that “Installing Azure Monitor Agent using the Windows Installer is not supported on Azure VM. Use VM Extension instead.” – which as we just explained, you can’t do.

As such, you cannot install the AMA on Windows 365 devices. Support has confirmed this, the two are simply designed with opposite ideas in mind. AMA is designed to hook into the devices Azure object. Windows 365 devices know they are Azure VMs and thus only want to be hooked into via Azure, and Windows 365 devices were specifically designed to not have their objects inside your tenant so you can’t reach that object. Fun!

Adding on to this, when I asked the AMA team about Windows 365 support, their response was a collective “What’s Windows 365?” – a response my peers are extremely tired of hearing from Microsoft support. After much back and forth to get on the same page, and them evaluating that same architecture diagram and explanation, they confirmed this simply isn’t possible. “The Microsoft 366 Cloud PCs are not officially supported for use with either the Legacy Log Analytics agent or the Azure Monitor Agent…. …And, not able to be managed from Azure data plane perspective to add the needed extensions (for AMA).” They went on to add that we could manually get the legacy agent going but that it wouldn’t be a supported scenario – not that I would want to use that anyways.

As such, I am moving for a DCR (Design Change Request) because it is just unfathomable to me that Microsoft’s big and hard-pushed solution for large, cloud, end-user VM computing is simply not at all supported by any of their in-house data collection tools. I did ask about alternate solutions with the AMA team and none were found.

AMA Conclusion:

Given the state of physical devices is rough, there would be a lot of re-engineering required, and the support for Cloud PCs doesn’t exist, using the AMA simply does not make sense to me as of current.

At best, I would have to re-engineer my solutions, deploy it to all physical devices, and then suffer through the mess that is having half my data (physical devices) in one place, and half of it (Windows 365) in another. That is simply not realistic.

But wait, you forgot about Arc! (Azure Connected Machine Agent)

I was talking about this problem in a few places online as I progressed through it originally and quite a few folks recommended Azure Arc. I too thought this was a good solution at first glance. Despite being labeled largely as a server solution, it does support the Windows 10/11 client OSs.

And if you read the OS guide section just under that…

“The Azure Arc service and Azure Connected Machine Agent are supported on Windows 10 and 11 client operating systems only when using those computers in a server-like environment. That is, the computer should always be – Connected to the internet – Connected to a power source – Powered on.”

… Hey, that sounds PERFECT for an always running virtual machine like a cloud PC! Maybe we can use this!

Unfortunately, scroll down just a little further to the section Short-lived servers and virtual desktop infrastructure and you will find that…

“Microsoft doesn’t recommend running Azure Arc on… …virtual desktop infrastructure (VDI) VMs. Azure Arc is designed for long-term management of servers and isn’t optimized for scenarios where you are regularly creating and deleting servers.”

…Which, any decently sized organizations are probably creating and destroying Windows 365 devices (also include re-provisions) on a daily basis.

And if that’s not enough of a nail in the coffin, you will find that the same scenario plays out where there are Azure and Non-Azure based options. Windows 365 knows it’s an Azure VM, and thus wants you to use similar commands which ask for a resource group which again, you don’t/can’t have because these VMs are not inside your tenant.

I did discuss this option with Microsoft support just for fun, and they too agreed that it wouldn’t be recommended, nor would it even work for that same reason. Fun!

The future (June 2023):

As of roughly June 2023, I can confirm that AMA support for W365 is on its way to the Microsoft Product Group as a feature request. Whether that request will be accepted at all, let alone when it would be completed, is not something I have any way of knowing at this time. Hopefully, one day, AMA will replace the existing PowerShell solution.

Update March 2024:

It’s now March of 2024 and I wanted to share some news on this. In touching base with the product owners, this is (to my understanding) now an official goal. “This” being adding Windows 365 support to the Azure Monitor Agent. However, 8-9 months later from my original feature request, there is still no ETA. While I am sure we will eventually get this solution, I would not hold your breath. This is saddening news as I greatly wish I could begin to move to and develop with the AMA however, that’s simply not possible as the Windows 365 solution is too important and the inability to collect data from those devices is unacceptable.

Conclusion:

At the end of the day, I think AMA has a bright future, but I can’t help but wonder how it isn’t labeled Preview because that future isn’t here yet. I feel like I have seen plenty of more functional solutions that for some reason retain that marker for years. Meanwhile, this solution has several documented future-hope items like Physical device targeting, perhaps even mobile device support, as well as the outright lack of support for other major Microsoft Azure solutions, and yet seems to have the full production green light. I just don’t get it.

Disclaimer:

The following is the disclaimer that applies to all scripts, functions, one-liners, setup examples, documentation, etc. This disclaimer supersedes any disclaimer included in any script, function, one-liner, article, post, etc.

You running this script/function or following the setup example(s) means you will not blame the author(s) if this breaks your stuff. This script/function/setup-example is provided AS IS without warranty of any kind. Author(s) disclaim all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall author(s) be held liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the script or documentation. Neither this script/function/example/documentation, nor any part of it other than those parts that are explicitly copied from others, may be republished without author(s) express written permission. Author(s) retain the right to alter this disclaimer at any time. 

It is entirely up to you and/or your business to understand and evaluate the full direct and indirect consequences of using one of these examples or following this documentation.

The latest version of this disclaimer can be found at: https://azuretothemax.net/disclaimer/