RegisterXLL: Testing Excel's XLL Security Boundary

Excel blocked the file exactly where I expected it to. The XLL carried ZoneId=3, used an unsupported .dev extension, and failed through the normal add-in workflow. When I passed the same file to the documented Application.RegisterXLL method, Excel immediately loaded it without a warning and called xlAutoOpen.

That difference is what this research is about: two supported paths into Excel making different trust decisions about the same native code.

I found the behavior while testing whether a per-user file association and custom extension could change how simple extension-based controls treated my XLL shellcode loader. Excel opened but refused to run the add-in. That failure pushed me toward Excel Automation and the RegisterXLL method.

This research was performed in an isolated lab and disclosed to Microsoft as MSRC case VULN-184497. Microsoft closed the report without a product change.

What an XLL actually is

An XLL is a native Excel add-in: a Windows DLL that implements the interface Excel expects from an XLL code resource. Its one required callback is xlAutoOpen, which Excel invokes when it activates the add-in. Legitimate developers use that interface to add high-performance functions and commands to Excel. Loading an untrusted XLL therefore means loading native code inside the Excel process.

Microsoft’s XLL development documentation describes the lifecycle directly: Excel loads the library, calls its exported add-in interface functions, and invokes xlAutoOpen when the add-in is activated.

XLLs are not VBA macros. Macro policy applies when a workbook uses VBA to call RegisterXLL, but it does not govern an external process automating Excel through COM.

The protections Microsoft added

Extension enforcement

Microsoft’s August 2022 security update added an extension check for XLL add-ins. The documented valid extensions are .xll and .dll. Microsoft stated that an add-in with an incorrect or missing extension would be blocked, and the temporary EnforceXllExtension compatibility switch was scheduled for removal in January 2023.

That behavior is documented in KB5017166.

Blocking XLLs from the internet

In March 2023, Microsoft rolled out another default change to Excel for Windows: XLL add-ins obtained from the internet would be blocked rather than presented with an easy enable button. The decision is based on the file’s zone information, commonly called Mark of the Web or MOTW.

Windows typically stores MOTW in the Zone.Identifier alternate data stream. A file downloaded from the internet commonly receives ZoneId=3.

Microsoft describes the intended behavior in Block untrusted XLL add-ins by default and its XLL support guidance.

Both controls worked through Excel’s interactive paths in my lab. Excel blocked an XLL carrying MOTW and rejected the same valid XLL after I gave it a custom extension. I wanted to know whether Excel made those checks when it loaded native code or earlier in the UI workflow.

A second path into Excel

Excel exposes a documented Automation method named Application.RegisterXLL. It takes a file name, loads the XLL code resource, and registers its functions and commands. The return value reports whether the load succeeded. The method call is also the execution point: Excel loads the library and calls xlAutoOpen immediately rather than waiting for a restart or later add-in activation.

The Microsoft API reference does not describe how the method interacts with MOTW, the BlockXLLFromInternet policy, extension enforcement, signatures, or user approval.

This was enough to reproduce the behavior:

$excel = New-Object -ComObject Excel.Application
$excel.Visible = $false
$loaded = $excel.RegisterXLL("C:\Lab\benign-test.dev")

Setting Visible to False keeps the Automation instance from opening a normal Excel window for the user. It does not hide the EXCEL.EXE process, the XLL image load, or the surrounding Automation telemetry.

Test environment

I retested the final comparison in July 2026 against the latest Excel build available to my lab at that time. I did not preserve the numeric build identifier, which limits exact reproduction and is worth stating directly. The result should be read as behavior observed in this environment, not as a claim about every Excel channel or historical build.

Component	Test condition
Endpoint	Isolated Windows lab endpoint
Excel	Latest build available when tested in July 2026; numeric build not preserved
Sample architecture	Matched the installed Excel architecture
Control sample	Valid `.xll` loaded through Excel’s interactive add-in path
Renamed sample	Byte-identical XLL code resource renamed with a `.dev` extension
Delivery	Sample transferred in a regular ZIP archive downloaded to the Windows lab VM
File provenance	`Zone.Identifier` present and manually verified as `ZoneId=3`
Trust configuration	Sample remained outside Excel trusted locations and was not approved as an add-in

I ran the interactive and Automation comparisons on the same endpoint. The direct RegisterXLL test did not use an XLAM workbook, VBA macro, custom file association, or registry change.

I used a benign XLL whose xlAutoOpen produced an observable local artifact. The .dev sample contained the exact code used by the .xll control sample. Only the file name changed.

Delivery mechanics are outside the scope of this research, but the download path blocked direct XLL delivery. I downloaded a regular ZIP archive to the Windows lab VM and extracted the sample there. After extraction, the file still carried its internet-zone provenance.

Before invoking the method, I verified that the alternate data stream was still present:

Get-Item "C:\Lab\benign-test.dev" -Stream Zone.Identifier
Get-Content "C:\Lab\benign-test.dev" -Stream Zone.Identifier

The stream contained ZoneId=3. I left it there, kept the file outside Excel’s trusted locations, and did not rename it back to .xll.

RegisterXLL returned True. Excel loaded the image and called xlAutoOpen without displaying the security notice I had seen through the interactive path.

Test	Interactive Excel path	`Application.RegisterXLL`
Valid `.xll`, no MOTW	Loads after the expected add-in workflow	Loads
Valid `.xll`, `ZoneId=3`	Blocked by the internet-origin control	Loaded in my test environment
Valid XLL code renamed to `.dev`, `ZoneId=3`	Rejected	Loaded in my test environment

The custom extension is not a parser trick. The file still has to contain valid native code that implements the XLL interface, and its architecture must match Excel. RegisterXLL treated the supplied path as an XLL code resource based on its contents without applying the extension and origin decisions made by the interactive path.

The minimal case

My first working chain used a per-user file association to route a custom extension through wscript.exe and into RegisterXLL. It made for a useful double-click demo, but it also distracted from the simpler finding.

The direct call needs none of that setup. It does not need HKCU file-association or ProgID changes, custom extension registration, an XLAM loader, a signed Office document, VBA, or an add-in persistence entry. The Zone.Identifier stream can stay on the file.

I also tested the direct call through VBScript. This version looks for benign-test.dev in the same directory as the script instead of requiring the payload path as an argument:

Set fso = CreateObject("Scripting.FileSystemObject")
scriptDir = fso.GetParentFolderName(WScript.ScriptFullName)
devPath = fso.BuildPath(scriptDir, "benign-test.dev")

If Not fso.FileExists(devPath) Then
    WScript.Quit 1
End If

Set xl = CreateObject("Excel.Application")
xl.Visible = False
xl.RegisterXLL devPath

The VBScript is another way to invoke the same Automation method, not a separate bypass.

The direct method still requires a way to invoke COM on the endpoint. This makes it an execution primitive available after script or code execution, not a one-file phishing chain by itself.

Prior research and novelty boundary

RegisterXLL is old, and so is using it for code execution. The part I could not find in earlier public research was its behavior against the controls Microsoft added later.

In 2020, MDSec documented RegisterXLL as an Excel DCOM lateral-movement primitive and explicitly noted that the file extension was irrelevant. Their example loaded XLL code from a file named excel.log over a UNC path. That work predates Microsoft’s August 2022 extension enforcement.

Optiv also documented local RegisterXLL abuse in 2021 while testing Office-related attack-surface-reduction rules. Their example created an Excel Automation object and registered a local .xll payload.

Outflank reached RegisterXLL through a different trust chain in 2021. They modified data consumed by Microsoft’s signed Analysis ToolPak XLAM so its signed VBA code would load an attacker-controlled, unsigned XLL. Their technique depended on the signed XLAM loader and predates the 2023 MOTW-based XLL block.

My test differs in four ways:

Invoke Application.RegisterXLL without an XLAM or workbook intermediary
Keep ZoneId=3 attached to the XLL code resource
Use a non-XLL file extension
Compare the result with the blocked interactive Excel paths on the same system

I found prior work on the execution primitive and on extension-agnostic loading. I did not find an earlier public test showing a direct RegisterXLL call loading XLL code across Microsoft’s later MOTW block and extension enforcement. That is as far as I am willing to take the novelty claim. I found a post-hardening control gap, not a new execution primitive.

Detection opportunities

Skipping the Excel warning does not make this quiet. Even the short version leaves process, script, file, and image-load telemetry.

Process ancestry

Watch for script engines, command shells, remote management processes, or unusual applications creating an Excel Automation instance. Depending on how COM activation occurs, EXCEL.EXE may appear under the calling process or under svchost.exe hosting DCOM Launcher.

An unexpected EXCEL.EXE instance with no workbook activity is worth correlating with nearby PowerShell, VBScript, JScript, WMI, or remote service events.

Image loads

An unusual native image loaded into EXCEL.EXE is the strongest endpoint signal here. Collecting every DLL load can be expensive. Start with images loaded from user-writable paths, temporary directories, downloads, network shares, and files with unexpected extensions.

Sysmon Event ID 7 records image loads when that event type is enabled. Microsoft notes that image-load monitoring is disabled by default and should be configured selectively because of its volume.

File provenance

Preserve and inspect Zone.Identifier streams during triage. A nonstandard file loaded as an image by Excel while retaining ZoneId=3 is a strong combination. Sysmon Event ID 15 can record creation of named streams, including the Zone.Identifier stream browsers attach to downloaded files.

Automation and script telemetry

PowerShell script-block logging can expose both creation of Excel.Application and the RegisterXLL method name. VBScript and .NET implementations require different collection, so a detection based only on PowerShell will miss them.

For the optional file-association chain, monitor writes under HKCU\Software\Classes that create or change shell\open\command handlers, especially when the command points to wscript.exe, cscript.exe, mshta.exe, PowerShell, or another script host. Sysmon Event IDs 12 and 13 cover registry key creation and value changes respectively.

Hardening

Keep BlockXLLFromInternet enabled. It removes the most direct user-driven path and raises the cost of commodity XLL delivery.

The ZIP archive used in my test also shows why download filtering cannot serve as the final native-code boundary.

The more complete defensive boundary is application control over native code loaded into Office processes. App Control for Business with user-mode code integrity can restrict which user-mode binaries and libraries are allowed to run. AppLocker can also enforce DLL rules, although Microsoft warns that DLL enforcement requires careful allow-rule design and testing.

I would start with the following controls:

Block or constrain Windows Script Host where business workflows do not require it
Inspect supported archive formats for blocked file types and preserve zone metadata during extraction
Constrain PowerShell and monitor Office Automation from script hosts
Alert on unsigned or unapproved libraries loaded into EXCEL.EXE from user-writable locations
Use trusted publishers and managed deployment paths for legitimate XLL add-ins
Inventory business-critical XLLs before enforcing DLL allowlisting
Correlate Excel image loads with MOTW, signer, prevalence, path, and parent-process data

Do not assume an ASR rule closes this path without testing the same Excel build and invocation method. Microsoft’s Block Office applications from creating executable content rule, for example, focuses on executable content written by Office. This technique asks Excel to load a native library that is already on disk.

Responsible disclosure

I reported the behavior to Microsoft on April 25, 2026 as VULN-184497. Microsoft closed the case without a product change and treated the behavior as consistent with the intended capability of RegisterXLL.

I understand why RegisterXLL exists. Legitimate add-in installers and automation depend on it. What bothers me is that Excel reached a different trust decision based on which supported path loaded the same code resource. Controls described as blocking untrusted XLL add-ins should be tested across those paths.

Where this leaves us

Excel is supposed to load native add-ins. The problem I observed is narrower: the same XLL code resource received different trust decisions depending on how it reached Excel. The interactive paths enforced origin and extension checks in my lab, while the documented Automation method loaded the file directly.

Red teams should test RegisterXLL against the exact Office channel, build, endpoint policy, and application-control posture in scope. Defenders should not treat a shell or UI file-type block as a native module-load policy. Trust needs to be enforced when code enters the process, with provenance and behavioral telemetry covering the paths that remain.

Check back later for the GitHub link and supporting code.