HowTo: Secure a Custom Webhook for Azure Event Grid

As I wrote before, I’m playing around with the new Azure Event Grid lately. As I mentioned in my previous post, custom event publishers and subscribers hold a lot of promise, especially while we are still awaiting the bulk of Azure services to be hooked up to Event Grid.

AEG

But for custom publishers and subscribers to actually lift off, we need some way to authorize calls, both those from the publisher to Event Grid and those from Event Grid to the subscriber endpoint. Now the first is pretty well covered in the docs. But the call from Event Grid to the subscriber endpoint is not very well described at this point in time. It just mentions some initial validation sequence, which is supposed to prove ownership of the endpoint but in actuality just verifies that the endpoint is expecting to handle Event Grid events.

If this were the whole story, having an Event Grid subscriber endpoint would imply accepting unauthorized calls containing event payloads, meaning anyone with knowledge of the endpoint address can send bogus events your way – and since you have no way to tell authentic from fake events, you’d also be opened up for Denial of Service.

503-error

Luckily, a conversation with the Product Team quickly revealed that this is not the whole story. When you register a subscriber endpoint in Azure Event Grid, you can include a query string. This query string will be included in each and every call to your endpoint, so both the initial validation call and subsequent event notification calls. If you put some sort of key in there and then verify its presence in each incoming call, you’ve effectively locked out the Man In The Middle, and you just made a Denial of Service a lot harder.

EditEventSub

Furthermore, query strings that are added this way are not visible when enumerating Event Grid Subscribers in the portal, as an added layer of security.

ListEventSubs

I’ve updated my code samples to include a possible way to handle this for a ASP.NET Core WebAPI webhook.

Thanks to Dan Rosanova for clearing up how authorizing Event Grid calls to custom webhooks can be done.

Advertisements

Handling Custom Azure Event Grid Events

Lately I’ve been exploring the new possibilities opened up by Azure Event Grid, which was introduced last month.

Azure Event Grid is a fully managed platform for publishing and subscribing to event notifications. It is intended to ultimately encompass all Azure services as event publishers and/or subscribers but it also allows for custom, non-Azure participants. At this point, the following publishers and handlers are available:

event-grid-functional-model

For a little bit of background information on how AEG relates to the other event offerings on Azure, such and Event Hub or Service Bus, see this write-up by Saravana Kumar.

Long story short: it all looks very promising, but since most Azure services are yet to be hooked up to Event Grid the custom topic publishers and WebHook subscribers may hold the most promise for the short term.

So I tried my hand at actually getting that to work with a console app for publishing and a ASP.NET Core WebAPI for handling; code is available here.

In general, it’s relatively straightforward. The only thing that took me some time to get right is the handling of the validation process. The issue was that the documentation says that the validation request contains a header ‘Event-Key’ with a value of ‘Validation’. In actuality, this is ‘aeg-event-key’ with value ‘SubscriptionValidation’. Since my API is routing the validation request to a special action based on the header, this is pretty relevant. But, let’s keep in mind that Event Grid is in preview at this point, and the documentation is part of that status.

Microsoft Build 2017 – Day 2

The first day of Build 2017 was packed with exciting announcements and great content, such as the announcement of IoT Edge, new stuff on .NET Core and .NET Standard, a lot of work on AI and ML, and much more; see Peter’s write-up for some more detail (in Dutch). So let’s see if the second day can top this :).

Keynote

The Thursday keynote was pretty much centered around Windows 10, and more specifically the Windows 10 Fall Creators Update, that will include new features such as OneDrive Files On Demand and Windows Timeline. With OneDrive Files On Demand, your files in OneDrive will be available for you to work with, regardless of whether or not they are actually present on the device. If they’re not, they’ll be pulled from the Cloud when needed. And with Windows Timeline, your work on these documents or whatever else you have going on, will travel with you from device to device, including Android and iOS devices. All this is made possible by combining Cortana and the Microsoft Graph to track your data and your activities. And how about copying something on your PC, and pasting it on your iPhone? You can do that with the Cloud Powered Clipboard. Obviously a lot more was covered during the keynote, which is available at Channel 9.

Discussing All Things Azure

An interesting new session format this year is the Open Q&A, and for me the one with Mark Russinovich and Corey Sanders was a must-attend. They discussed upcoming features in Azure, such as the future possibility to deploy most of the storage options, including Azure SQL Database, in a private network to cut it off from direct Internet access; or the expansion of Azure AD to include identity information for compute objects such as VM’s with roughly the same capabilities as computer objects in on-prem AD’s; or upcoming support for encryption at rest for all storage services. They also touched upon the state of Cloud Services as pretty much the oldest service in the book: it’s not going anywhere as long as customers depend on it, but don’t expect a lot of new innovations coming to it anymore.

Serverless, Containers, Service Fabric

Of course, serverless architectures are also among the top-ranking topics during the conference, as well as container services and Service Fabric. Some highlights in the serverless computing area are the availability of the Azure Functions Runtime for on-prem deployment of Azure Functions, increased Visual Studio tooling support for Azure Functions an so on. Service Fabric becomes increasingly integrated with all sorts of related technologies, such as Azure Networking, API Management, containers and .NET Core 2.0.

Presenting As A Form Of Art

And in closing this post, a honorable mention goes to the Anders Hejlsberg session on TypeScript. He did one of those sessions where it’s just a very experienced presenter with a microphone and a code editor, and he showed off some very cool stuff that’s made possible just by layering a type system on top of JavaScript. It’s impossible for me to do it justice here, so just watch out for the session to appear on Channel 9, and treat yourself to an hour of entertainment.

HowTo: Perform “On Behalf Of” Calls Using Azure Active Directory

Probably every developer out there is familiar with the scenario of a UI-driven application (let’s say a web app) that needs to make calls to a backend service, and in quite a few of those scenarios the backend service needs to know which user is logged in in order to fullfill the request. And if you have ever been in charge of deciding on an implementation for this, you have been at the crossroads: do I go with the full-fledged impersonation / delegation solution, or do I conveniently decide that I trust the web app to make the correct calls?

If you’ve chosen the latter, you went with the so-called trusted subsystem architecture. Simply put: your backend service is treating the web app as a system that can be trusted to properly authenticate end users, and only perform backend service calls if and when appropriate, possibly including end user identifiers (such as usernames) as part of these calls.

Trusted Subsystem
The Trusted Subsystem solution

If you opted for the full-fledged impersonation and delegation solution, you probably learned very soon that this is hard. In the old on-prem enterprise world, you would have to learn about the intimate details of Kerberos Constrained Delegation. And if you were ‘lucky’ enough to be working with WIF and WS-Federation or SAML, you would find out that these protocols do support these scenarios, but still make it pulling-your-hair-out-difficult to implement. And now we’re just calling one downstream service from our web app; once we need to call yet another service from the first service, we more often than not just give up and go with the trusted subsystem approach after all.

Azure Active Directory To The Rescue

Luckily, SAML and WS-* are no longer the only protocols available. OAuth 2.0 and OpenID Connect have been gaining momentum for some time now, and are treated as first class citizens in the latest Identity & Access Management solutions that Microsoft is offering, especially Azure Active Directory. To add to that, Microsoft has provided a client-side library called ADAL (Active Directory Authentication Library) for a variety of platforms (including AngularJS and iOS for example) to simplify interaction with Azure Active Directory as much as possible.

And the good news is: even impersonation and delegation has gotten really simple, with a lot less moving parts on the client. (Everyone who has ever struggled with config files trying to get this to work using WS-* and WIF knows exactly what I mean…)

The guys at Microsoft are also putting a lot of effort into code samples on Github that show how to use Azure AD and ADAL to get all sorts of scenarios working.

image

The On Behalf Of scenario is also available on there. It’s a native client that calls an API, which in turn calls the Graph API on behalf of the logged in user. Obviously, the native client app can be substituted for an ASP.NET Core MVC web app, as shown in this repo.

Not every platform-scenario-combo is available. For example, the API calling another API scenario (i.e. the On Behalf Of scenario) is not available in its ASP.NET Core incarnation. And since the code to achieve this for ASP.NET Core Web API is not readily deducible from the native client sample that is only available with a ASP.NET Web API, I’d like to share some of it here.

First of all, the middleware to wire up an ASP.NET Core Web API to actually consume tokens is a bit different from how it used to be done. You can take you queue from the aforementioned repo; just make sure to save the token you receive so that you can access it later:

app.UseJwtBearerAuthentication(new JwtBearerOptions
{
    AutomaticAuthenticate = true,
    AutomaticChallenge = true,
    Authority = String.Format(Configuration["AzureAd:AadInstance"], Configuration["AzureAD:Tenant"]),
    Audience = Configuration["AzureAd:Audience"],
    SaveToken = true
});

Actually using this token to bootstrap the On Behalf Of flow works like this:

var authority = [insert authority here];
var clientId = [insert client ID here];
var clientSecret = [insert client secret here];
var resourceId = [insert the resource ID for the called API here];

AuthenticationContext authContext = new AuthenticationContext(authority);
ClientCredential credential = new ClientCredential(clientId, clientSecret);
AuthenticateInfo info = await HttpContext.Authentication.GetAuthenticateInfoAsync(JwtBearerDefaults.AuthenticationScheme);
var token = info.Properties.Items[".Token.access_token"];
var username = User.FindFirst(ClaimTypes.Upn).Value;
var userAssertion = new UserAssertion(token, "urn:ietf:params:oauth:grant-type:jwt-bearer", username);
AuthenticationResult result = await authContext.AcquireTokenAsync(resourceId, credential, userAssertion);

The AuthenticationResult that ADAL is returning here contains an Access Token that can be used to call the downstream Web API. Simple, right? OK, it involves some code, but it’s pretty straighforward when compared to a WS-*-and-a-WCF-service scenario I wrote about earlier.

Enter Microservices

As said before, we’ve all encountered On Behalf Of scenarios and the perils of getting them to work using SAML, WS-* or Kerberos, and more often than not we gave up on the full-fledged scenario. But in an increasingly API-centered world, we are calling other external services much more frequently than we did only a couple of years ago. And now that microservices gains a lot of momentum as an architectural style, this frequency increases even more since fulfilling a user request in a microservices environment is pretty much always a matter of multiple services collaborating.

Advocates of microservices recognize that flowing user identities through services is a concern that deserves more attention in a microservices architecture. Sam Newman, for example, discusses this issue in his book Building Microservices, in a paragraph aptly titled “The Deputy Problem”.

He recognizes the ease of use that comes with OpenID Connect and OAuth 2.0. And while he is still somewhat skeptical about whether these protocols will make it into the mainstream market any time soon, for all you dev’s out there that are on the Microsoft ecosystem, this is not a concern anymore.

Extending The Scenario

Obviously, we want to do more than simply impersonate end users when calling downstream services. Especially in a microservices environment, where multiple clients are calling multiple services for even the most mundane of tasks, we may want to have varying levels of trust: “Sure, I’d be more than happy to perform this request for the user, but only if he is calling me through an application that is entrusted to make these types of delegated calls.” In order words, you may want to base your authorization decisions on characteristics of both the end user and the calling app.

Azure Active Directory is capable of handing these types of scenarios as well, for example by using scopes. I’m not getting into those now, but I’ll be teaming up with my colleague Jurgen van den Broek for a session at the Dutch TechDays 2016, in which we will cover these and a lot more scenarios – including a peek into the future by discussing what the AAD v2 endpoint brings to the table.

Immediately after the TechDays session, I’ll update this post with a link to the full code sample. So stay tuned, and feel free to post a comment if you need help in the meantime.

What Exactly Is That CORS-Thing?? The What, the Why and the How Explained

If you’ve stumbled upon this post, chances are you’ve encountered some strange behavior while trying to call an endpoint, like a REST API for example, from within the browser. You may have seen your browser issue an OPTIONS request that is greeted with a 405 Method Not Allowed issued by the API.

fiddler

If this has happened to you, you are probably serving the JavaScript from another application than the one hosting the API.

You were probably expecting an XmlHttpRequest fetching a JSON document instead of that failed OPTIONS request, so read on to find out what’s going on and what you can do to successfully call that API.

So What’s CORS?

In short, you would probably benefit from enabling CORS on your API. CORS is short for Cross Origin Resource Sharing, and enabling CORS is basically a way of allowing your web application to call the API from the client browser, while that API is hosted on a host different than the one your web application is served from. You are not allowed to do that out of the box for security reasons. If you are only interested in actually getting this to work, feel free to skip to the How-part of this post.

OK, so you are interested in a little more background. As said, you can’t call API’s from the browser out of the box if they do not reside on the same host (‘have the same origin’) as the web application. Same origin here means: same URI scheme, hostname and port number. This behavior is enforced by the browser. If a piece of JavaScript attempts to call an API of different origin than its own, the browser will first makes a pre-flight request to the target server to ask whether the server is OK with being called from another origin. Enabling CORS means: instructing the API on how to meaningfully respond to such pre-flight requests. Without CORS enabled, API’s typically respond with the 405 we talked about. Most modern browsers support this pre-flight request, which is a prerequisite for using CORS.

Why CORS?

The ‘security reasons’ behind all this are known as the same-origin policy. According to this principle, resources are isolated from each other on the basis of their origin. So, a piece of script for example can only access other documents in the browser when they share the same origin, and it can only call endpoints on that same origin; all resources from other origins are off-limits.

cross-blocked

Source: http://www.lucadentella.it/en/2013/07/11/javascript-same-origin-policy-e-jsonp/

This makes good sense, because failure to restrict this would mean that a malicious web page that is opened in a user’s browser session would have access to all documents and endpoints for other websites the user is also visiting. Imagine one of these other websites being your personal banking environment, and you probably get why the same-origin policy is kind of a good thing.

But obviously, there are also legitimate use cases for cross-origin API calls. Strategists, visionaries and evangelists preach the API-driven world, in which every company should disclose their processes through API’s to be consumed by clients. Those clients typically will not reside on the same origin, but we do want them to be able to call our API’s.

In recent years, several hacks have been conjured up to bypass the same-origin policy, with JSONP being one of the more prominent ones. I won’t dive into the specifics here; you can read all about it online. The issue with JSONP (apart from some sophisticated exploits) is that, as an API publisher, you open up your API for all origins by definition. And this is where CORS comes in: a controlled way of whitelisting some origins while treating all others according to the same-origin policy. And, as a bonus, the implementation is much easier: CORS is entirely a server-side setting, whereas JSONP requires the client to do part of the heavy lifting.

HowTo CORS

So, on to the actual way of doing this. And this is actually the simplest part: you just need to make sure that the API responds differently to the OPTIONS request. What the browser is actually asking, by means of the Origin header it sends along, is whether the specified origin is allowed to call the API. The API may either not allow this at all (the default), only allow a specific list of origins, or allow all origins. And it communicates this by including a Access-Control-Allow-Origin header to the response to the pre-flight request.

fiddler2

A specific value is indicative of an API that allows this specific origin. Alternatively, an asterisk (*) indicates that all origins are allowed.

For a .NET-based WebAPI, you can use OWIN middleware or the WebAPI CORS package, depending on your application architecture and the requirements. The use of CORS through OWIN middleware is nicely described here, while the CORS package method is detailed over here.

public partial class Startup
{
    public void Configuration(IAppBuilder app)
    {
        app.UseCors(Microsoft.Owin.Cors.CorsOptions.AllowAll);
    }
}

And yes, you can only enable CORS on the API side; not on the caller side. After all, the same origin policy is meant to protect the API from access by malicious websites the user may be visiting.

Hope this helps!

Blast from the Past: A Delegation Scenario using WS-Federation and WS-Trust

A couple of weeks ago, I wanted to outline some of the different flavors and protocols available for delegation scenarios using a federated identity. One of the protocols on my list was WS-Federation and WS-Trust. Yes, I know, all the cool kids are doing OpenID Connect these days, but some of us are working for enterprises that bought into the whole federation-thing rather early and while still on-premise. For those environments ADFS is most likely the Identity Provider. And if the relying parties are .NET-based apps, the protocol of choice for identity federation is WS-Federation.

Of course, I did want to use the latest and greatest as much as possible, so I checked out the new OWIN/Katana gear for WS-Federation. And sure enough, getting identity federation to work using ADFS as the Identity Provider was a breeze. However, delegating the federated identity to a backend WCF service: not so much…

The theory here is that, firstly, the WCF service is also registered as a relying party in ADFS; secondly, that the web application is allowed to delegate identities to that relying party; and thirdly, that the web application can use the ADFS-issued user token to send back to ADFS as part of the request for a delegation token. Now the issue I encountered is that the token, as persisted by the OWIN middleware, does not have the same format as is expected by the time the delegation call is being made. More specifically, the token is persisted as a string, whereas the delegation code is expecting a SecurityToken.

I’ve tried to work this out in just about every way I could think of. This was not exactly made easier by the utter lack of online resources when it comes to WS-Federation (especially in its .NET 4.5 and OWIN incarnations). Still, I did not get this to work using the OWIN middleware. So I defaulted back to the ‘classic’ way of making this work, configuring the initial federation with ADFS through the web.config for both the front end MVC application and the backend WCF service that the web app is calling into. And as said, the online resources on WS-Federation in .NET 4.5 are limited, so I figured I’d share my sample on Github.

There’s a lot of moving parts to this sample, and principles to grasp if you want to fully understand the code. Luckily, all of that is pretty much covered in this guide. The ADFS part of it is pretty accurate as it is, and even though it is aimed at ADFS 2.0, it’s easily transferable to ADFS 3.0. As far as the code goes, the principles remain the same but the implementation is based on WIF on .NET 4.0. So you’ll have to do some digging through my sample to match it to the way it is described in the guide. Just see it as a learning opportunity ;).

I will reveal one difference: the guide assumes that the account running the web application is domain-joined so the web app can authenticate itself to ADFS using Windows Authentication when it makes the call to get the delegation token. To simplify the setup, I chose to authenticate to ADFS using a username and password so that I wouldn’t have to set up Kerberos. To make the username-based binding work, I used Dominick Baier’s UserNameWSTrustBinding. This was available in WIF on 4.0 but did not make it into 4.5, so Dominick added it to his Thinktecture.IdentityModel NuGet package.

Oh, and don’t expect the sample to be production ready. In fact, it won’t even work out of the box when you run it. You will have to configure several URL’s to match you environment. And as said, you’ll have to configure ADFS using the tutorial I mentioned.

Of course, I haven’t totally given up on the OWIN route, nor am I finished outlining the different delegation flavors. So stay tuned, because there’s more to come on coding identity federation and delegation!

TFS Build: How To Customize Work Item Association

Recently I got involved in implementing Team Foundation Server 2012 for a large development project. And even though I’ve worked with several versions of TFS over the years, this was the first time I really dove into the options for customization beyond your average workflow modifications. I’ve picked up some interesting tips and tricks over the last couple of months. Most of these customizations were already available somewhere on the web (see for example Ewald Hofman’s excellent series on customizing Team Build 2010, which in general applies quite nicely to TFS 2012 as well), but some modifications are my original work in that I did not find it anywhere else on the web. I thought I’d share some of the things I encountered.

A standard build based on the default build template will associate changesets and work items to builds. The way it does that is by retrieving the label for the previous successful build for the given build’s Build Definition, and by determining which changesets are included in the current build that were not included in the previous build. Some or all of those changesets might have work items associated with them, and those work items get associated with the build. Assuming the default build template, this is done as part of the “Compile, Test, and Associate Changesets and Work Items” parallel sequence. Look for a sequence like this in the template editor:

AssociateChangesetsAndWorkitems

Now if your team is anything like my team, they will not deliver a specific piece of functionality or fix a particular bug in a single checkin. A single functional requirement as documented in a Product Backlog Item may require the involvement of multiple persons, all with different specialties. So if one team member checks in his changes and associates it with a Task, that does not mean that the Task’s parent Product Backlog Item should be considered part of the next build. And the opposite can also be true. Let’s assume, for example, a scenario in which the project’s goal is to deliver a standard software product (an ERP system for example) and some customizations to go with it. Let’s also assume that the metadata for this ERP system does not reside in TFS. Instead, during a release build, the metadata is extracted from the development environment, committed to TFS, and from there pushed to QA, and ultimately to Production. Now, if a bug is filed, the bug may very well be caused by some configuration setting in the ERP system. The ERP guy on the team fixes the bug by modifying the configuration, which is pushed to QA with the next release. The bug is then marked as Resolved, but TFS Build will never associate this bug with a build because no changeset occurred that contained the fix.

To solve this problem, I tried my hand at an alternative method for associating a build with work items. What I wanted was to associate all work items that have a specific status (such as Done or Resolved), and are not yet associated to any previous builds. That means that we have to query TFS to get that specific set of work items, since the set is no longer a function of the changesets for this build. Now, querying TFS for work items is pretty well covered on the web, so I’m not going to elaborate on how to do that. The relevant part here is that the result of this query is stored in a variable scoped to the topmost Sequence, and that the query is the first thing that gets executed on the Build Agent. This is to ensure that the correct work items are retrieved before the sources are downloaded so as to avoid synchronicity issues. This variable is then used as a parameter in a custom build activity. For each work item in the list, the activity sets the value of the IntegrationBuild field to the build number, and it associates the work item with the build. The code for this activity looks something like the following:

using System;
using System.Activities;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Build.Workflow.Activities;
using Microsoft.TeamFoundation.VersionControl.Client;
using Microsoft.TeamFoundation.WorkItemTracking.Client;

namespace CustomActivities
{
    public sealed class AssociateResolvedWorkItems: CodeActivity
    {
        [RequiredArgument]
        public InArgument<IList> WorkItems { get; set; }

        protected override void Execute(CodeActivityContext context)
        {
            var workItems = context.GetValue(WorkItems);

            var buildDetail = context.GetExtension<IBuildDetail>();
            var store = buildDetail.BuildServer.TeamProjectCollection.GetService<WorkItemStore>();

            foreach (WorkItem workItem in workItems)
            {
                // Update the workitem
                workItem.SyncToLatest();
                workItem.PartialOpen();
                workItem["Microsoft.VSTS.Build.IntegrationBuild"] = buildDetail.BuildNumber;
                workItem.History = ActivitiesResources.Format(ActivitiesResources.BuildUpdateForWorkItem, null);
            }
            var array = workItems.ToArray();
            var errors = store.BatchSave(array);

            var associated = new List();

            foreach (WorkItem workItem in workItems)
            {
                var error = errors.FirstOrDefault(item => item.WorkItem.Id == workItem.Id);
                if (error != null)
                {
                    // Alert update error
                    context.TrackBuildWarning(String.Concat("Unable to associate work item '", workItem.Id.ToString(CultureInfo.InvariantCulture), "': '", workItem.Title,
                                      "' - ", error.Exception), BuildMessageImportance.High);
                }
                else
                {
                    // Write update message...
                    context.TrackBuildMessage(String.Concat("The work item '", workItem.Id.ToString(CultureInfo.InvariantCulture), "' was updated with build label '",
                                      buildDetail.BuildNumber, "'."), BuildMessageImportance.Low);
                    // ... and add to the list of work items to associate
                    associated.Add(workItem);
                }
            }

            // Associate updated workitems to build
            buildDetail.Information.AddAssociatedWorkItems(associated.ToArray());
        }
    }
}

You see that this is a two-way association. First, the build number is set as the value of the IntegrationBuild field on the work item. This makes sure that work item queries that use this field still work as expected. Second, the work item is added to the Build Information. This makes sure that the work items are displayed on the Build Details. And as an added bonus, Microsoft Test Manager’s Recommended Tests view also correctly displays the list of work items when comparing two builds.

The activity is inserted in the template at the same place as where the standard associations occurred:

AssociateChangesetsAndWorkitemsCustomized

One final thing to do is determine whether you also still want the standard association (based on changeset-related work items) to occur. If you don’t want that, the “Associate Changesets And Workitems” activity (which appears right above our newly inserted activity) has an interesting parameter for you to edit. If you right-click on the activity and select Properties, you notice that one of the parameters is named “UpdateWorkItems” and has the value ‘True’. Simply changing that to ‘False’ will make sure that only the work items you selected are associated with the build.