Building reliable (and fast!) directory sync

“Who’s your User?”— Master Control Program, TRON (1982) If user identity is important to your application, you need a way to manage it. Usually this means user accounts, and you create them when someone signs up. But allowing employees to sign up for whatever SaaS app strikes their fancy is a management nightmare. So organizations like ways to control which users exist (or do not exist) in your app. That controls who can sign in, but how about what they can do? For that you need roles, or groups, which, like the users, come from the organization's identity provider - Entra, Google, Okta, etc. Groups form the basis of Firezone's access model. They determine who can access what. The process of getting users and groups into your app is called directory sync, and it's surprisingly tricky to do well. In this post we'll cover what directory sync is exactly, the leading standard for implementing it, and why we opted to forgo it entirely to build our own engine from scratch. Directory sync: a gentle primer An organization's directory is the list of who works there and what they're allowed to access. It lives in the identity provider and is made of three things: Users: the people in the organization. Each has a name, an email, and a status like active or suspended. Groups: named collections of users, like Engineering or Product. Groups let you grant access to many people at once. Members: who belongs to what. A member of a group is either a user or another group. A user's access comes from every group they belong to, directly or through other groups. Directory sync is the process of copying that directory into your app and keeping it up to date. The identity provider is the source of truth, so what your app holds is a copy.

It's worth clarifying what directory sync is not: Single Sign-On (SSO). This may seem obvious but many folks conflate the two. Leading authentication standards like OpenID Connect say nothing about how to bring users into the application. In directory sync we're referring to precisely the mechanism of bringing users (and groups) in, but not how they're authenticated (for that, see this post). A simple example Say your organization has the following directory: An example directory with a nested groupA directory with two groups and two users. The Product group has two members: the user Alice and the group Engineering. The Engineering group has one member: the user Bob. Because Engineering is a member of Product, Bob belongs to Product indirectly.Group: ProductUser: AliceGroup: EngineeringUser: Bob In your database it might look something like this: The example directory as database tablesThree tables. Users lists Alice and Bob. Groups lists Product and Engineering. Members has a user column, a group column, and a parent group column. Alice is a user in Product. Bob is a user in Engineering. The group Engineering is in Product. Each row leaves one of the first two columns empty, and answering whether Bob is in Product means following parent groups through recursive queries.UsersnameAliceBobGroupsnameProductEngineeringMembersusergroupparent_groupAlice-ProductBob-Engineering-EngineeringProduct To answer common access questions in your app like "is Bob a member of Product?", you'd have to look at Product's members, then the members of any groups in there, and so on until you find Bob. To avoid that, we can flatten groups, giving each user a row for every group they belong to, directly or not: The example directory with flattened membershipsThree tables.