Attomus Infrastructure - an overview

We are often asked what the Attomus architecture looks like and how internally we approach development and client work and what tools we use.

Clearly as partners of Netlify, Jetbrains, XCP-ng and Atlassian we tend towards using their products and services as the basis of our infrastructure, “eat your own dogfood” being the internal mantra, but there’s value in explaining the reasoning and how we utliise each to combine into the enterprise landscape that we operate.

Websites

So starting with what is visible to you, the user, we host our static sites which include this site, attomus.com, and the marketing sites for our products such as semafore.io, on Netlify using the JamStack architectural approach, building from repos that are hosted on our internal BitBucket server. This allows anyone to publish changes to a branch for internal review (including external stakeholders) before public release, but in a very easy and well controlled pipeline. Very little human interaction is required, and all public releases automatically compress images, pick up Netlify forms, minify javascript and css and run a LightHouse audit as part of the build. Each time there’s a public release the results of these tests are automatically sent to the build team via a dedicated Slack channel. The main benefits of this approach are the seperation from our own servers in the event of an attack targetting our public interface, and secondly the very much reduced attack service that using static sites gives us.

Support / Ticketing

Like all technology companies we operate a public support site, that in our case runs under Jira Service Desk (installed behind our firewall rather than in the cloud) accessible at support.attomus.com, which also monitors emails sent to our support mailbox. This allows customers to interact through whichever means is preferable for them, and then receive updates through the same mechanism. In the same way all the contact forms from our websites get captured as tickets so that the team have a single repository to manage inbound information and customer communications.

Development

Looking at our development processes, both for clients and our own products such as Semafore, we use a combination of Jetbrains’ Space and Jira Software. Space, using the new behind firewall version (currently in Beta), is used where for security reasons we need physical separation from our other projects - and provides us with code repositories, development commentary and issue tracking, whilst the majority of non-security related work takes place using Bitbucket as the code repository, Jira Software for tracking and commentary, and Confluence for documentation and as an internal knowledge base.

Incident Management & Status updating

As an Atlassian partner, we did try and make use of OpsGenie to manage incidents and alerting for a period, but as a relatively small enterprise we found that it was overkill for our needs and so dogfooding was outweighed by the need to use something proportional and appropriate to our needs. So instead we run an in-house monitoring solution that alerts the product owner of any degraded infrastructure via Slack, including status errors or if there is a trend that would cause low memory, disk space or similar within a set period, and then the owner manually assigns the resolution task to the relevant expert. In client implementations where there is less overlap of expertise and multi-skilling we normally look to implement a more highly automated solution, but for our purposes a more manual approach is more appropriate and suits us well.

This addresses the internal resolution of infrastructure incidents, but we still need to inform you the user when some infrastructure is either giving cause for concern (therefore a risk to live service) or for planned maintenance, and for this we use StatusPage which all users internal and external can monitor or subscribe to at status.attomus.com. This in turn links to our internal management system so that we can track effort and effectively have tickets raised automatically in the event of long-running issues or to trigger the team before a planned maintenance event.

Development Tooling

As a company our standard development tools are based upon the JetBrains products set - be it specialist tools such as GoLand or DataGrip, or IDEA Ultimate for our generalists. Since most of the development team are experienced hires rather than starting afresh, they are welcome to use whatever tools they wish that do the job, providing if they stray from the JetBrains tools they support themselves, and so as a company we have developers using tools from Visual Studio to Xcode to Atom and bbEdit.

Then looking to the DevOps side, including build, we use JetBrain’s TeamCity as a build server, with access very strictly controlled and the machine and agents inaccessible from the internet. This is a very important point that we come across time and again on client sites - connecting a build server that can have a number of build artefacts and even access across the client-side infrastructure to the internet - which makes a very large threat surface available with potentially very high-rewards for an attacker - the worst of both worlds. Build servers should never be publicly visible.

Server tooling

In general we run most of our products and tools on Debian servers on top of XCP-ng, this gives us the flexibility to dynamically assign memory and CPU to match demand without wasting resources on lots of excess hardware, and also without the overhead or costs of running systems in the cloud. Having said that most of our products are designed to be able to burst out to the cloud when wechave exceptional demand, particularly those that are built on top of Kubernetes, but for the majority of our Use Cases we have a need to run behind our firewall, be it for data sovereignty or specific security requirements. By standardising our server farms to the XCP-ng / Debian stack, issues are much quicker to address since the institutional knowledge is so much deeper and despite bouncing between servers the engineer has a standard install to work from. Similarly, security updates are much simpler to manage and cascade when the entire company operates under a single server OS, making patch management and server management much easier - which means fewer mistakes and less friction when conducting scheduled maintenance or fixes.

The future

In time we will look we probably look to make more use of Space as it leaves Beta ; it is much lighter weight than Jira / Bitbucket which makes it more performant at our scale, but we won’t make that move until we have a suitable ticketing solution in place to sit alongside it. Similarly as we look to architect more client products under the JamStack paradigm (due in no small part to its considerably lower attack surface) we will move installations away from our datacentres and into Netlify. This is still some time in the future; from its start with static websites without any ability to run server-side functions or query databases, the technology has moved forwards in leaps and bounds, but is not yet mature enough to cover the breadth of our needs. It won’t be long though.

Fancy reading something else - what takes your fancy?