Deploying Azure Pipelines agents as containers to Kubernetes

A common problem I have seen across the teams I’ve worked on that use Azure Pipelines for building and releasing code is the lack of enough pipeline agents to handle the increasing number of builds/releases that need to be executed simultaneously. Most teams would start by just using Microsoft-hosted agents, which is super straightforward, no setup required. However they come with a few downsides:

  • They only allow for one parallel job at any given time for private projects unless you opt for purchasing more parallel jobs (public projects allow 10 free parallel jobs).
  • They use the Standard_DS2_v2 vm size which gives you only 2 vCPUs, 7 GiB of memory and 8000 IOPS. They also come with at least 10 GB of storage.
  • They come with a large list of installed software which rarely matches the exact set that you need, which is usually a very small subset from what they provide there. See the software list for Windows and Linux.

To overcome those issues you would usually come up with your own self-hosted agents where there is no limit on parallel jobs and you can decide what vm size to use and what software to put there. But then this comes with its own downsides:

  • You have to figure out the entire provisioning of the VM yourself, which can involve a lot of error prone steps if done manually. If not doing it manually you have to figure out how to automate the quick creation and deletion of VMs depending on team needs
  • Provisioning a VM can take several minutes, even more if setup scripts are run after provisioning
  • You have to keep the VM well maintained and updated

One way to find a middle ground among all these issues is to turn the agents into docker containers and then have Kubernetes orchestrate the provisioning of those containers. Yes, you still have to stand up and maintain a k8s cluster but then you get a bunch of benefits:

  • You can easily scale up/down the number of agents as needed. No parallel job limits
  • You get to choose the k8s node vm size
  • You get to pick and customize the docker image to use so it only has the software you need
  • You let k8s deal with all the provisioning stuff. It’s will ensure the number of required agents are always there
  • Provisioning is pretty fast, especially after provisioning the first agent

A bunch of people have come to this conclusion already, so when I found this open source project a while ago I gave it a try and worked pretty well. However since then the Azure DevOps team stopped supporting the docker image used in that project and you are now expected to come up with your own docker image. Plus the old VSTS went into a few changes. Therefore I forked the project and updated a few things to match the latest guidelines and added some more guidance on how to create the docker image and the kubernetes cluster.

To get started you can go to my azure-pipelines-kubernetes-agents GitHub repo and follow the steps described there. Here I’ll just summarize what you’ll end up doing to quickly come up with your own k8s hosted Azure Pipelines agents:

  1. Create a Personal Access Token (PAT) with the Agent Pools(read, manage) scope
  2. Create your pipelines agent pool
  3. Create and publish your pipelines agent docker image (a sample is provided in the repo)
  4. Create your k8s cluster. The repo provides steps for Azure Kubernetes Service (AKS)
  5. Install the Helm chart

So, for instance, once you have the pipelines pool and the k8s cluster created as well as the docker image published, this is all I did to provision my pipelines agents:

helm install --set image.repository=julioc/azpagent --set image.tag=ubuntu-16.04 --set azpToken=[your token here] --set azpUrl=https://dev.azure.com/julioc --set azpPool=MyPool -f helm-chart\azp-agent\values.yaml azp-agent helm-chart\azp-agent

Which in the pool management page looks like this:

And then I was able to start running pipelines in those agents right away:

And if I need to scale up to say 10 pipeline agents I could just do this:

kubectl scale statefulset/azp-agent --replicas 10

Which after a few mins (depending on your node vm size) looks like this in the pool management page:

So check it out and let me know if you have any comments or questions.

New Jobs Section

From time to time I get approached by an overwhelming amount of recruiters for several software engineering positions for locations across the US and a few of them remote too. This is probably not unusual for folks working in software engineering, especially if you work on cloud tech. Some of these positions are actually really interesting. Looks like HBO keeps growing in the Seattle area with the upcoming HBO Max service starting in spring of 2020, Amazon expanding into Cupertino, CA, Mexico City and Vancouver, B.C, and Volkswagen coming up with it’s own Automotive Cloud in Redmond, WA.

However since I’m not interested in switching jobs these days (having some good fun at the Microsoft Project xCloud team) and I keep just replying with a “not interested” to all these recruiters, I thought I might as well share these opportunities here so others get to know about them.

You can check out the new jobs section here and if you are interested feel free to drop me an email or comment on this post with your updated LinkedIn profile so I can pass it over.

Azure Pipelines Tutorial – Continuous Integration

When software developers start collaborating in a project one of the main things to address is how to prevent build and test breaks. Here is where Continuous Integration (CI) shines and one tool that enables it and that has worked for me fairly well in the past is Azure Pipelines.

I just published a tutorial on how to enable CI with Azure Pipelines:

The tutorial will teach you:

  • How to enable continuous integration (CI) with Azure Pipelines
  • What is a YAML based pipeline and why use it
  • How to create a pipeline that runs on any change to your GitHub repo
  • How diagnose and fix issues detected by the pipeline
  • How to report the status of the pipeline on GitHub

Enjoy!

Building a CI/CD pipeline for a containerized Asp.Net Core 3.0 Web API

To close on the Asp.Net Core 3.0 Web API series I just published a video on how to build a CI/CD pipeline to fully automate the deployment of a Web API docker container to AKS. Here it is:

In this one you will learn:

  • How to push your project files to a GitHub repository and how to show its build status
  • How to create a yaml based Azure Pipeline to continuously build and publish the container image
  • How to use the same pipeline to continuously deploy the container image to AKS
  • How to use Azure Pipelines Environments to get the state and history of deployments

I hope this series has been useful and, as always, please leave me a comment here or in the video with any feedback, which is highly appreciated.

Enjoy!

Deploying an Asp.Net Core 3.0 Web API on AKS

Last week I published a video on how to deploy an Asp.Net Core 3.0 Web API to a local Kubernetes cluster. This week I thought I would move one step forward and show how to deploy the same Web API container to Azure Kubernetes Service (AKS). So here you go:

In this new video you will learn:

  • How to create a container registry and an AKS cluster
  • How to push your Web API container to a container registry
  • How to generate Kubernetes yaml files to describe your deployment and service using Visual Studio Code
  • How to deploy your Web API container to AKS

Please leave me a comment here or in the video itself about any feedback you might have on this video.

Enjoy!

Deploying an Asp.Net Core 3.0 Web API on Kubernetes

As a follow up from last week’s video on Containerizing an ASP.NET Core 3.0 Web API here for a step by step on how to deploy an Asp.Net Core 3.0 Web API on a local Kubernetes cluster:

In this new video you will learn:

  • How to enable a local Kubernetes cluster in your box
  • How to add Kubernetes files to describe your web API deployment using Visual Studio Code
  • How to deploy your containerized Asp.Net Core Web API to Kubernetes

Again, please let me know your thoughts on this video, either here or in the video comments section. All feedback is very appreciated.

More videos coming soon!

Containerizing an Asp.Net Core 3.0 Web API

It’s been ages since I wrote anything here, but recently I decided it’s time I start sharing a few of the things I have learned in the past few years. Also, since .NET Core 3.0 just got released today and since I’ve been working with containers for a while I thought it would be appropriate to start with a video on how to containerize an Asp.Net Core 3.0 app, specifically a Web API type of app since that’s what I’ve mostly been using for building microservices. So here it is:

There I talk about:
• How to create an Asp.Net Core 3.0 Web API project
• How to add Docker artifacts with Visual Studio Code, including the generation of the Dockerfile
• How to build and run the Asp.Net Core project as a Docker container

Let me know your thoughts on this video, either here or in the video comments section. Would appreciate all feedback to incorporate it in future upcoming videos.

Cheers!

Julio

Translator is now a Universal App

Just about a week ago it seemed like the most popular of my apps was my Desktop Translator, which was a Silverlight Out Of Browser (OOB) app. However, since Silverlight does not seem to play very well with Edge, the default browser in Windows 10, people was struggling with installing the app. At that point I also had a Translator for Windows 8 and a Translator for Windows Phone 8. So, it seemed like a good opportunity to try out the Windows Universal platform and make my life easier along the way.

And here is the result:

Get it now for Windows 10 here (and probably with the same link for Windows Phone 10 when it becomes available).

I didn’t quite have time to add new features to this universal version of the Translator. However I did add the capability for translating the text as you type it. I thought that could be handy.

Besides the fact that all the app source code now lives in a single place (which is already cool) I like the fact that I can keep offering a Windows 8 only version and a Windows Phone 8.1 only version under the same umbrella in the Store:

Keeping the Windows 8 and Windows Phone 8.1 versions alive is important for people that has not made the move to Windows 10 yet. This because universal apps won’t work on anything before Windows 10. And nicely the Store is very kind of offering me appropriate links for people on each platform:

And interestingly, the Windows 8.1 URL would work even for Windows 10 people, which is great!

It has been a long way for this little app since releasing it as a gadget for the Windows Vista sidebar. Who knows what will come next!

Translator 2.0 for Windows Phone 8 Now Available

The Windows Phone 8 SDK was publicly available last month so I thought the Translator for Windows Phone could take advantage of some of its new features. The upgrade process took me a few days, mostly because of not been able to work on it for more than an hour or two on each session, but I’m glad to announce it’s finally ready and available for download!

Get the Translator 2.0 here.

Besides all the great new stuff that all Windows Phone 8 apps get for free, these are the two new features that really make a difference for the Translator:

1. Speech Recognition: You can now use the new Speak button to say a word or phrase to the app. It will recognize your spoken text and immediately translate it to the currently selected target language. Cool feature and actually ridiculously easy to implement thanks to the new Speech APIs.

Screenshot8_EN

2. Better language support: Many people complained in the past about the Translator not been able to display text in a few languages, been Hindi among the most requested. This was sad because the Translator was actually able to translate the text to those languages but could just not display it in the Windows Phone 7 platform. Gladly, and without having to add a single line of code, the Translator can now display text in Hindi and 20+ new languages!

languagesupport

 

I hope this helps and please keep sending feedback either from the app itself or via this blog. I’ll do my best to keep improving the app to make it as useful and easy to use as possible.

Julio