I recently had a chance to configure API functional tests and run them under the Azure DevOps pipeline. I struggled a bit and spent a couple of days spotting the way to get the security tokens from Azure AD B2C in the Azure CI/CD pipeline. Also handling the anti-XSRF token is a bit tricky part too. Here I am documenting the steps for those searching for solutions with the below background.
- API’s — in our case, it is .net core based Web APIs hosted in Azure. But it can be any kind of RESTful APIs.
- The API is secured with Azure AD B2C-based authentication. In your case, it can be any similar kind of authentication mechanism such as Azure AD. The logic still remains the same.
- API’s are protected against XSRF by expecting anti-XSRF tokens in the request headers. Doesn’t matter whether this protection is there or not on the API side in your case.
We are going to use the below tools to test the API’s in the Azure DevOps pipeline
- Postman as the test tool for Web API’s
- Azure DevOps (TFS in old terms) release pipelines to automate the tests
- Newman command-line tool in Azure DevOps pipeline to run the automated tests
- Replace tokens task in Azure pipeline to replace run-time configurations in DevOps pipeline. We can use this setup for user login credentials, targeting different dev/test sites, and role-based tests.
What all are the main problems we are about to solve?
- Making Postman tests compatible for Azure DevOps pipeline
- How to securely configure user credentials in API functional tests to authenticate/authorize users.
- How to login and receive JWT tokens from Azure AD B2C in the Azure DevOps pipeline
- Utilize the JWT token and anti-XSRF tokens for API test calls
- How to configure API tests with different user roles
Let’s dive into…
Setting up the API
- Arrange the API actions protected by Azure AD B2C. Detailed explanation for this is already available here: https://docs.microsoft.com/en-us/aspnet/core/security/authentication/azure-ad-b2c-webapi?view=aspnetcore-2.2
- Make the Web API protected against XSRF. Here is the documentation for the setup: https://docs.microsoft.com/en-us/aspnet/core/security/anti-request-forgery?view=aspnetcore-2.2
To demonstrate this article, I just exposed two API actions from TodoAPI samples from the first link above.
- A GET API action that returns a response with XSRF-Token header. This is needed to make the anti-XSRF token retrieved at once upon user login and use it for subsequent API calls. The second link above will tell you how to configure this on the API side. This action is protected by AUTHORIZE attribute to check valid access tokens from Azure AD B2C.
- A GET API action to return Todo list by its id. This method is protected by AUTHORIZE attribute to check valid access tokens from Azure AD B2C. It also validates valid anti-XSRF tokens in the request header.
Register Postman in Azure AD B2C
Since there is no user interactive session under the Azure DevOps CI/CD pipeline, we need to have some passive authentication flow to handle this. There is one good article explaining the possible options here: https://www.deliveron.com/blog/three-ways-get-oauth2-access-token-api-testing-azure-ad-secured-apis/
Some of the possible options:
- Use Native App to sign on via Resource Owner Password Grant
- Send the Client Secret via Client Credentials Grant
- Use an automated UI Test to get the access token via Authorization Code Grant
We are going to use the first option, Resource Owner Password Credentials (ROPC) Grant for authentication. An article explaining the ROPC registration is explained here: https://docs.microsoft.com/en-us/azure/active-directory-b2c/configure-ropc
Please note that this registration is for registering Postman to request tokens from Azure AD B2C.
Arranging the Postman tests
Let’s start by creating simple tests in Postman. We will arrange the tests by the following steps
a) Set up the environment in Postman
With Postman, click on ‘Manage Environments’ icon in the top right corner of the Postman UI. Add a new environment with the name ‘TodoApiDev’. Set up the following values
Where,
user-name — is the name of the user name to authenticate against Azure AD B2C. The value for this will be configured in the CI pipeline by a secured release variable.
user-password — is the password to authenticate against Azure AD B2C. The value for this will be configured in the CI pipeline by a secured release variable.
role-name — The role of the user against the tests is going to be conducted. The value for this will be configured in the CI pipeline.
b2c-domain-name — The domain name in B2C where the application is registered. The value for this will be configured in the CI pipeline to support different dev/test environments.
b2c-policy-name — The ROPC policy created under B2C to authenticate. The value for this will be configured in the CI pipeline to support different dev/test environments.
Below are few additional variables for the site URL, to store the access token received from Azure AD B2C and to store the anti-XSRF token.
b) Create tests in Postman
When the environment variables are created in Postman, the next step is creating the API tests. We are about to create the following requests against the exposed APIs. Create a test collection to add your tests. Add the following requests under the collection.
i) A POST request to get access token
URL:
where b2c-domain-name and b2c-policy-name are already configured in the environment variable.
Body:
Tests:
var jsonData = JSON.parse(responseBody);
pm.environment.set(“access-token”, jsonData.access_token);pm.test(“Status code is 200”, function() {
pm.response.to.have.status(200);
});pm.test(“Response time is less than 3000ms”, function() {
pm.expect(pm.response.responseTime).to.below(3000);
});
Where the first two lines will set an environment variable named id-token with the access token received from Azure AD B2C. This is going to be used by other subsequent tests.
ii) A GET request to get the anti-XSRF token from API
We expect a cookie set with anti-XSRF token in the response by calling this API action. This API action is protected by Azure AD B2C.
URL:
{{test-site-base-url}}/todo
Tests:
var xsrfToken = decodeURIComponent(pm.cookies.get(“XSRF-TOKEN”));
pm.environment.set(“x-xsrf-token”, xsrfToken);pm.test(“Status code is 200”, function() {
pm.response.to.have.status(200);
});pm.test(“Response time is less than 3000ms”, function() {
pm.expect(pm.response.responseTime).to.below(3000);
});
iii) A GET request to get the Todo list by specific Id
As stated before this API action is protected by Azure AD B2C and anti-XSRF validations.
URL:
{{test-site-base-url}}/todo/1
Headers:
Tests:
var roleName = pm.environment.get(“role-name”);
var jsonData = pm.response.json();pm.test(“Status code is 200 for administrator and 401 for other users”, function() {
if (roleName === ‘administrator’)
{
pm.response.to.have.status(200);
}
else
{
pm.response.to.have.status(401);
}
});pm.test(“Response time is less than 3000ms”, function() {
pm.expect(pm.response.responseTime).to.below(3000);
});pm.test(“Content-Type is valid”, function () {
pm.response.to.be.header(“Content-Type”, “application/json; charset=utf-8”);
});pm.test(“Response returns the expected id”, function() {
pm.expect(jsonData.id).to.eql(1);
});pm.test(“Response body matches with the expected results”, function () {
pm.response.to.have.body(‘{“id”:1,”name”:”Item1″,”isComplete”:false}’);
});var schema = {
“definitions”: {},
“$schema”: “http://json-schema.org/draft-07/schema#“,
“$id”: “http://example.com/root.json“,
“type”: “object”,
“title”: “The Root Schema”,
“required”: [
“id”,
“name”,
“isComplete”
],
“properties”: {
“id”: {
“$id”: “#/properties/id”,
“type”: “integer”,
“title”: “The Id Schema”,
“default”: 0,
“examples”: [
2
]
},
“name”: {
“$id”: “#/properties/name”,
“type”: “string”,
“title”: “The Name Schema”,
“default”: “”,
“examples”: [
“Item2”
],
//”pattern”: “^(.*)$”
“pattern”: “^[0–9a-zA-Z]*$”,
},
“isComplete”: {
“$id”: “#/properties/isComplete”,
“type”: “boolean”,
“title”: “The Iscomplete Schema”,
“default”: false,
“examples”: [
false
]
}
}
};pm.test(‘Response results matches with Schema’, function() {
pm.expect(tv4.validate(jsonData, schema)).to.be.true;
});
We are good in arranging the tests now. Let’s make the tests exported in order to make it suitable for CI pipeline.
Exporting the Postman tests
We have to export both the environment and the test collection to make it run under the Azure DevOps CI pipeline.
To export the environment, from Postman, select ‘Manage Environments’ and choose the environment to export. Click on the ‘Down’ arrow to save the environment information as a JSON.
To export the test collection, select the triple dots (‘…’) on the collection name and click on the ‘Export’ option and save it as a JSON file.
Since our tests and environment files now source code ready, let’s place those under the source code repository.
Upload test collection and environment files into GIT
For this demonstration, I used the GIT repository to store the source code. But it is of course your choice to choose your repository. I leave this section open without any explanation since instructions will vary based on each type of repository. There are a lot of materials on the internet to configure a source code repository. Hope you will find good articles on this topic on the internet.
Now, we are more closer to get our tests into action. Let’s configure the tests under the CI pipeline.
Configuring Azure DevOps pipeline
If you want to learn how to configure the CI/CD pipeline, please have a look at the DevOps lab in below link: https://www.azuredevopslabs.com/labs/vstsextend/azuredevopsprojectdotnet/
Assuming you are familiar with configuring CI/CD pipelines under Azure DevOps, I am diving into the pipeline tasks directly. Following is the list of tasks to be configured under the Azure DevOps pipeline.
Let us configure pipeline variables as below:
The ‘AdministratorUserName’, ‘AdministratorUserPassword’ should be filled with the actual user name and password for a user with a specific role (Administrator in this example). Similarly, you can configure multiple username-password combinations for different user roles in the variables. For example, ‘AccountantUserName’, ‘AccountantUserPassword’, ‘SalesmanUserName’, ‘SalesmanPassword’ etc., These values are actually going to be replaced in the variables user-name and user-password in the Postman environment configuration at run time. For the role-based tests, we are going to play a trick with the ‘Replace Tokens’ task. We will see that in a moment.
Debugging info: If you leave any variables with empty values, sometimes, the pipeline will complain that the variable doesn’t exist. So, you can see those filled with some ‘abc’ values.
Following are the final list of tasks in the pipeline:
Let’s check those tasks one by one.
Task 1: Command Line
Display Name: Install Newman
Script:
npm install -g newman
Task 2: Powershel
Display Name: Set the user and role
Type: Inline
Script:
# Write your PowerShell commands here.
Write-Host “Hooking administrator users for tests”
Write-Host “##vso[task.setvariable variable=UserName;]*<AdministratorUserName>*”
Write-Host “##vso[task.setvariable variable=UserPassword;]*<AdministratorUserPassword>*”
$Env:RoleName = “administrator”
This is the script that sets a place holder names for the user who has that role. In this case, ‘*<AdministratorUserName>*’ and ‘*<AdministratorUserPassword>*’ are going to be replaced as the values for variables ‘UserName’ and ‘UserPassword’ in the pipeline. This is going to be differ with different user roles on each pipeline. The values are going to be replaced with the actual values in Task 4 below.
Task 3: Replace Tokens
Display Name: Replace environment variables
Root directory: Choose your source code folder where the test and environment JSON files are placed under source control
Token prefix: #{
Token suffix: }#
This task will replace all the variables in the environment with the prefix ‘#{‘ and suffix ‘#}’. But this will leave the user name and password variables user-name and user-password with the temporary placeholder values set from Task 2. In this setup, user-name is filled with value ‘*<AdministratorUserName>*’ and user-password is filled with value ‘*<AdministratorUserPassword>*’. This will be updated with the actual values in the next step.
Task 4: Replace Tokens
Display Name: Replace test user credentials
Root directory: Choose your source code folder where the test and environment JSON files are placed under source control
Token prefix: *<
Token suffix: >*
This task will replace all the variables in the environment with the prefix ‘*<‘ and the suffix ‘>*’. In this case, user-name and user-password fields are filled with the actual values for the administrator role.
Task 5: Command Line
Display Name: Run API Tests
Advanced — Working Directory: Choose your source code folder where the test and environment JSON files are placed under source control
Script:
newman run TodoAPITest.postman_collection -e TodoApiDev.postman_environment.json — reporters cli,junit — reporter-junit-export Results\junitReport.xml
The above command line will run the tests and post the test results in xml format using junit.
Task 6: Command Line
Display Name: Publish Test Results
Test results format: JUnit
Test results file: **/*.xml
Search folder: $(System.DefaultWorkingDirectory)
In case you want to test more roles, you can just copy the list of tasks, create a new pipeline, configure the roles in Task 2. You can run all different role-based tests in parallel if your pipeline supports parallel runs (it is primarily based on the type of subscription you have).
We are done, when you run the pipeline, you can see the test results published under the pipeline.
