Testproject OpenSDK with Bitrise CI

Spice up your mobile app tests using TestProject OpenSDK platform, Bitrise CI and SauceLabs

7 min readApr 17, 2021

Dear readers,
I truely value and respect your time, therefore, here is a short glimpse of this article to know whom this is meant for
1. If you are passionate about test automation and keep on exploring new solutions for their test stack, then you are landed at the right place 🙂.
2. If you are wondering how to run your tests continously with your CI/CD pipelines, then let’s get started.
The article focuses on how we managed to place TestProject with our current CI setup.

Journey before adopting TestProject

Our use case was to run mobile app tests using appium but to integrate the tests with our CI platform was way more challenging. We wanted to avoid, false failures, flaky tests due to test stack issues, delays in receiving results.

To overcome this we explored and implemented some solutions:

Initially, we planned to run tests on our CI environment itself which is Bitrise, but we had to set up the entire stack before running the tests such as install and run appium server, create and launch emulators.
After facing issues in our first approach we decided to move the execution from Bitrise CI to AWS Device Farm. This introduced another challenge for us to refactor the existing framework to make it compatible with the device farm’s acceptance criteria. After all the hard work it was working well for us, but device waiting time and lack of one single dashboard for reports were some of the challenges we were still facing.

Now the biggest question was how we should improve our test execution and serve realistic results to our developers 🤔.

Here comes the TestProject, the answer to all our questions, a platform that absorbs all your test infrastructure needs into a single agent and allows you to run your tests seamlessly on device farms like BrowserStack and SauceLabs.

Journey to explore TestProject begins…

The best part about TestProject is that it’s completely free to use, you just need to download the agent on your machine and sign up for the TestProject account that’s it. This will eliminate the need of managing appium server, web drivers, or any test infrastructure dependencies.

Use Case 1: Run our current codebase with TestProject

We were running a Cucumber based project with Java+TestNG+Maven for our mobile automation. To make it more compatible with TestProject we started with the integration of TestProject OpenSDK for Java and moved our project from Maven to Gradle (this was a personal choice, but maven support is also available with TestProject). All the integration information is available on the official GitHub of TestProject.

testproject-io/java-opensdk

TestProject is a Free Test Automation platform for Web, Mobile and API testing. To get familiar with the TestProject…

github.com

dependencies
{
   implementation 'io.testproject:java-sdk:0.65.4-RELEASE'
}

Use Case 2: TestProject integration with Bitrise CI

TestProject doesn’t provide direct support to Bitrise, however, the best part is they provide you with a bunch of REST APIs which can be executed as a part of your CI workflows. The entire API documentation can be found here:

TestProject API v2

Edit description

api.testproject.io

Before you try out any of these API’s please make sure you generate an API key from your account.

Create API Key at TestProject’s Integration section

Once we got enough clarity on both the use cases we decide to pick SauceLabs as our device farm for all the mobile test executions because:

TestProject supports SauceLabs integration :).
And, we were not able to achieve the best results by using AWS Device Farm.

Time to connect all the dots

Now the time was to connect all the pieces so that both our use cases can compliment each other. We started this integration process by picking the API’s which will help us to build a CI pipeline for test execution.

Before API integration part, it was important for the implementation to upgrade the JAVA version.We were using JAVA8 and I know few people might find this funny in the age of JAVA17, but trust me this was very stable and we never got a chance to upgrade. Again, thanks to TestProject for this as they have SDK support starting from JAVA11.
For TestProject to run the coded test you need to upload your project in the form of a JAR file, which was a sign that we need to use an API that will upload our code as JAR from Bitrise.

POST:https://api.testproject.io/v2/projects/{projectId}/test-packages/{packageId}Authorization: Test Project API key (header)
fileName: Name of the JAR file (Query Param)
platform: Platform name iOS or Android (Query Param)

3. Next thing was to run the tests, but before that, we need to add those tests in a TestProject job, which we managed to complete in two phases:

Step i): Get the list of test IDs that need to be added to the particular job. Use the below API that will help you to get your test IDs

GET:https://api.testproject.io/v2/projects/{projectId}/test-packages/{packageId}/tests
Authorization: Test Project API key (header)

Step ii): Once you get the Test IDs, you need to add those tests to your TestProject job.

POST:https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/tests
Authorization: Test Project API key (header)
Body:
   {
     "position":0,
     "tests":[
               "{$TEST_ID_1}"
               "{$TEST_ID_2}"
               "{$TEST_ID_n}"
             ]
   }

4. Finally, it’s time we tell TestProject to run our job on SauceLabs. We will be using run a job API, where we will define our agent (SauceLabs) ID to run our tests.

POST:https://api.testproject.io/v2/projects/{projectId}/jobs/{jobId}/run
Body:
{
   agentId: "{SAUCE_LAB_AGENT_ID}"
}

Note: To integrate SauceLab with TestProject you can follow the official documentation.

Sauce Labs Integration

allows users to run tests in the cloud on 1,000's of different browsers, operating systems, and device combinations…

docs.testproject.io

Architecture Diagram

When Bitrise meets TestProject

This section will be more familiar for the folks who have worked with Bitrise in the past, however the entire flow is more towards calling API’s at different stages. We can configure something similar with other CI/CD platforms like Jenkins, CircleCI, Buddybuild, etc.

Git Clone Repository: Clone your test automation code base to Bitrise using this step in a folder at run time or you can directly clone into the Bitrise source directory as well.

2. Set JAVA_HOME to JAVA 11: This step is required because stacks on Bitrise might use JAVA 8 as the default JAVA version. For TestProject OpenSDK to work, we need to switch the JAVA version to 11.

jenv global system
export JAVA_HOME="$(jenv prefix)"
envman add --key JAVA_HOME --value "$(jenv prefix)"

3. Generate Test Package as a JAR file: Since we need to upload our coded project in JAR, we will be running another script step using the Gradle command.

echo Switching to Project Directory.......
cd $BITRISE_SOURCE_DIR/PROJECT_DIRECTORYecho Performing Gradle wrapper task......
gradle wrapperecho Generating .JAR file.......
./gradlew clean jar

4. Upload JAR file to TestProject: We will be calling the POST API to upload the JAR file in the form of a CURL command.

curl --location --request POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID?fileName=$TEST_JAR&platform=Android" \
--header "Authorization: $TEST_PROJECT_AUTH_TOKEN" \
--form 'name=@"/$PROJECT_DIR/build/libs/APPLICATIO_NAME.jar"' \
--form 'content-type="application/octet-stream"'

5. Update TestProject Job with Test IDs: As mentioned earlier, we need to update our job with the test IDs we want to run with that job.

TEST_ID1=$(curl -X GET "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID/tests" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" |  jq -r '.[0].id')echo ""TEST_ID2=$(curl -X GET "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/test-packages/$TEST_PROJECT_PACKAGE_ID/tests" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" |  jq -r '.[1].id')echo ""

Update TestProject job with another API call

curl --location --request POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/jobs/$TEST_PROJECT_JOB_ID/tests" \
--header 'Content-Type: application/json' \
--header "Authorization: $TEST_PROJECT_AUTH_TOKEN" \
-d '{"position":0,"tests":["'"$TEST_ID1"'","'"$TEST_ID2"'"]}'

6. Run your job on SauceLabs: This will be the final step of our workflow, where we will be executing our TestProject job on SauceLabs. Let’s use another CURL command to run the job.

curl -X POST "$TEST_PROJECT_URL/projects/$TEST_PROJECT_ID/jobs/$TEST_PROJECT_JOB_ID/run" -H "accept: application/json" -H "Authorization: $TEST_PROJECT_AUTH_TOKEN" -H "Content-Type: application/json" -d "{ \"agentId\": \"$TEST_PROJECT_AGENT_ID\"}"

Important Note: We are uploading our APK to AWS S3 bucket from the build stage of our workflow and configured the S3 bucket path as appium capability in our code.

cap.setCapability(MobileCapabilityType.APP, AWS_S3_BUCKET_APK_PATH);

Well, that’s all we have to do for an E2E pipeline for mobile test automation.

Please let me know in case of issues you face while designing the workflow. Also, I’ll be happy to know if anyone has any better approach to achieve this. Improvements & Learning are the keys to success. 🙂