I'm working to build an Android App through Visual Studio Team Services with the code hosted on GitHub.
Pain...
I'm going to not write up the long, painful, and unsuccessful time spent trying to use a Windows and Hosted Agent.
If anyone has this working - PLEASE let me know. I've abandoned the path...
EDIT: It appears it's a tracked issue and there is a work around
- PowerShell task to download SDK zip to Build.SourcesDirectory
- PowerShell task to extract the SDK zip
- PowerShell task to upgrade the SDK components
- Copy licenses to SDK from repository
- Create local.properties to make gradle task use downloaded SDK instead of ones provided in hosted machine
MORE BIGGER UPDATE
It was posted in the tracked issue but I will reproduce the comment here.
My build tasks to download, extract, update, and accept SDK licenses scripts content are inline:
PowerShell: Download Android SDK to sources directory
$url = "https://dl.google.com/android/android-sdk_r24.4.1-windows.zip"
$output = "$(Build.SourcesDirectory)\android-sdk_r24.4.1-windows.zip"
Write-Host "Download '$url' to '$output'"
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
Extract file: Extract the downloaded SDK zip file
Archive file patterns:android-sdk_r24.4.1-windows.zip
Destination folder:$(Build.SourcesDirectory)
PowerShell: Create local.properties file so the build will use downloaded SDK
$sdkLocation = "$(Build.SourcesDirectory)\android-sdk-windows"
$sdkLocation = $sdkLocation.Replace("\","\\").Replace(":", "\:")
$content = "sdk.dir=$sdkLocation"
$file = "$(Build.SourcesDirectory)\local.properties"
Write-Host "Create file '$file' with content '$content'"
$content | Set-Content "$file"
Run cmd: Update local Android SDK
echo y | $(Build.SourcesDirectory)\android-sdk-windows\tools\android.bat update sdk --no-ui --all --filter tools,platform-tools,extra-google-m2repository,extra-google-google_play_services,extra-android-support
Copy files: Copy accepted local build Android SDK licenses
Source:$(Build.SourcesDirectory)\android-licenses\**
Target:$(Build.SourcesDirectory)\android-sdk-windows\licenses
note: I have the copy of accepted license files from my PC included in my code repository. the content of this folder are: android-sdk-license, android-sdk-preview-license, intel-android-extra-license. these files are never need to be updated so far
Then after SDK is in place, you can run gradle tasks to build, test, etc. If you need clarification of above script please shout, I am happy to help :)
END UPDATE
But... No. I'm not gonna go down that road. It will make the build process more painful than it should be. A VM is a better option for me right now; though cost incurring. I'll take the cost vs the fragility those steps are going to bring into the CI/CD pipe.
Private Agent
Now I'm looking at doing a private agent; and what do you know - I have a Linux VM just sitting...
Instead of this simply costing me money to never look at it; I'm going to configure a private agent on this machine.
My Visual Studio Team Services (VSTS) is what I'm using to set up these builds. I could utilize VSTS as the GitRepo as well; but ... I like GitHub better as a repo... I wonder how much trouble my team at work is gonna hate on me for this since I'm pushing for all work code to be moved to VSTS.. hehehe
To get a private agent started... I need to hunt around for the agent definition... queue? Something to download the private agent configuration.
Found it
Then we click on the giant "Download agent" button
We're selecting Linux and ... Well... here's that screen
OK; It's downloaded onto... my mac - Not the Linux VM.
One of the things I'm planning to utilize in this project's build process is spinning up and down the VM... I don't need to pay for idle time.
Which means there's gonna be a bit of a hang time while the VM is woken up. This is likely to make the build times REALLY long; other options may need to be investigated.
And now I download the CORRECT agent - I have Ubuntu 16; not 14...
Now I scp it into the machine...
As you can see in the above image; it gives us the steps to get the agent running...
Got it up; and extracted; let's run stuff!
... and get errors...
Apparently there's some Pre-requisites required - Like specified here https://github.com/Microsoft/vsts-agent/blob/master/docs/start/envubuntu.md
OK; got those pre-reqs installed... It says git's required to be above 2.9.0; and now I have 2.11.0. It was originally 2.7.4; so YAY!
I ran ./config.sh and then ./run.sh and it's listening for jobs... But when I queue something up; it's complaining about capabilities...
OK, next I'll need to install JAVA... and the android SDK...
JAVA
I'm mostly following the commands here to get java installed
https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04
sudo apt-get update
sudo apt-get install default-jre
sudo apt-get install default-jdk
ANDROID SDK
Following the steps here to install android on a headless server
http://sblackwell.com/blog/2014/06/installing-the-android-sdk-on-a-headless-server/
wget http://dl.google.com/android/android-sdk_r23-linux.tgz
tar -xzf android-sdk_r23-linux.tgz
sudo mv android-sdk-linux /usr/local/bin
Additions to the ~/.bash_profile
export ANDROID_HOME=/usr/local/android-sdk-linux
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/platform-tools
Reinitialize the shell environment
source ~/.bash_profile
See all the shiny things -
android list sdk
install some stuff via
android update sdk --no-ui --filter 1,2,4
where 1,2,4 are the numbers of the items to install.
Install just one or two things; because it'll update and have higher SDK things to install.
I assume I'll get it wrong and need to update with more things once I get the build to run on it. Which will be interesting, it thinks it needs msbuild... :\
Trying the build again...
I have everything installed, so let's see what happens when I run the agent...
It found the android install... I think. Yea?
Who knows - I needed a new Build to fix some lingering requirements *cough*msbuild*cough*. Which is running on the Private Agent; so I have that going for the project.
Error when building
You have not accepted the license agreements of the following SDK components:
2017-02-01T06:11:06.3687760Z [Android SDK Build-Tools 25.0.1].
Gotta stay on top of the libraries being pulled in. The android tool won't always show everything. I have to run
android update sdk --no-ui --all --filter "build-tools-25.0.1" to get the license accepted for the build to progress. This might be annoying as things progress. shrug Small issue that's easy to resolve.
Note: If you don't restart the private agent... it won't build on it... ProTip.
Also; change the default agent to whatever one is hosting the private agent - makes it far easier. :|
Hmmm - It's building; lots of gradle stuff happening
The Build IS SUCCESSFUL!!!
OK; onto SonarQube!
... uhh... It's not awake.
Is SonarQube up?
The linux VM ipAddress changed; which makes it hard to set up my DNS... I've gone and given my VM a static IP. I assume this is gonna up the cost a smidge; but meh - needed.
A few hiccups getting SonarQube back online. I gotta give a shout out to CloudFlare; Easy Peasy DNS management.
Fantastic.
ANNNND... back to a 502... not sure what's happening here. The DNS dance isn't my strong suit.
Code Coverage
In the mean time; I'm seeing what I can do about producing code-coverage.
I'm really hoping it's just a DNS propagation issue for the domain and having to up date the IP address.
Hmm.... logs suggest it's a memory contention issue... Changed the memory allocation from 2G to 1G and BAM - We're back online.
Yay logs!
After some breaking; I've added Jacoco
apply plugin: 'jacoco'
jacoco {
toolVersion = "0.7.5.201505241946"
}
to the app gradle.build. Doing a commit - which ... won't trigger a build. Gotta turn that on - Next Time.
... didn't build.
Now I'm following the example here and setting up a Jacoco.gradle file.
The commit started a build; so yay to that.
Dinking around with some gradle clean up; and let's see if it builds... Or Code-Coverages... and nope...
hmmm... It's been a bit of putzing around...
This https://room-15.github.io/blog/2016/01/21/Static-code-quality-measurements-with-SonarQube-Jacoco-and-UnitTests/
seems to be pretty useful so far...
I'm waiting for some build locally... takes too long - Pushed. let's see what VSTS does
...
Still has issues running the build with jacoco set up. :\ I can run it locally...
(and sonarQube went bork again)
Gonna try to include jacocoTestReport in the gradle commands; see if it runs it that way.
I think I'm running into memory issues. I'm upgrading my VM from A2 Basic to A2-V2 It's got 4GB of RAM! .... ooooooooo
Let's see what this does for us... if anything...
OK - I have it building. I need to pass the jacocoTestReport in the args and not use the CodeCoverage section.
The SonarQube Section also fails...
Not sure why these things are all broke. The SonarQube step requires msbuild; so... bleh.
I'll have to configure it in the project... which I was trying to avoid; but Configuration as Code would say to keep it in and as part of the project. (I think); so we'll try it.
This is part of the process where I get to go away and come back to work on it more.
At work; I'm also attempting to get an Android app pumping data into SonarQube; and it works well. SO - I'm going to update my build process to also have sonar integrated into the build flow.
Work also requires a mac for our iOS app; which if done as a cloud hosted mac; we can configure it to build android as well (I think).
SonarQube.gradle
Time to configure the build process to auto pump into SonarQube. In the end; it will be an easier solution
I've configured sonarqube in a sonarqube.gradle file at the project level. This is so multiple projects can use it; though... not actually. ... I've moved it into the app level. Except for the project name and key; it's generic. I'm leaving them in for now; so it gets moved down.
Here's the sonarqube.gradle file. It's primarily configuration
// recommend to specify the flavor once and dynamically adapt paths to it
def flavor = "free" // flavor we want to have tested. Should be static
def Flavor = "Free" // flavor again, but starting with upper case
// noinspection is used to remove some "warnings" from Android Studio
sonarqube {
//noinspection GroovyAssignabilityCheck
properties {
/* Sonar needs to be informed about your libraries and the android.jar to understand that methods like
* onResume() is called by the Android framework. Without that information Sonar will very likely create warnings
* that those methods are never used and they should be removed. Same applies for libraries where parent classes
* are required to understand how a class works and is used. */
def libraries = project.android.sdkDirectory.getPath() + "/platforms/android-25/android.jar," +
"build/intermediates/exploded-aar/**/classes.jar"
property "sonar.host.url", "http://localhost:9000"
property "sonar.projectKey", "Android-HackerNews-Reader-${Flavor}" // some shortcut name
property "sonar.projectName", "Android HackerNews Reader ${Flavor}"
property "sonar.sourceEncoding", "UTF-8"
property "sonar.sources", "src/main/java,src/main/res" // first defines where the java files are, the second where the xml files are
property "sonar.binaries", "build/intermediates/classes/${flavor}/debug"
property "sonar.libraries", libraries
property "sonar.java.binaries", "build/intermediates/classes/${flavor}/debug"
property "sonar.java.libraries", libraries
property "sonar.tests", "src/test/java" // where the tests are located
property "sonar.java.test.binaries", "build/intermediates/classes/${flavor}/debug"
property "sonar.java.test.libraries", libraries
property "sonar.scm.provider", "git"
property "sonar.jacoco.reportPath", "build/jacoco/test${Flavor}DebugUnitTest.exec" // path to coverage reports
property "sonar.java.coveragePlugin", "jacoco"
property "sonar.junit.reportsPath", "build/test-results/${flavor}Debug" // path to junit reports
property "sonar.android.lint.report", "build/outputs/lint-results-${flavor}Debug.xml" // path to lint reports
}
}
The Project level build.gradle file needs to have the following added to the buildscript.dependencies section classpath 'com.dicedmelon.gradle:jacoco-android:0.1.1'.
The app build.gradle will need a small modification at the top to include sonarqube and jacoco
plugins {
id "org.sonarqube" version "1.2"
}
apply plugin: 'com.android.application'
apply plugin: 'jacoco-android'
apply from: 'sonarqube.gradle'
jacoco {
toolVersion = "0.7.8"
}
jacocoAndroidUnitTestReport {
csv.enabled false
html.enabled true
xml.enabled true
}
To get this to work with the build system; I need to either have some fancy arguments to the gradle script - OR; to do DevOps as code - I'm writing a script!
This is the core of a script I've been using for almost 7 years now.
I'm fairly certain there's better ways... it's what I got and I can do it quick; so...
#!/usr/bin/env bash
#
#: Title : Continuous Integration Build script
#: Description : This script is responsible for constructing and configuring the build of the app.
#
#: Author : "Quinn" <Quinn@QuantityAndConversion.com>
#: Version : 1.0
#: Date : February 2, 2017
#
##Prints the usage of the orchestrator
usage()
{
printf "\n"
printf "usage: bash %s <options>\n" $0
printf "OPTIONS:\n"
printf "\t-?\t\tPrints this.\n"
printf "\t-s\t\tSonarQube Server. This should be the \$(SonarQubeURL) property.\n"
printf "\t-u\t\tSonarQube Server Token. This should be the \$(SonarQubeToken) property.\n"
printf "\t-k\t\tSonarQube Project Key. This should be the \$(SonarQubeProjectKey) property.\n"
printf "\t-n\t\tSonarQube Project Name. This should be the \$(SonarQubeProjectName) property.\n"
printf "\t-v\t\tVersion Name. This should be built as \$(version.major).\$(version.minor).\n"
printf "\t-p\t\tProject. This should be the project to build as defined in settings.gradle.\n"
printf "\t-g\t\tGradle Flags. For debugging. Use as quotes for multiple flags.\n"
printf "\n\n\n"
}
SONAR_QUBE_URL=
SONAR_QUBE_USER=
SONAR_PROJECT_KEY=
SHORT_PLAN_NAME=
VERSION_NAME_OVERRIDE="0.0.0.0"
BUILD_PROJECT=
GRADLE_FLAGS=
DATE=`date +%Y%m%d`
#Parse the arguments
while getopts "s:u:p:k:n:v:p:g:" OPTION
do
case ${OPTION} in
s)
SONAR_QUBE_URL=${OPTARG}
;;
u)
SONAR_QUBE_USER=${OPTARG}
;;
k)
SONAR_PROJECT_KEY=${OPTARG}
;;
n)
SONAR_PROJECT_NAME=${OPTARG}
;;
v)
VERSION_NAME_OVERRIDE="${OPTARG}.${DATE}.${BUILD_BUILDID}"
;;
p)
BUILD_PROJECT=${OPTARG}
;;
g)
GRADLE_FLAGS=${OPTARG}
;;
?)
usage
exit 0
esac
done
echo ${VERSION_NAME_OVERRIDE}
if [ -z "$SONAR_QUBE_URL" ]
then
./gradlew clean \
${BUILD_PROJECT}:build \
${BUILD_PROJECT}:test \
${BUILD_PROJECT}:lint \
jacocoTestReport \
${GRADLE_FLAGS}
else
./gradlew -Dsonar.host.url=${SONAR_QUBE_URL} \
-Dsonar.login=${SONAR_QUBE_USER} \
-Dsonar.projectKey=${SONAR_PROJECT_KEY} \
-Dsonar.projectName=${SONAR_PROJECT_NAME} \
-PversionNameOverride=${VERSION_NAME_OVERRIDE} \
-Dsonar.scm.disabled=true \
clean \
${BUILD_PROJECT}:build \
${BUILD_PROJECT}:test \
${BUILD_PROJECT}:lint \
jacocoTestReport \
sonarqube \
${GRADLE_FLAGS}
fi
if [ "$?" != "0" ]
then
exit 1
fi
Some of these we will need to set up as variables in VSTS. We'll cover that shortly.
As can be seen; there are no specifics of the app in this file. I've tried to abstract that into parameters (just wait till I get around to template files) so it's easy to move between Android projects. Which means the BUILD command should be abstracted into a template... hmmm....
I've split the build command into if-else which I don't like; but otherwise the sonarqube task fails if it can't find the server.
This builds; minus the small glitch with no current local sonarqube. I'll be checking that before I spin up the VM.
Going through this - I find I dislike the app module being app. It's getting renamed to HackerNewsReader.
I think I have the script and sonarqube set up and ready for building. Just need to configure the custom variables to be used as inputs
I'm no longer using a Gradle Build Task; it's now a Shell Script Task.
For DevOps reasons, I prefer this. All script/build changes are reflected. If we want to change the build in a branch, it doesn't require breaking other builds or creating a new build. It's a very flexible solution - I've employed it previously and it works great.
I'm spinning up the VM to build on...
With this scripting; it might be pretty doable to do a hosted linux build agent. I'll investigate that later.
I also look at this setup for work purposes - If we go with SonarQube hosted in Azure; then the VM will be alive all the time; there's no(few?) downsides to also running a single build agent on it.
Scripted Build on Azure
The build script runs. Not currently exporting artifacts or plugging into sonar (which isn't running still...)
I'm using a secret token to auth against SonarQube
Running a build and sonarqube at the same time still encounters issues w/4GB of RAM. I'm upgrading to a 7GB machine... Over $100/m. BLEH... but if I shut down when not doing things; not too bad.
I realized that my projectVersion value in the script isn't correct.
This -PversionNameOverride=${VERSION_NAME_OVERRIDE} is a form for how I set the version for the Android app. I'll need that too; just haven't configured it yet.
The projectVersion got changed to be -Dsonar.projectVersion="${VERSION_NAME_OVERRIDE}" which matches the form of the other sonar settings.
The updated script is running on the updated VM and SonarQube appears to be staying up. There's even ~2 GB free. I think all I needed was 5GB; not 4; BUT... Wasn't available. Or was higher priced. This probably has some slow disk speed; but I'm going cheap for now. Wellllll.... Cheaper. Cheapish? I could get away with 2 machines and one of them be hella cheap for the build agent. The other dedicated for SonarQube.
The higher cost will be the machine capable of running SonarQube; which running the build isn't meaningful if SQ isn't up - So ... It's fine. This is PoC stuff; the VM going up and down is fine.
Hmmm... Errors during build
:HackerNewsReader:sonarqube
Invalid value for sonar.java.libraries
FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':HackerNewsReader:sonarqube'.
> No files nor directories matching 'build/intermediates/exploded-aar/**/classes.jar'
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 3 mins 46.534 secs
/bin/bash failed with return code: 1
Bash failed with error: /bin/bash failed with return code: 1
I think the JackCompiler screws up how SonarQube wants to do things. I'm ditching a couple entries that follow the build/intermediates/** pattern.
It's set up for CI so a build is already running...
Holy buckets - It worked!
Minus a small hiccup with the test report Reports path not found or is not a directory: /_work/1/s/HackerNewsReader/build/test-results/freeDebug
Time to tweak that and see if it builds again! Then onto artifacting!
The big test though; since the sonarqube task passed - Is it ACTUALLY for REALLIES in SonarQube...
Sorta...
The ProjectName stuff I mentioned... I may need quotes around it...
Or - I don't want to deal with it - Underscores. DEAL WITH IT!
Deleted that project; it'll get re-inserted when it runs again. Hopefully with the correct name. Or I can not give up and allow spaces...
I think that's best. Can't slack. Not the correct way - even in side things like this. Gotta hold to a higher bar.
In the arguments I wasn't quote wrapping the name.
Changed
-v $(version.major).$(version.minor) -p HackerNewsReader -s $(SonarQubeURL) -k $(SonarQubeProjectKey) -n $(SonarQubeProjectName) -u $(SonarQubeToken)
to
-v $(version.major).$(version.minor) -p HackerNewsReader -s $(SonarQubeURL) -k $(SonarQubeProjectKey) -n "$(SonarQubeProjectName)" -u $(SonarQubeToken)
Running again; we see the underscores that were poorly added
I mis-capitalized the test result path; so changed, pushed; and it's running.
And yes - I'm eyeing that "C" rating on Reliability. Not such a fan. :/
Finished and correctly named
but no coverage... which means it's not getting the jacocoTestReport properly.
I wonder if there not being class files is screwing it up...
Or there are class files and me commenting things out is screwing it up... So many wondrous ways I can do it wrong.
COVERED
It's all up and doing stuff!!! YAY!
Digging in; it's an interesting view
Some bugs; code smells; and a little lower coverage than I'd be hoping for. Not shown is the 0% duplicate code.
There's one or two that I'll fix; the rest are one's I'll set as Won't Fix.
An example it complains about is At most one statement is allowed per line, but 2 statements were found on this line. and it is trigged by instances of if(itemIds == null) { throw new IllegalArgumentException("itemIds cannot be null"); }
Yep; that's two statements. It's not changing. I agree in most cases 2 statements on a line is a problem. GuardClauses; I don;t see a problem.
Code coverage includes the Activity. Which we don't have counted tests on.
It does show a few area's I didn't do right. Tests to add.
Conclusion
With the VM shutting down; This post is going to come to an end. I fought through a convoluted process and finally got my Android project tied into SonarQube and displaying data!
Next steps will be fixing; coding; and seeing what other analysis packages I can pull in. I tried at work to pull a few in and SonarQube wouldn't launch; so I'll have to double check the 6.2 usability.
After SonarQube is polished up a bit; the code turned as Green as can be; maybe figure out some Appium and Espresso testing in the build? These things might happen... I don't know!!!
Anyway; once we get green; I'll look at having an artifact pulled out of the build. Maybe also some scripts to spin up and down the VM itself. That'll help with cost a bit.
Until next time...