Intro
One of the achievements am proud of was to decrease the CI/CD pipeline of Camelan from 55 mins to 5 mins
Problem
In Camelan, we had to use the whole suite of Firebase for various stuff, real-time database for chat, Firestore for realtime updates for messaging and notifying the users if there are any new messages, performance benchmarking, some analytics, Crashlytics, and a lot more
This caused lots of overhead on our compile-time and of course, our CI/CD time
And of course, as any default dependency manager, your go-to dependency manager would be CocoaPods
Now, using CocoaPods is completely fine, until your build time expands to large numbers like 10-15 mins for clean builds, then you need to look for other solutions to handle your dependencies
We spent like 3 weeks trying to figure out how to cut our time and boost our iOS productivity as we were the slowest team regarding the feedback loop as any build we push over the CI/CD, we had to wait for at least 40 mins and at some time it goes to 55 mins.
Our CI/CD provider was AppCenter, so we had to do something about boosting our local compile-time and our pipeline
So let’s analyze our pain points:
1- Locally Xcode takes 15 mins to do a clean build
2- Any small change causes the whole project to rebuild (CocoaPods known setback)
3- CI takes a lot of time to finish the pipeline and send the build over to the QA team or to the store
Long build time and unnecessary rebuild
Our main pain points were due to that any small change in Camelan’s codebase triggered a recompile in CocoaPods, this caused us to wait for an unnecessary wait and caused us a lot of pain when tweaking things in the UI
So I decided to look for some solution to this issue, the key here was to convert the dependencies into binaries which are precompiled, and that would help Xcode to ignore those dependencies as they are already precompiled, so that would fix the issue of having to recompile dependencies with each small change
The question here is, what do we do to precompile the dependencies?
There were some options we tried like CocoaPods-binary but they were immature or didn’t work out at the end, so instead of twisting CocoaPods to fit our use case, we decided to go with Carthage since it was already designed to manage dependencies as precompiled frameworks and it was already supported by most of our used dependencies
So after we managed to migrate the project from CocoaPods to Carthage, we were enjoying far less compile and build times, locally it went from the 15 mins for clean builds to 3 mins, and on incremental builds, it was finishing in 24 seconds 🎉
So that helped us in our daily work and boosted our productivity, but we weren’t done yet
Our CI needed some love as well, but it was far trickier than the normal setup
Applying the same solution to App Center
Since App Center didn’t have much customizability, we had to somehow hack our way into customizing it with Bash scripts
In-app center, it detects what dependency manager you are using, is it CocoaPods or Carthage, then it reinstalls the whole dependencies again, we didn’t want that
What we wanted is to make App Center use our already precompiled dependencies without any work from his side, he would just go and compile the project without the need to compile any dependencies
So first thing to come to mind was to include the frameworks into the project’s repository on git
Of course, things didn’t go smoothly, as BitBucket had a limit of 2 Gigabytes on the size of the repository, and with Firestore only, we had around 800 Megabytes occupied (the whole repo size was around 1,800 MB)
So we had to think of another solution, we decided to upload the frameworks into another provider, We decided to go with Google Cloud hosting and that gave us exactly what we needed
Now that the frameworks are up and we clone them when needed
We had to somehow pull those frameworks when the CI pipeline works
Luckily, App Center allows for setting up a script after the project is cloned, which was a perfect place to pull the frameworks
But now there comes a question, how would the project compile after simply pulling the frameworks?
We had to reference the frameworks in the project without copying them, just set up a reference to their path and by that, Xcode expects the framework to be in that path, from this idea, we set up another script responsible for pulling the frameworks and putting them in the place where Xcode expects them to be, so that when it is time to compile, it can.
git clone $frameworks_repo
mv $frameworks_repo_local_name/Carthage Carthage
rm -rf $frameworks_repo_local_name
- We put here the git cookies of the Google Cloud Repository so that when the CI/CD fetches the frameworks, it doesn't get into any authorization errors
- we clone the frameworks' repo into the project's directory
- We extract the
Carthage file into the project's directory in the path that Xcode expects the frameworks to be
- we remove the repository empty directory
Now we try to run a test build on App Center, we found that it redownloads the dependencies again, which is not what we wanted, we wanted it to skip any building or fetching to the project’s dependencies, to do so, we had to disable any Carthage command that run, so we set up a symbolic link to Carthage in App Center’s bin folder
Then make it execute a fake script, like so.
echo "creating mycarthage directory"
mkdir ~/mycarthage
cd ~/mycarthage
echo "creating fake file to intercept appcenter's call"
echo "echo fake" >> new_carthage
echo "making it executable"
chmod a+x new_carthage
echo "linking interceptor with bin/carthage"
ln -s new_carthage /usr/local/bin/carthage
chmod 0777 /usr/local/bin/carthage
PATH=~/mycarthage/:$PATH
echo "trying calling carthage like app center"
echo /usr/local/bin/carthage
./installation.sh
This worked great until we found out that the app was crashing with the QA
After some debugging, we realized that the frameworks were not getting attached to the project after compiling and thus causing the app to crash
This was caused due to our script that was disabling any Carthage command, which also disabled the command that attached the frameworks into the iOS app in the build phases
So we had to modify our script that hacks App Center’s Carthage Commands to conditionally allow the command responsible for attaching the frameworks
mv /usr/local/bin/carthage /usr/local/bin/carthage.bak
echo "\
if [[ \"\$1\" == \"copy-frameworks\" ]]
then
/usr/local/bin/carthage.bak copy-frameworks
else
echo "fake"
fi
" > new_carthage
chmod a+x new_carthage
cp new_carthage /usr/local/bin/carthage
z
echo "try new edits"
/usr/local/bin/carthage
./installation.sh
With that done, we finally enjoyed faster build times both locally and our testers now receive builds more frequently and the bottleneck was resolved!
Conclusion
I hope this little challenge becomes useful to anyone, as this allowed us to boost our productivity 11 times after lots of frustration with the build time and a frustrating business, we are more productive than ever and our users who love our product a lot are entertained with lots of updates every week (sometimes twice a week!)
