Building Lineage for a supported device

Following the clues in the search for answers to my questions, I quickly stumbled across the build section in the LineageOS website. That section provides detailed instructions one needs to follow in order to build their own LineageOS image starting from source code. Armed with this guide, and with the source code needed for your device, it is straight forward to build a working image for your device.

Making a first successful android build

Now, as I have said earlier, I am still new to this, and this write-up’s aim is to help me make sense of all the things I have been learning in the last few weeks. So please bare with me as we go through this learning experiment together.

Figure 1: Sony Xperia Z

Before moving forward, this might be a good time to ground things more. Specifically, since we will be looking more into things like the Kernel, drivers and device trees¹, I need to specify the device I have been talking about up to this point, and which I will be using as a reference device for the most (if not all) of this series.

My beloved android device is a Sony Xperia Z, code-name: Yuga². On its release back in 2013 the Xperia Z was running Android 4.1.2 Jelly Bean, which itself had been released the year prior. Sony kept issuing updates for the Xperia Z for two years after its release. The final update was on May 2015, and pushed Android 5.1.1 Lollipop to the device. Currently, this is the latest officially supported android release for the Xperia Z. As of this writing, this version is SIX releases and FIVE years behind on the latest Android version…

As I have mentioned in my previous article, my Xperia Z used to be supported by Lineage, and was only dropped a couple of years ago. Since LineageOS is an open-source project, this means that its source code is available for anyone interested. I thought it would be a great starting point to simply try and build a working (but outdated) android image for my device using the available Lineage source for it.

The Lineage wiki includes a build guide for the yuga. The guide is very straight forward and quite easy to follow. In the remainder of this article, I will try to go through the guide step by step, my goal is not to regurgitate what is well explained by the Lineage team in their guide. However, I would like to provide some additional explanations as to what the different steps achieve and to give a short glimpse into the behind the scenes of the build process. The best way to take advantage of the following sections is to read along in the LineageOS build guide for the yuga, and come back here for extra notes on each step. I have tried to make the explanations here as generic (device-wise) as possible so that you can eventually follow on the build guide of your own device if it is supported by Lineage.

Prerequisites

The main prerequisite is a build-capable machine. Building programs from source is generally a resource-heavy task, and you will need a machine that would be capable of it. In practical terms, this translates to:

• a 64-bit CPU.
• a recommended minimum of 16GB of RAM.
• storage capacity of 200GB (preferably SSD for the R/W speed gain).

Additionally, the guide recommends you have the phone you would like to make a build for on hand, with a working Android or Lineage OS. This is because in a later step in the build process, you will need to extract certain components (mainly binary BLOBs, more on these later) from the working OS on your phone. However, you can also extract these components from an image file (either for Lineage or Android) if you have one. When I made my first build, I used a previous image that had been made available on the LineageOS website, back when my device was supported. Ultimately, this sounds counter-intuitive: if one already has a working LineageOS image, why go through all the trouble of rebuilding it? I have two reasons for doing that:

• Firstly, the newly built image will not necessarily be identical to the old one. We will use the old image to extract some components we need in the new build. In my case, I used an image of version 14.1 to extract binaries I went on and used to build an image of version 15.1.
• Secondly, even if the newly built image would have been the same as the old one (which for my first try it admittedly was), this whole experiment is a learning exercise, so I want to go through the trouble in order to learn why it is necessary.

The guide mentions needing platform-tools, i.e. adb and fastboot. These would be needed if you have to interface your phone with your build machine. This will be the case if you need to extract binary blobs from your phone before you start the build, and if you want to flash your final image onto your phone after the build is done.

Step 1: Dependencies and build packages

Because the Android project involves multiple programming languages (mainly C, C++ and Java), you will be needing compilers and other needed tools for these languages. Additionally, in this step we create the directory which will be our root source directory. I will follow the LineageOS build guide and will also be using ~/android/lineage as a root source directory. I will be referring to it as such for the remainder of this article.

Step 2: Downloading repo

This step warrants a short explanation. As you probably know by now, the Android project is a very large one, spanning millions of lines of code, and involving multiple companies, organizations, and developers. The work of all these entities is organized into a multitude of projects that are hosted on different repositories all over the internet.

The repo tool is a python script that plugs on top of Git, and that handles the complicated task of going around the internet gathering all the projects that are required by the android project. The tool was developed by google in order to automate parts of the development workflow of the project, and allows for much more than simply downloading remote repositories.

What we do in this step, is download the repo tool from google, and add it to our $PATH so that we can call it later on when we start downloading the source code.

Step 3: Downloading the source code

In this step we use the repo tool from the previous step in order to download the source code for our build. This is done in two steps:

1. We initiate our root source directory to the LineageOS remote repository.
2. We use repo to sync up our source directory and download source code from all the needed remote projects.

Initiating the root source directory is accomplished running the following command from within your root source (in my case ~/android/lineage):

$ repo init -u https://github.com/LineageOS/android.git -b lineage-15.1

What this command does is initiate our source directory to the branch lineage-15.1 of the LineageOS project. Concretely, repo accomplishes this by creating a .repo/ directory in which it stores its manifests. I will leave a detailed discussion about manifests for the next installment of this series. However, for now, suffice it to say that manifests are XML files which are used by the repo script to store remote repositories and projects, where to find them online, and where to store them locally once downloaded during the syncing step.

Taking a look into your .repo/ directory is a great way to get an initial idea of the different projects involved in a successful build, and of their whereabouts online. Using the tree command to see what the .repo directory contains, reveals the following:

$ tree -L 3 .repo/
.repo/
├── manifests
│   ├── default.xml
│   └── snippets
│       └── lineage.xml
├── manifests.git
│   └── [...more files..]
├── manifest.xml
└── repo
    └── [...more files...]

In the following is a short description of each of the elements shown above. As you can see from the output snippet, I did not include all files listed in the output of tree as that would have been too much, and those files have no interest for us. Additionally, I am introducing the different files and directories in an order that makes for a better understanding and that is different from the alphabetical order used in the output:

• manifests.git: Is a bare repository directory which contains the cloned content of the remote repository given in the -u argument (in my example: https://github.com/LineageOS/android.git)
• manifests: Contains the checked-out content of the branch given in the -b argument (in my example: lineage-15.1)
• manifests.xml: Our first example of a manifest file. In this case it is a very rudimentary one which simply points to manifests/default.xml.
• default.xml: Contains information about all projects needed to be synced for the build and their remote locations. In my case there are multiple hundred remote projects defined in this file.
• lineage.xml: Contains information and location of repositories of applications provided by LineageOS. This manifest is pointed to in default.xml.
• repo: Simply contains the cloned content of the repo repository including: libraries, modules and python files used in the repo tool.

More details of the different manifests will be given in upcoming parts of this guide.

The initiating step of our local source tree takes little time since only the manifests are downloaded. The actual source code is downloaded in the syncing step. This is achieved by running the command:

$ repo sync

What this command does, is instruct repo to go through all the different projects defined in all the different manifests which were downloaded in the previous step, and download their remote content to our local source directory. This step will take quite some time depending on your internet connection. There are hundreds of projects to sync up with, and as of this writing, the total volume of files to download is close to 100GBs.

Other than waiting, there isn’t much more to this step.

Step 4: Preparing the build environment for your device

As it stands now, the source code included in our source tree is not enough for a successful build. If you have been paying close attention, something that might have tipped you off about it, is the fact that there was no mention of any commands specific to the device we want to build for. The source code we have downloaded so far is generic, and excluding some google devices included in the remote AOSP repository, it doesn’t contain any device specific components.

A very important thing to make clear at this point is that for a successfully build, you will need the following ingredients:

1. The ROM source code: This is the source code of the OS you want to build, and from my understanding, is by far the largest component of your source tree. This code can vary depending on which OS or ROM you would like to build. These can be any one of the multitude of custom Android distributions in existence.
2. The device tree: This is a hierarchy of a multitude of files organized in multiple directories, which specify parameters specific to your device. These parameters include but are not limited to: Details of the device (screen size, memory, storage, sensors, cellular bands, etc), list of binary files needed for the build (in addition to their location in the source tree), and other build parameters. Although some commonality can be found between devices sharing a similar architecture, a device tree is generally unique to each device.
3. The kernel: As you might know, the Android Kernel is based on the Linux Kernel. Now, if you are somewhat familiar with Linux on the desktop (or the server), you might know that generally speaking, the same Linux kernel will run on almost any computer (given we remain within one chip architecture). This is because as it currently stands, the bulk of the Linux kernel is made out of driver code³. This ensures that the kernel will overcome the diversity of computer hardware out there (think network interfaces, graphic cards, and all sort of peripherals). Sadly however, this diversity takes on a whole new meaning when it comes to the smartphone world. The sheer diversity of sensors, SoCs, screens, radio chips, and additional manufacturer specific components, all of which are subject to yearly change, makes it impossible to have a one-for-all kernel. This is why, the kernel will also be unique to your device.
4. The binary blobs: In addition to the open drivers included in the kernel code, multiple closed source drivers will almost certainly be needed for your device. These are additional drivers that are unfortunately proprietary and kept as a close secret by your device manufacturer. Because they are proprietary they are kept as opaque binary blobs (Binary Large OBjects), which are added as-is to your image later in the build process.

In order for your build to start, run successfully and for the resulting image to boot on your device, you will need each one of these four components⁴. So far in our steps, we have only gathered the first component (the ROM source code) which is independent of our device, in order for us to be able to start our build we will need the other three. Luckily for us, since my device was once supported by LineageOS, the Lineage team did most of the hard work for us: The device tree and the kernel are both maintained on the LineageOS repositories. All we need to do to add them to our source tree, is to use repo to query them from the LineageOS repositories. This is achieved with the commands:

$ source build/envsetup.sh
$ breakfast yuga

We first source the build/envsetup.sh which contains a multitude of commands and parameters we will need for the build. Among the commands defined is the breakfast command which we call with the code-name of our target device as an argument. The breakfastcommand will only work for devices officially supported by our ROM (in our case Lineage). This is because what it does is look through the Lineage repository for the remote locations of both the device tree and the kernel in order to sync them up to our local source tree. When done, both these ingredients will be available for our build.

Now all what is left is the final ingredient: The binary blobs. As mentioned earlier, these are closed source drivers, and the only way to obtain them is to extract them from a working android image. This can be done by connecting your android device to your build machine through adb and running the extract-files.sh script which breakfast would have added to our source tree when we run it earlier. Another source of the binaries would be to extract the files from the installable zip of a working ROM. If neither of these methods is possible for you, you can always try your luck and check if anyone online has done the work for you and extracted the binaries for your device. A nice place to start looking is the Muppets project which gathers binaries and device trees for a wide range of devices. In any case, no matter the method, after extracting the binary blobs we can add them to our source tree, thus putting the last piece we needed in place before launching our build.

Step 5: Start the build

If you went through all previous sections without a hitch, this remaining step should be a breeze. Before running the build command the Lineage build guide recommends multiple caching settings and compiler configurations. With that out of the way, we position ourselves in the root of our source tree and launch the build with the two commands:

$ croot
$ brunch yuga

Depending on your machine the build might take a couple to several hours. When the build is done, an installable zip image file as well as a recovery file will both be waiting for us in the $OUT directory. These can readily be flashed into a device, or plugged into an Android emulator if you want to test them first.

What we have learned and what is to come

Through the steps mentioned in the previous sections we managed to build a working image for our device. However, this was only an educational exercise, and we haven’t really achieved anything of great value. I say this because we have only built a working image of a ROM that already supports our device. The hardest parts of a build: Constructing the device tree, finding a Kernel (perhaps even having to patch and re-build it), and extracting the binaries, were all done for us beforehand by the Lineage team. All we did was follow in their foot steps, and manually do what their build servers do constantly every week for every device that is officially supported.

Nevertheless, through this exercise we have learned a lot about the build process, and more importantly, about the different ingredients we need to have if we want to successfully build and image for our device. The real test comes when we have to gather the different ingredients by ourselves, which we will have to do if we want to build a working image for a device that is not supported by the Lineage project. My goal through the upcoming installments of this series will be to do just that. The next step for me will be to successfully build a working image of LineageOS version 17.1 which was never officially supported for my device. This will be a middle ground between the learning exercise that we just accomplished and the dreadful task of having to start from scratch and gather ingredients for a device that was never supported by LineageOS.

Additionally, I do have further plans in mind for how to push the envelope a bit further and reach a bigger milestone. For now however, I hope this will do.

1. Don’t worry, all will be explained in good time. ↩︎

2. Manufacturers use different names for their devices for internal use, these are called code-names. I will be using “Xperia Z” and “Yuga” Interchangeably. ↩︎

3. I found out this cool project while looking for references of my assertion here. ↩︎

4. Well more like three, the ROM source code is a given. ↩︎