Best practices using compilers in Android Studio (Google I/O ’18)

Good afternoon. I’m Jeffrey van Gogh. I’m a tech lead manager
on the Android Studio team where I’m in charge of
the D8R8 Compiler Project. So last year a lot
happened in compiler space. We added incremental
dexing, which makes your debugger build faster. We added desugaring of
Java 8 language features. So you can now use Java
8 features like Lambdas in older versions of Android. We added a new dexers, which
compiles your Java bytecode to Dalvik bytecode that
runs from the art runtime. We added a new shrink
and optimizer called R8. And we added sculpin specific
optimizations into that as well. And then today, we also
have the Android app bundle. And so I wanted to talk
to you about all of these, show how it works inside
Gradle, and give you some tips of things you
need to know when you start using these new tools. So let’s first start looking
at incremental dexing. So here, you have a very,
very simplified version of what happens in Gradle
when the compiler runs. So first we run either
Java C or Kotlin C to take your Java
or open source code and generate Java bytecode. And then we run DX that
takes a Java bytecode and produces Dalvik bytecode. Now the nice thing
that in Gradle, because of Gradle
and Jet Brains’ work, Java C and Kotlin C are
actually incremental. And that means that if you
change one source file, it only compiles that one
source file and potentially any sources that have different
semantics based on that change. Now unfortunately, in
Android Studio and Gradle, before 3.0, DX didn’t
do any incremental work. So it still took all
your Java class files and compiled each one of
them to Dalvik bytecode, even if those class
files haven’t changed. So in Android Studio
3.0 and above, we actually change it so
that we can be incremental. So we split the DX
and do two steps. One that takes the Java bytecode
and compiles it to the Dalvik bytecode. And we do that per class file. So we actually generate now
one dex file per class file. And that way, that step
can be incremental. And then after that, we take all
those dex files and merge them into a single or multiple
dex files if needed. And the reason this works
is that most of the time spent in dexing is actually
in the compilation phase, where we take the
Java bytecode, which is a stack-based machine,
to Dalvik bytecode, which is register-based. And then the dex merging is more
or less like a fancy concat. And so that’s a lot faster. And so we enable it by
default in your debug builds. In release builds,
we don’t do that. And so you pay a little bit
extra for the initial build, because we need to
generate more files. But then each incremental
build that do you afterwards, it’s a lot faster, just because
we have to do less work. There are some things you
need to be aware of, though. So Java C and Kotlin C are only
incremental when you are not using annotation processors. Any time you enable
annotation processors, these annotation
processors can reach into any part of your source. And so we cannot make
that incremental. Now, Gradle is working hard
to make that supported. So in Gradle 4.7, they
introduce a preliminary support for incremental
annotation processors. This requires some work by
the annotation processor to support that, because
annotation processor needs to tell Gradle how
incremental it can be. And so what I’d
like to ask you all is if you are an annotation
processor writer, look at the stuff
the Gradle is doing and see if you can support that. And if you’re using
annotation processors, please read out to the
developers of those that they should look at
this, because it will really, really speed up your bills. So let’s go on to desugaring. What is desugaring? We hear from all
people, they have it many times that they want to
use modern Java features, Java 8, like lambdas, default
methods and interfaces, try-with resources, et cetera. Unfortunately, a lot
of these features require new bytecode and
language APIs to support them. And of course, a lot
of Android devices out there run older
versions of the Dalvik VM that doesn’t yet support these. And the developers really
want to use these features, especially as they start
using new frameworks like re-active extensions. Now, I’m not sure exactly
what these libraries do, but they use a lot of callbacks. And it would be much nicer
if you can use lambdas there. And so what we do
with desugar is we take the bytecode
and calls that are generated by these new
features in the Java compiler, and we convert them
to something that is supported in the old system. So for instance, if you
use a lambda in Java 8, we can take that and
replace it with a class as if you were to
handwrite it, so that you don’t have to handwrite
it and we do it for you. So let’s take a look at
how that works in practice. So let’s switch to the demo. So here for Android
Studio project, that is just created by
following the wizards and selecting a basic activity. And so if you go to the module
and the module settings. You’ll see that I have set
the source language to 1.8. And now it will allow me to
write Java 8 language features. And if you do this,
Gradle automatically figures out that it needs
to run desugar for you. And so here I have some code. I have a floating
action button that I want to hook up some code
to, that when you click it– and all because I
use Java 8, I don’t have to write new
OnClickListener and implement the
whole interface. Instead, I can
just write a lambda and have it in be invoked. So if I go and look at the
output of the Java C compiler, there is a main
activity of class. And I can copy that path. There is a tool in the Java
JDK called Java P, which allows you to take a class file
and look at the bytecode that’s in there. And so I’ll run that. And because it generates
a lot of output, I’ll pipe it to a file. And then I’m opening it in my
favorites ID, Android Studio. And so if you haven’t looked at
Java bytecode, don’t be afraid. It is still pretty high-level. It’s kind of readable. The only thing that’s not there
is for loops, if statements, and so on. But like, if you
read through it, it’s still pretty
understandable. So just a lot of constants that
we’re just going to skip over. And then here, we have
your onCreate method that we were looking at. And in first call here, you see
that there is a virtual call to the set OnClickListener. That’s the thing where
we passed our lambda to. And then see that the
argument before that is this invoked
dynamic construction. And it tells us that it’s going
to pass this OnClickListener. So what does invoke dynamic? It’s a nice feature in the Java
VM that is kind of reflective. So instead of the VM immediately
invoking your method, it allows the application
to provide a hook in there and dispatch the
method anyway you want. And so in Java, they have these
things called metafactories that they use to
implement these features. And so there is the
specific LambdaMetafactory. And you see that here in
the bottom of the file. Let’s get this argument to view. And then it passes to this
lambda onCreate method, which we can see here as well. And if you look carefully,
you see that it actually has the snack bar codes that we
had in the body of our lambda. So what is going on here is
that the first time the app is run on the JVM,
it knows it needs to call this LambdaMetafactory. And that thing will
actually generate the class that implements
the interface on the fly. And then it will call that
for the rest of the program. Now the problem is that
that takes time at runtime. It adds more memory at runtime. So we don’t do that
on an Androids, even in newer
version of Android. And of course, in old
version of Android, they don’t know about this
invoke dynamic construction or the metafactory. So instead, desugar
will take care of this. So let’s take a look at what’s
happened in this project and when you build
it using desugar. So I’m going to open the APK. There is this tool
in Android Studio 3.0 and above called
the APK Analyzer. It allows you to
look inside the APK, both for file size
of your resources, but also to see what’s
inside the dex code that’s going to run on the device. So here I see all the
packages in the dex codes. And I’m going to navigate to
my main activity, and then the onCreate method. And I’m going to
say show bytecode. So you see bytecode that is kind
of similar to Java bytecode. There’s a couple of differences. Instead of using stack-based
machines, we have registers. But if you are not familiar with
that, don’t worry about that. So at the end of the
method, we see the same call to the set OnClickListener. But the big difference is
that one line above, it doesn’t show invoke
dynamic or invoke custom as it would be on Android. Instead, it calls this magic
class dash dash tilde lambda, and then it gets the
interface field of that. So let’s take a
look at that class. So you see that the class
is right along there. And so what we see
in the class is that it implements the
OnClickListener interface. It has a static field instance. And then it has the onClick
method for the OnClickListener interface. And all it does is call the
generated method it contained to method body for lambda. And so now there is no Java 8
features, no Java 8 bytecodes in this code. And we can execute it
on any Android version, even as low as Ice
Cream Sandwich. So let’s switch
back to the slides. So this is how that is
integrated into the Gradle build system. After the Java C
compiler runs, we run the separate
process called desugar. What it does, it reads
the Java bytecode. It takes out all these functions
that are not supported, emits new bytecodes, and
then we pass it on to DX. And so the rest of
the pipeline doesn’t have to know anything
about desugaring. So this is nice. You can use new Java 8 features. There’s a couple
of things you need to be aware of if you do your
own bytecode transformations. So there’s people who do their
own bytecode transformations for code injection, crest
reporting, et cetera. Because we you run
desugar, we run your bytecode transformations
after desugar, which means that you see all our
crazy patterns like the dollar lambda codes when you’re
doing your own processing. So be aware of that if you’re
doing your own bytecode transformations. So let’s move on to D8. D8 is our new dexer. As I said, Android, it
runs Dalvik bytecode, not Java bytecode. And dexer is the tool
that takes Java bytecodes from the stack-based
machine and converts it into Dalvik bytecode,
which is register-based. We had this tool before called
DX, but it’s pretty old. People had problems with it. And so we decided to build
a new version called D8. And so the reason
we build it is we want to have faster completion,
because everybody always wants faster builds, we want
to generate smaller code, and give people
better diagnostics. By better diagnostics, I
mean both the error messages that you get when you run the
compiler as well as better debug information
that when you’re running your app
in the debugger, that you have a better
understanding of what is going on. So how is D8 integrated into
the Gradle build system? It’s actually quite
similar to what DX is. We just swap out the X for D8. The interesting thing there
is that in Android Studio 3.2, we also integrate the
desugaring step into D8. So that saves us a round trip
between reading and writing the class files. And so it will
provide more speed up. The side effect of
that, though, is that if you’re writing your
own bytecode rewriters, we now run them
before D8, which means that your bytecode
rewriters have to support the Java 8 language. So let’s look at the
demo of D8 in action. So here, I have another
project that I just created using the Project Wizard. And then because I’m
using Android Studio 3.2, D8 is already
enabled by default. So I went into the
Gradle property files and I explicitly
disabled D8 because I want to show you the behavior
of DX before we use D8. So in my main application,
I added some code to my OnClickListener to have
the snack bar print a custom message. And then I have a
method get message, and I set a breakpoint. So let’s look at
that when I run that in the emulator on the debugger. So the app is running. And I’m going to hit the button
so that I hit the break point. So I initialize x to be the
length of the empty string. So that should be 0. And so in the if
statement, I expect to step through the true case. And of course,
that’s what happens. And I’m going to step further. This is bizarre, right? I don’t expect my
code to evaluate both the true and false case. So that’s kind of weird. So let’s see what happens if I
run and see what the output is. So luckily, the
output on the screen is hello there, what I expected. But there was something
weird going on. But what was going on? So let’s remove this,
and sync our Gradle build so that we are switching to D8. And let’s stop that
and redeploy it. So while this is going on–
so in DX we have this issue. And it was actually a very
high start bug report. And the reason it was happening
is that not only is the VM very different between
stack and register, the way that debug
information is stored in Java, the class files, and
Dalvik is very different. In Java, it starts
with the instructions, and in Dalvik, it’s
a state machine. And so we had to translate
both from sector register and the debug information. And so sometimes
information got lost. So it might end up with a single
return statement in the Dalvik bytecode, and then
we couldn’t map that in the debug information. In D8, we track all the
debug information carefully. And we have a whole
system of assertions to make sure that we don’t
lose debug information. So let’s hit the button here. We’re hitting the
break point again. We step through. We’re getting to
the truth branch. And we jump out of it. [APPLAUSE] Thank you. So let’s switch
back to the slides. So we have done a
lot of work on D8. It has better debug information,
but it’s also faster. So here are some data
around build time on the Google Nest app. So we shipped D8 as a preview
in Android Studio 3.1. There were not always faster,
but we had a lot more work since 3.1. And so in 3.2, on
average, we’re about 16% faster in clean builds. And of course,
incremental to builds, the delta is smaller because
there is less code to compile. So still, 16% is pretty nice. So D8 is already widely used. If you’re using Android P beta,
that was a release yesterday, Android P is completely
built with D8. The Google Docs app is
already built with D8, and then Google Photos is
right now in Canary using D8. And many more Google
apps will follow soon. So let’s move over to R8. R8 is our new shrinker. So why do you want a shrinker? So most people who
build apps, they use a lot of libraries like
Google [INAUDIBLE],, Apache Commons, RX Java,
and you usually don’t use that full library. The IP might use maybe
10%, 15% of that library. Yet, if you ship it as is, you
would be shipping all that code that you don’t use. And application size
is important, right? People don’t want to pay
for it in their bandwidth. It uses disk space
on the device. And so the smaller
app, the better. And there was a
previous solution to this, the ProGuard tool. But we hear from people that
they have issues with it. It was taking a long time. The code wasn’t as
small as they wish. It didn’t really
understand Android. And so we invested in
building a new shrinker. We also made the error
messages clearer. And of course, we understood
that people are already using ProGuard, and so
we decided that we wanted to be ProGuard compatible. And so we understand all
of ProGuard’s keep rules. So how does ProGuard work? [INAUDIBLE] So
before in Gradle, we would run ProGuard between your
Java compilation and the dex generation. And the reason for
that is that ProGuard is a Java-to-Java compilation. And so this added more
time to your build. In Android Studio 3.2,
you can enable our R8. It’s still experimental. You can enable it
using the setting. And what will happen is it
will replace ProGuard, desugar, D8 with one single
process, R8, that does all of those steps in one go. So we ran R8 on several
apps internally. This is the Nest app data. So by just swapping
ProGuard for R8, we’re able to save 100K
on the dex file size, and save 25% in
compilation time. Now, the Nest app
is highly optimized. It has very specific
ProGuard keep rules, and still we were able to
save quite a bit of space. We also run this on some of the
system apps that are shipping with the Android
OS, and on average were able to save 25%
of the dex file size by just swapping
ProGuard with R8. So of course, last
year we announced that Kotlin is now a
supported language on Android. And so we figured we need to do
something for Kotlin as well. Kotlin this is amazing language. It allows you to write
very succinct codes. But of course, if you
write a succinct code, and it’s so powerful, it needs
to generate a lot of bytecode. And so we looked
into places what we could strength that further
than the standard analysis. And so we found a couple
places where we could do things like class merging,
especially around lambdas, we do more nullness
analysis, et cetera. So let’s take a
look at that demo. So here I have a
Kotlin application. For those of you haven’t
programmed Kotlin, this is a data class, which
is a class that it generates a lot of code for you. So you have a couple fields
here that are really properties. So it generates
getters and setters, equals, get hash
code, et cetera. And then here, I have an
extension method that tells it that if I see a collection
that’s instantiated to the type car, at this extra
method, it allows me to search for make and model. And so in the class, I’m
using the sequence operators of Kotlin to do a couple
of filters and groupize. Now, normally you wouldn’t write
this many filters in a row. You would probably just
put all the Boolean logic in one filter. But they wanted to show you what
happens with multiple filters. So normally, Kotlin will
compile each of these lambdas into its own class. And so each lambda you
use adds a new class. And that’s not really what
you want in a dex file, because you always want to
keep the methods reference count low. So let’s take a look at what
happens when we do with R8. In this project, enabled R8
by setting this property. And let’s take a look what
happens in the output APK. So here, my class is dex again. The main activity– and I
added a call to that method, in the onCreate method. And so here in the
onCreate method, you see that there is this new
instance to this lambda group class. And the interesting thing is
that lambda group class is not defined in a package. It’s defined at the top level. So let’s take a look at that. Here in the dex file,
there is this [INAUDIBLE] lambda group class. And you see it implements the
Kotlin function one interface. And it has two instance fields. One, which is of type
objects, is named capture, and the other one is ID. And then the constructor takes
both the ID and the objects and sets the variables. And then here, in
the invoke method, we see that it has
this packet switched. And so what we’re doing is
we’re regenerating, basically, the bytecode equivalent
of a switch statement. We switch over the ID that was
passed into the constructor to figure out which
piece of code to call. And so if you
scroll through here, you’ll see that the call to get
here, the property read model, are all in this piece of code. And then you see here
that we have the switch. And so what is
going on is we find that we have lambdas that
are of the same signature, basically they implement
the same interface, and they have the same
capture variables. And so we can take all
the different lambdas and merge them into
one class, which allows you to have less metadata
and less method references. So let’s switch
back to the slides. Kotlin is something
we’re very excited about, and it’s becoming more
and more important. And so with R8, we’re going to
keep adding new optimizations for Kotlin. We’re doing the lambda
merging, no analysis, and we’re adding many more. And so hopefully, R8
will really help you get your Kotlin code even smaller. Lastly, we announced Android
app bundle yesterday. One of the things that comes
with Android app bundle is dynamic features. And so you cannot only
split your APK by resources, languages, et cetera. You can now also split your
features into multiple APKs. And that’s of course,
great, because not everybody uses every feature of your app. And now the downside– and we
heard this already with Instant Apps last year– instead, it makes
it harder to run ProGuard or R8 over your
app, because these tools, they are based on doing
whole program analysis. And now you don’t have a
single program anymore. And so what we came
up with is a way to take all your
different features, add them, and pass
them as one command line into ProGuard or R8
so that it is effectively a whole program. And then program R8 will spit
out a single jar or dex file. And then we can take that
information, the ProGuard mapping file and the
original feature jars, and with that information,
we have enough data to reconstitute the
different dex files. And so we have this
new dex splitter, which is based on the D8 code base. And it will spit out
the whole program again into different modules. And now you can apply
shrinking and optimizations to your features as well. So this is coming soon. It will be in Android Studio 3.2
by the time it reaches stable. So we looked at a whole bunch
of different compiler work that we’ve done
over the last year. Some of it is already stable. Incremental dexing, it was
introduced in Android Studio 3.0, desugar stand
alone was Studio 3.0. In Android Studio
3.2 we’re enabling D8 and desugaring as part of that. And then in Android
Studio 3.2, we’re introducing R8 as an
experimental feature. Please use it. We announced that DX will be
deprecated as soon as we find no more major issues in D8. So before even a year
or so, DX will be gone. So we’d like you to try it out. File bugs– so if you
go in Android Studio to the Help menu, there is
a Submit Feedback option, which will immediately dump
you into the Issue Tracker. And then you can
easily file a bug. Our team is very
responsive to these bugs. There is another
session tomorrow that’s called
Effective ProGuard Keep Rules for Smaller Applications. And it’s basically
a how-to on how to start using ProGuard or R8 by
one of the developers on the R8 team. Please fill out the survey on
the Google.com I/O schedule about this talk as well. Thanks, everybody! [MUSIC PLAYING]

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,


Leave a Reply

Your email address will not be published. Required fields are marked *