R8 踩坑记

前言

最近调研了下升级android gradle plugin的可行性，在调研过程中发现了R8的几个坑，记录下解决方法。

升级Android Gradle Plugin

首先需要升级android gradle plugin 到 3.4.1

buildscript {
    dependencies {
        classpath "com.android.tools.build:gradle:3.4.1"
    }
}

然后将项目中所有禁用选项移除，如d8，r8，如下

#android.enableR8=false
#android.enableD8=false
#android.enableIncrementalDesugaring=false

ClassNotFoundException

尝试升级后遇到的第一个问题就是R8开启后在multidex启用的情况下，且minSdkVerison<21时，在Dalvik上启动发生crash，出现ClassNotFoundException。

通过阅读android gradle plugin源码发现，构建R8Transform时，传递给R8Transform构造函数的multidex keep proguard文件未将构建时aapt计算产生的manifest multidex proguard file传递进去，导致很多应该出现在maindex中的类实际上跑到了second dex中去了，在Daivik上出现启动crash的问题。

具体关键源码如下：

@NonNull
private Optional<TaskProvider<TransformTask>> createR8Transform(
        @NonNull VariantScope variantScope,
        @Nullable FileCollection mappingFileCollection,
        @Nullable ProGuardTransformCallback callback) {
    final BaseVariantData testedVariantData = variantScope.getTestedVariantData();

    File multiDexKeepProguard =
            variantScope.getVariantConfiguration().getMultiDexKeepProguard();
    FileCollection userMainDexListProguardRules;
    if (multiDexKeepProguard != null) {
        userMainDexListProguardRules = project.files(multiDexKeepProguard);
    } else {
        userMainDexListProguardRules = project.files();
    }

    File multiDexKeepFile = variantScope.getVariantConfiguration().getMultiDexKeepFile();
    FileCollection userMainDexListFiles;
    if (multiDexKeepFile != null) {
        userMainDexListFiles = project.files(multiDexKeepFile);
    } else {
        userMainDexListFiles = project.files();
    }

    FileCollection inputProguardMapping;
    if (testedVariantData != null
            && testedVariantData.getScope().getArtifacts().hasArtifact(APK_MAPPING)) {
        inputProguardMapping =
                testedVariantData
                        .getScope()
                        .getArtifacts()
                        .getFinalArtifactFiles(APK_MAPPING)
                        .get();
    } else {
        inputProguardMapping = MoreObjects.firstNonNull(mappingFileCollection, project.files());
    }

    R8Transform transform =
            new R8Transform(
                    variantScope,
                    userMainDexListFiles,
                    userMainDexListProguardRules,
                    inputProguardMapping,
                    variantScope.getOutputProguardMappingFile());

    return applyProguardRules(
            variantScope,
            inputProguardMapping,
            variantScope.getOutputProguardMappingFile(),
            testedVariantData,
            transform,
            callback);
}

对于该问题，我们可以简单的hook下R8Transform中的mainDexRulesFiles字段进行修复，判断mainDexRulesFiles中是否已经有manifest multidex proguard file, 如果没有，则将该文件添加进去并替换mainDexRulesFiles字段。

android.applicationVariants.all {variant ->
    def r8Task = getR8Task(project, variant)
    if (r8Task != null) {
        def r8Transform = r8Task.getTransform()
        //R8 maybe forget to add multidex keep proguard file in agp 3.4.0, it's an agp's bug!
        //If we don't do it, some classes will not keep in maindex so crash will happened in Dalvik.
        if (r8Transform.metaClass.hasProperty(r8Transform, "mainDexRulesFiles")) {
            File manifestMultiDexKeepProguard = getManifestMultiDexKeepProguard(variant)
            if (manifestMultiDexKeepProguard != null) {
                //see difference between mainDexRulesFiles and mainDexListFiles in https://developer.android.com/studio/build/multidex?hl=zh-cn
                FileCollection originalFiles = r8Transform.metaClass.getProperty(r8Transform, 'mainDexRulesFiles')
                if (!originalFiles.contains(manifestMultiDexKeepProguard)) {
                    FileCollection replacedFiles = project.files(originalFiles, manifestMultiDexKeepProguard)
                    project.logger.error("R8Transform original mainDexRulesFiles: ${originalFiles.files}")
                    project.logger.error("R8Transform replaced mainDexRulesFiles: ${replacedFiles.files}")
                    //it's final, use reflect to replace it.
                    replaceFinalField(r8Transform.getClass(), "mainDexRulesFiles", r8Transform, replacedFiles)
                }
            }
        }
    }
}

void replaceFinalField(Class clazz, String filedName, Object instance, Object fieldValue) {
    def field = clazz.getDeclaredField(filedName)
    //noinspection UnnecessaryQualifiedReference
    def modifiersField = java.lang.reflect.Field.class.getDeclaredField("modifiers")
    modifiersField.setAccessible(true)
    //noinspection UnnecessaryQualifiedReference
    modifiersField.setInt(field, field.getModifiers() & ~java.lang.reflect.Modifier.FINAL)
    field.setAccessible(true)
    field.set(instance, fieldValue)
}

Task getR8Task(Project project, def variant) {
    String r8TaskName = "transformClassesAndResourcesWithR8For${variant.name.capitalize()}"
    return project.tasks.findByName(r8TaskName)
}

File getManifestMultiDexKeepProguard(def applicationVariant) {
    File multiDexKeepProguard = null
    try {
        def buildableArtifact = applicationVariant.getVariantData().getScope().getArtifacts().getFinalArtifactFiles(
                Class.forName("com.android.build.gradle.internal.scope.InternalArtifactType")
                        .getDeclaredField("LEGACY_MULTIDEX_AAPT_DERIVED_PROGUARD_RULES")
                        .get(null)
        )

        //noinspection GroovyUncheckedAssignmentOfMemberOfRawType,UnnecessaryQualifiedReference
        multiDexKeepProguard = com.google.common.collect.Iterators.getOnlyElement(buildableArtifact.iterator())
    } catch (Throwable e) {
        e.printStackTrace()
    }
    return multiDexKeepProguard
}

关于mainDexRulesFiles和mainDexListFiles的区别，可以查看文档：https://developer.android.com/studio/build/multidex?hl=zh-cn

mainDexRulesFiles对应于multiDexKeepProguard属性，该文件使用与proguard相同的格式，并且支持整个proguard语法，我们可以在gradle中手动配置该文件

android {
    buildTypes {
        release {
            multiDexKeepProguard('multidex-config.pro')
        }
    }
}

mainDexListFiles对应于multiDexKeepFile属性，该文件应该每一行包含一个类，并且采用/来分割包路径，如

com/example/MyClass.class
com/example/MyOtherClass.class

我们可以在gradle中手动配置该文件

android {
    buildTypes {
        release {
            multiDexKeepFile file('multidex-config.txt')
        }
    }
}

至此maindex的问题已经解决。

关于tinker R8的支持，可以见 PR https://github.com/Tencent/tinker/pull/1112

Illegal invoke-super to 某某某方法返回值 某某某方法名() from class 某某某类

解决完maindex的问题后发现又遇到另一个问题，当构建patch时，会apply基线的mappping文件，此时，如果R8开启，就会报如下错误

Illegal invoke-super to 某某某方法返回值 某某某方法名() from class 某某某类

经过测试发现，Android Gradle Plugin 3.5.0-beta02已经修复了该问题，但是对于Android Gradle Plugin 3.4.0-3.4.1该问题依旧存在，那么是否就无解呢？其实不是的，Google此处留了一个小后门，可以独立更新R8。

首先有个前提条件，就是R8是被打包在android gradle plugin插件中的，并非使用的远程依赖。R8的所有相关类被打入了如下依赖中：

com.android.tools.build:builder:3.4.1

而该依赖是被如下依赖传递依赖的：

com.android.tools.build:gradle:3.4.1

这个问题的解决得从gradle的classloader机制说起，我们只需要明确如下概念即可：

• buildscript中classpath声明的一级依赖，声明在前的，classloader会优先查找
• buildscript中classpath声明的一级依赖，其优先级高于任何传递依赖

我们可以用如下代码打印一下加载R8相关类的classloader中相关文件路径

project.afterEvaluate {
    Class.forName("com.android.tools.r8.R8")?.getClassLoader()?.getURLs()?.each {
        project.logger.error("url: ${it}")
    }
}

假设我们classpath依赖有如下依赖

buildscript {
    dependencies {
        classpath "com.android.tools.build:gradle:3.4.1"
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.31"
    }
}

输出如下

url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/gradle/3.1.4/8f9a726f69c0c8fa3f447566717a21e6b394ed9/gradle-3.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-gradle-plugin/1.3.31/4fcdcfc17948d6ccc730551d313852de8d008eab/kotlin-gradle-plugin-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/gradle-core/3.1.4/4c846d065331c2a11ed605619613833a842a7b8d/gradle-core-3.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/bundletool/0.1.0-alpha01/f7c303e37818223bd98566fcbea29aa0964c4d06/bundletool-0.1.0-alpha01.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-gradle-plugin-api/1.3.31/a120b7f0c7b5981aab41c51aa9d6229484835681/kotlin-gradle-plugin-api-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-gradle-plugin-model/1.3.31/29f635dd4f9669e237f9d9d478605e7c2049129e/kotlin-gradle-plugin-model-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-native-utils/1.3.31/23f0ac1ae03ef42ca952ca45de1d99213ce8f5bd/kotlin-native-utils-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-annotation-processing-gradle/1.3.31/e6e76345392e660078aaf08a5f2886be4d1f6a2/kotlin-annotation-processing-gradle-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-android-extensions/1.3.31/57bdeb88c9265052448a7a8422ed64ad92e9b262/kotlin-android-extensions-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-compiler-embeddable/1.3.31/965ef6d9abe1555cc5d0a34dd24ce93b49123e7c/kotlin-compiler-embeddable-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/builder/3.1.4/afbcd4b7002c61fe898b1b4c50ed9e62386125d8/builder-3.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.lint/lint-gradle-api/26.1.4/cbc00782604b7d0ad50e9c50b84b074af79394f0/lint-gradle-api-26.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/gradle-api/3.1.4/eb41dfb5596afd8933c804595ca8596952fad450/gradle-api-3.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.databinding/compilerCommon/3.1.4/11005423fee93309c0cd512a8783647702c20c27/compilerCommon-3.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/manifest-merger/26.1.4/ddb4dcb3bb44c7053fa583967dfa9030f43f1c01/manifest-merger-26.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/sdk-common/26.1.4/5dbdefb2dc6cb5ba1b4b059bf11c964a830c5755/sdk-common-26.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/sdklib/26.1.4/7424640f2bd3ca3faccb6f656e29547430cd464a/sdklib-26.1.4.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/repository/26.1.4/349dfc72ae53dedc32a17f5d47440838fe2527f1/repository-26.1.4.jar

从日志可以看到，一级依赖，先声明的会优先进行查找，任何传递依赖，都位于一级依赖之后。

有了以上理论知识，我们就可以修复该问题了。

google为我们提供了R8的远程依赖，将 http://storage.googleapis.com/r8-releases/raw 添加到maven中并声明r8远程依赖（com.android.tools:r8:1.5.37） 为一级依赖

buildscript {
    repositories {
        maven {
            url "https://maven.google.com"
        }
        maven {
            url 'http://storage.googleapis.com/r8-releases/raw'
        }
    }
    dependencies {
        classpath "com.android.tools.build:gradle:3.4.1"
        classpath 'com.android.tools:r8:1.5.34'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.31"
    }
}

再输出classloader中的url

url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/gradle/3.4.1/195bd39d36b255d333d6493dcac0d542258d2a3d/gradle-3.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/r8/1.5.34/bbbc00e9fe01795284cee43b7696dcd1156caead/r8-1.5.34.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-gradle-plugin/1.3.31/4fcdcfc17948d6ccc730551d313852de8d008eab/kotlin-gradle-plugin-1.3.31.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/builder/3.4.1/5fd3cae600dcbda8adaf1791c0e844e585ac0d3d/builder-3.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.analytics-library/tracker/26.4.1/2574526ca59f2ddf4bd0ed6c3f77ecd89b51512d/tracker-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.analytics-library/shared/26.4.1/8b6f40928c3aecc2e08125a2e3baf3c2228244a2/shared-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.analytics-library/crash/26.4.1/4019df9bc90acb3abc94fcf0c790bf1df799cb77/crash-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.lint/lint-gradle-api/26.4.1/f9dc1f1fba2b9b51f04e42497af0932119ea8c9b/lint-gradle-api-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/gradle-api/3.4.1/26d1d2ccf5b8580669d52cb21a9b7c846b78180b/gradle-api-3.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/androidx.databinding/databinding-compiler-common/3.4.1/f18b37dd0306f680e4ebec6f508beb66bc10f5f8/databinding-compiler-common-3.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/manifest-merger/26.4.1/653712f9996c0cb1f6b721a7712c2084b9dc28eb/manifest-merger-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/sdk-common/26.4.1/c1bff8a9ff4684cce461d28fa0ff6380bdad8aa6/sdk-common-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.build/builder-test-api/3.4.1/54d24a8d74dfdc087c40854fef6adcdad059285e/builder-test-api-3.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.ddms/ddmlib/26.4.1/3da8e5681c69f932e598a9fed1ca8508e069481a/ddmlib-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/sdklib/26.4.1/5df176d7cd2064c23006f63dce8cee12abff0f7c/sdklib-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools.layoutlib/layoutlib-api/26.4.1/d8c42109324425b0997efa35ed842f84a435908/layoutlib-api-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/dvlib/26.4.1/389d5879b2602e3de5b91ad38588be5dbd57276c/dvlib-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/repository/26.4.1/614275977f26ff273fcab4b2e67f872868a9af95/repository-26.4.1.jar
url: file:/Users/lizhangqu/.gradle/caches/modules-2/files-2.1/com.android.tools/common/26.4.1/d14471b3a2c7c21a705c67980fae4d3d3a8ca4eb/common-26.4.1.jar

从日志输出再次验证了我们的结论

• buildscript中classpath声明的一级依赖，声明在前的，classloader会优先查找
• buildscript中classpath声明的一级依赖，其优先级高于任何传递依赖

如果对R8源码感兴趣，可以见代码库：https://r8.googlesource.com/r8/

这里注意一个问题，目前可用的r8远程依赖版本有很多，android gradle plugin 3.4.1打包的版本为1.4.94，我们使用的远程依赖必须高于这个版本，测试下来，1.5.25、1.5.26、1.5.27、1.5.33、1.5.34这几个版本是正常的，1.5.36、1.5.36、1.5.37及以上版本会卡死在R8Tranform任务上，使用的时候需要留意。可以使用如下类似代码进行约束

buildscript {
    dependencies {
        def androidGradlePluginVersion = "3.4.0"
        classpath "com.android.tools.build:gradle:${androidGradlePluginVersion}"
        if (androidGradlePluginVersion >= "3.4.0" && androidGradlePluginVersion < "3.5.0") {
            //agp 3.4.0-3.4.1的r8有bug，apply mapping的时候会出现编译错误，因此使用独立依赖
            classpath 'com.android.tools:r8:1.5.34'
            
        }
    }
}

那为什么android gradle plugin 3.5.0-beta02没有这个问题呢？因为3.5.0用的R8版本就是1.5.34，该版本在android gradle plugi 3.4.1上经过验证确实没问题。

总结

R8的坑还很多，目前不建议启用，但可以踩踩坑先，看看有什么未知问题，提前进行解决。目前建议在gradle.properties中加入如下配置进行禁用

android.enableR8=false