Resolving Dependencies

OSGi’s Best Kept Secret

This Application Note is about resolving in OSGi. The OSGi Framework has always used a resolver to wire a given set of bundles together, ensuring that only valid wires are made. However, the same OSGi resolver can also be used to select a set of bundles from a much larger set. This application note discusses this secondary usage.

The resolver model is based on technology developed in OSGi since 2006 with RFC-0112 Bundle Repository. This RFC laid out a model that was gradually implemented in the OSGi specifications and for which many tools were developed. Resolving automates a task that is mostly done manually today.

What Problem is Being (Re)Solved?

The pain that the resolver solves is well known to anybody that builds application from modules, the so called assembly process. All developers know the dreadful feeling of having to drag in more and more dependencies to get rid of Class Not Found errors when the application starts (or after running for an hour). For many, Maven was a significant improvement because it made this process so much easier because dependencies in Maven are transitive. If you depend on some artefact, then this artefact drags in its own dependencies to the runtime.

Unfortunately, the Maven dependency model has limitations for a number of reasons. Though it undeniably has become easier to get a result, the result is often littered with unnecessary or wrong artefacts. Often application developers have no real understanding what is in their runtime.

These are the causes:

• Our small minds – The human mind and human motivation provide little support for properly maintaining any amount of meta-data. Many open source projects show unnecessary dependencies that were once needed but no longer necessary. Fortunately, Maven Central/Sonatype has become much more restrictive. However, in the past, a lot of Maven dependencies were quite bad.
• Aggregate problem – The aggregate problem stems from a dependency that is used for one small part but where a co-packaged part drags in other (and therefore unnecessary) dependencies. For example, many JARs provide support for Ant, a former build tool. This was useful if you happened to use Ant but it caused the addition of the Ant runtime in systems that never used Ant. The aggregate problem causes a serious increase of dependency fan-out in many projects.
• Version Conflicts – When you drag in a lot of dependencies you invariably run into a situation where two of your dependencies require the same third dependency but in different versions. (This is called the diamond problem for obvious reasons if you draw the graph.) Sometimes these versions are really incompatible, sometimes the higher would suffice. However, in Maven, the first artefact encountered in a breadth first search defines the version.
• Variations – In many situations it is necessary to create different variations of an application. For example, an application could be delivered for different platforms or in a demo variation and a full monty variation. Since the Maven transitive dependency model creates a fixed graph it is not suitable to assemble these variations easily. That is, there often is not a single runtime.

The result is that all too often the class path of the final application is littered with never used byte codes and/or overlapping packages that came from artefacts with different versions. If the customer could see the mess the average class path is they would run away screaming.

Establishing the perfect class path for an application is a process for which the human mind is extremely badly suited and the transitive model of Maven has too many sharp edges. The resolver provides an alternative model that focuses on looking at the whole solution space instead of having to live with the (sometimes arbitrary) decisions of developers or of deeply nested transitive dependencies.

Principles

The design of the resolving model is quite simple. It consists of the following entities:

• Resource – A resource is a description of an artefact that can be installed in a system. When it is installed, it adds functions to that system. However, before it can be installed it requires certain functions to already be present. A resource can describe a bundle but it could also describe a piece of hardware or a certificate. It is important to realise that a resource is not the artefact, it is a description of the relation between the artefact and the target system. For example, in OSGi the bundle is a JAR file that can be installed in a framework, this is the artefact. A resource describes formally what that bundle can contribute to the system and what it needs from the system.
• Capability – A capability is a description of a resource’s contribution to the system when its artefact is installed. A capability has a type and a set of properties. That is, it is a bit like a DTO (or struct). The type is defined by name and is therefore called the namespace. The properties are key value pairs, where the keys are strings and values can be scalars or collections of String, Integer, Double, and Long.
• Requirement – A requirement represents the needs of an artefact when it is installed in the system. Since we describe the system with capabilities, we need a way to assert the properties of a capability in a given namespace. A requirement therefore consists of a type and an OSGi filter. The type is the same as for the capabilities, it is the namespace. The filter is constructed from a powerful filter language based on LDAP. A filter can assert expressions of arbitrary complexity. A requirement can be mandatory or optional.
• Namespace – Since a requirement can match any set of properties it is necessary to scope the capabilities it can be asserted against. This is achieved by giving a requirement and a capability a namespace. A requirement can only match a capability when the capability has the same namespace.

Resolving in this App note is therefore the process of constructing an application out of resources. Resolving takes a list of initial requirements, a description of the system capabilities, and one or more repositories. It will use the list of initial requirements to find resources in the repository that provide the required capabilities. Clearly, these resources have their own requirements, retrieving applicable resources is therefore a recursive process. A resolver will find either a solution consisting of a set of resources where all requirements are satisfied or that there is no solution.

It is important to realise that a resource and its capabilities and requirements are descriptions. They provide a formal representation of an external artefact. Since these formal representations can be read by a computer, we can calculate a closure of resources that, when installed together, have only resources where all their mandatory requirements are satisfied by the resources in the closure.

Core Namespaces

Although the OSGi specifications started out with a set of headers that each had their own semantics, over time the specification migrated fully to the more simple and formal model of Resources, Capabilities, and Requirements. Since the function of the legacy headers were still needed, it was necessary to map these legacy headers to the formal model. This resulted in a number of OSGi core namespaces.

• osgi.wiring.identity – Bundle-SymbolicName header.
• osgi.wiring.bundle – Bundle-SymbolicName and Require-Bundle header.
• osgi.wiring.package – Import-Package and Export-Package headers.
• osgi.wiring.host – Bundle-SymbolicName and Fragment-Host header.
• osgi.ee – Bundle-RequiredExecutionEnvironment header.
• osgi.native – Bundle-NativeCode header.
• osgi.content – Provides the URL and checksum to download the corresponding artefact. (This namespace is defined in the compendium Repository specification.)

Each namespace defines the names of the properties and their semantics. For the OSGi namespaces, there are classes like org.osgi.framework.namespace.IdentityNamespace that contain the details of a namespace.

Example Resource Model

To make the model more clear let’s take a closer look to a simple bundle com.example.bundle that exports package com.example.pe and imports a package com.example.pi. When the bundle is installed it will require that some other bundle, or the framework, provides package com.example.pi. We can describe this bundle then as follows:

 Resource for bundle com.example.bundle
   capabilities:
     osgi.wiring.identity;  osgi.wiring.identity=com.example.bundle
     osgi.wiring.bundle;    osgi.wiring.bundle=com.example.bundle
     osgi.wiring.host;      osgi.wiring.host=com.example.bundle
     osgi.wiring.package;   osgi.wiring.package=com.example.pe
   requirements:
     osgi.ee;               filter:="(&(osgi.ee=JavaSE)(version=1.8))"
     osgi.wiring.package;   filter:='(osgi.wiring.package=com.example.pi)'

Who Wants to Provide That Gibberish???

Clearly, an Export-Package: com.example.pe is a bit easier to read than the corresponding capability, let alone the filter in the requirement. This is especially true when the versions are taken into account, with the assertion of the version range the filters become quite unreadable. Clearly, if this had to be maintained by human developers then a model like Maven would be far superior. Fortunately, almost all of the metadata that is needed to make this work does not require much extra work from the developer. For example, the bnd tool that is available in Maven, Gradle, Eclipse, Intellij and others can easily analyse a JAR file and automatically generate the requirements and capabilities based on the source code.

In certain cases it is necessary to provide requirements and capabilities that bnd cannot infer. However, Manifest annotations support in bnd can be used to define an annotation that will add parameterised requirements to the resource.

Really, when you use an adequately appointed toolchain (like Bndtools) none of this gibberish is visible to normal developers unless they have a special case. The gibberish is left to the tools that prefer gibberish over natural language.

How to Write Resolvable Bundles

For a bundle to be a good citizen in a resolution it suffices to follow the standard rules of good software engineering. However, since there is so much software out there that does not follow these rules, a short summary.

• Minimise Dependencies – Every dependency has a cost. Always consider if all dependencies are really necessary. Although a dependency can provide a short term gain for the coder, it will have consequences for the later stages in the development process. Perfect bundles are bundles that have no dependencies. Unfortunately they are also often useless (although you can still build a lot from Java SE java.* which is an implicit import). Therefore, a developer must always be aware of this trade off.
• API Driven – Always, always, always separate API and implementation. A lot of developers avoid the overhead of developing an API when they consider the (initial) problem too simple to do the effort of a separate API. The author of this App note is one of them but he never, ever did not regret this. Usually he then spent a lot of extra effort to add an API afterwards.
• Service Oriented – The best dependencies are service oriented dependencies. Services make the coupling between modules explicit and allow different implementations to provide the same service. For example, use a simple DTOs service that handles type conversions and JSON parsing/generating. In projects where I see the explicit use of Jackson it always seems to cause a mess of required JARs visible in each bundle project.
• Package Imports – A bundle can require other bundles (the Maven model) or it can import packages from other bundles. The best imported packages are the ones used for services since they are specification only. Second best are libraries. Library packages have no internal state nor use statics. Importing implementation packages for convenience is usually deemed bad practice because it often indicates that the decomposition in bundles is not optimal.
• Import the Exports – If a package is exported and used inside the same bundle then the exported package should also be imported. This allows the resolver more leeway.

To see the requirements and capabilities of a bundle Bndtools has a special Resolution view. This shows the requirements and capabilities of a selected JAR file or a bnd.bnd file.

This view uses a number of icons to represent the requirements/capabilities. You can hover over them to see further details.

• – osgi.service.
• – osgi.identity
• – osgi.wiring.package