Working with SpringSource Application Platform's provisioning repository

One of the main advantages of the SpringSource Application Platform is its ability to provision dependencies on an as-needed basis. The benefits of this are two-fold: it ensures that the Platform's memory footprint is as small as possible and it allows applications to be deployed without encapsulating all of their dependencies in a monolithic deployment unit, e.g. a WAR file. To take advantage of these capabilities you will require an understanding of the Platform's provisioning repository and this blog intends to provide just that.
Where is the provisioning repository and how does it work?
By default the Platform's provisioning repository can be found in the repository directory at the root of the installation:

As you can see, there are three main directories: bundles, installed and libraries. installed is for the Platform's internal use so we'll focus on the bundles and libraries directories here. Each contains a number of subdirectories to separate the different types of dependencies:
- ext contains external dependencies that are provided with the Platform but are not part of the Platform itself.
- subsystems contains all of the subsystems that comprise the Platform.
- usr is initially empty and is intended to contain user-added dependencies, i.e. anything upon which your applications depend that is not already provided by the Platform.
The Platform searches the repository directory structure for both bundles and libraries during its initial startup. I'll talk about how this searching can be configured later on in this entry. As bundles and libraries are found within the repository, details of their symbolic names, exported packages etc. are added to an in-memory index of the repository. Upon completing the scan the in-memory indexes are cached to disk. Minimising the Platform's startup time was a priority for us during development. This caching allows the Platform to save some time during startup: it can skip the scan unless it detects that the contents of the repository have changed.
Runtime provisioning
In a plain OSGi environment a bundle's dependencies can only be satisfied by other bundles which have already been installed in the environment. For example, installing and starting a bundle that imports the org.apache.commons.dbcp package will fail if no bundle which exports that package has already been installed. This can be a real pain for users as they have to manually install all of a bundle's dependencies. Thankfully, the SpringSource Application Platform improves upon this significantly by dynamically installing dependencies on an as-needed basis.
When a deployed application is started by the Platform its bundles are installed into Equinox. The Platform then asks Equinox for a list of all of the bundles' unsatisfied dependencies and tries to satisfy them. Let's analyse this process by walking through a simple example scenario:
- A bundle which imports org.apache.commons.dbcp is installed by the Platform into Equinox.
- The Platform asks Equinox for the bundle's unsatisfied dependencies and is told that the import of org.apache.commons.dbcp cannot be satisfied.
- The Platform searches its index of the provisioning repository for a bundle which exports org.apache.commons.dbcp
- The Platform installs the bundle which exports org.apache.commons.dbcp into Equinox
- Having satisfied the original bundle's dependencies the Platform successfully starts the original bundle.
We hope that this will make our users' lives much easier: as long as your application's dependencies are available to the Platform you can simply deploy your application and go! There's no need to get your hands dirty with the time consuming process of manually installing all of your application's dependencies. One great thing about this process is that dependencies are only installed when they're needed. You can add as many bundles as you wish to the provisioning repository with almost no impact upon the Platform's memory footprint.
If the Platform's unable to satisfy a dependency from its repository a log message is generated that details the dependency that could not be satisfied. Armed with this information you can use the SpringSource Enterprise Bundle Repository to get hold of what you need. We'd like the repository to provide as complete a set of bundles as possible: if there's a dependency that you need and it's not available then please let us know.
Adding items to the provisioning repository
So you've downloaded a dependency or two and now you want to add them to the Platform. All you need to do is copy them to the appropriate directory in the repository. For example, a new bundle would typically be added to the repository/bundles/usr directory and a new library descriptor would typically be added to the repository/libraries/usr directory.
As described above, the Platform uses an in-memory index of its provisioning repository that's populated during startup. In addition to this, the Platform will also check that its view of the repository is up-to-date whenever an application is deployed. When you want to install an application with new dependencies simply copy the dependencies to the appropriate locations in the repository and then deploy your application. During its deployment processing, the Platform will notice that the repository has been updated and will refresh its view of it. This means that you don't need to spend time restarting the Platform every time you want to install an application with new dependencies.
Sharing the provisioning repository between installations
The locations where the Platform searches for items in the provisioning repository can be easily configured to suit your needs. For example, it's possible for multiple Platform instances to share some or all of a provisioning repository so you don't need to waste effort maintaining duplicate sets of bundles between Platform installations.
The locations which are scanned by the Platform when creating its provisioning repository can be configured in the config/platform.config file. The default configuration, which is used in the absence of any specific configuration, is:
"searchPaths": [
"repository/bundles/subsystems/{name}/{bundle}.jar",
"repository/bundles/ext/{bundle}",
"repository/bundles/usr/{bundle}",
"repository/libraries/ext/{library}",
"repository/libraries/usr/{library}"
]
}
Any relative paths are interpreted by the Platform as being relative to its installation root, with absolute paths being supported too. The entries in the search paths within curly braces are simply wildcards, e.g. the subsystems search path repository/bundles/subsystems/{name}/{bundle}.jar will find any file with a name ending in .jar in any of the subsystems directory's immediate sub-directories.
Hopefully you can see that it's a trivial change to share some or all of a provisioning repository between Platforms. For example, to make the Platform search a directory named shared-bundles in the root of the filesystem as well as in its own subsystems and ext directories, all you need to do is add the following snippet of JSON (JavaScript Object Notation) to the platform.config file:
"searchPaths": [
"repository/bundles/subsystems/{name}/{bundle}.jar",
"repository/bundles/ext/{bundle}",
"/shared-bundles/{bundle}",
"repository/libraries/ext/{library}",
"repository/libraries/usr/{library}"
]
}
By configuring two or more Platform installations with this configuration they can be made to share the bundles in /shared-bundles. This can easily be taken one step further by configuring all of the search paths to point to shared locations so you wouldn't need to manage a per-Platform provisioning repository at all.
What's next?
We're planning to make it simpler to run multiple instances of the Platform from the same set of binaries by providing some tooling or scripts. These will do most of the work for you, providing a sensible default configuration that you can then tweak to meet your specific needs.
We also intend to combine the power of the Platform's on-demand provisioning and the SpringSource Enterprise Bundle Repository by allowing the Platform to be configured to optionally search a remote repository when it's trying to satisfy a dependency. If a dependency is found in a remote repository the Platform will handle its download and installation automatically. Hopefully this'll make developers' lives a lot easier, especially in the initial stages of an application's development as new dependencies are being added on a regular basis.
We'd love to hear your suggestions as we work on the above-described enhancements and the Platform in general: please don't hesitate to comment on this blog entry, raise a JIRA, or post on our forums.
Sebastien Arbogast says:
Added on May 9th, 2008 at 4:49 pmHow about downloading and installing dependencies automatically during development. For example, if I use Maven 2 as a building tool, all my dependencies are automatically downloaded to my local repository. So Maven 2 local repository could be added to the default search path. WDYT?
Christian Dupuis says:
Added on May 9th, 2008 at 7:18 pmSebastien,
as part of the SpringSource AP Tools we will shortly release an addition to the current feature set that allows you to browse, search and download/install bundles and libraries from the repository. In combination with the classpath container that is created based on the MANIFEST.MF that should help streamlining the development process.
Christian
Sebastien Arbogast says:
Added on May 10th, 2008 at 6:55 amBut I feel like you're focusing all your tooling effort on Eclipse. As far as I'm concerned, I hate Eclipse and I've been an IntelliJ fan for a very long time. Now I understand that you cannot support all IDE's on the market but at least there should be significant effort to support IDE neutral tools. How about Maven support for example? Is Maven Bundle plugin enough? Is it possible to develop a whole S2AP-targeted application using only Maven and a text editor?
Ben Hale (blog author) says:
Added on May 12th, 2008 at 3:21 am@Sebastian, using your Maven2 cache as a repository search path is actually a great idea and has been discussed on the SpringSource Application Platform forums. The issue is that right now the platform can\'t support arbitrarily deep search paths like Maven uses for groupIds (interestingly since Ivy has flat orgs it will work). However, Andy tells me that he\'s got ANT-style paths on the docket for this week. When he gets those in, I\'m going to blog about exactly what you\'ll need to do to use your Maven cache as a repository with the next release.
There\'s also a JIRA issue open for it: http://issuetracker.springsource.com/browse/PLATFORM-35
Joseph Shore says:
Added on May 12th, 2008 at 6:09 amWhat is the availability of http://www.springsource.com/repository ?
e.g. 95% of the time, 99% of the time, 99.9% of the time.
Ben Hale (blog author) says:
Added on May 12th, 2008 at 7:04 am@Joseph, your question actually has two answers.
The first is in relation to the web-app that runs at http://www.springsource.com/repository. This application is simply a searching front end on the actual repositories themselves. It runs on the platform (and was actually the first production system on the platform) and on one of our many servers managed by Contegix. As such, there is no guarantee about it's availability but we expect it to be pretty good what with it running on our products.
The second answer is more interesting and more to what I think you're actually asking and that's about the actual repositories hosted at http://repository.springsource.com. These repositories are currently hosted on Amazon S3 which advertises an SLA of 99.9% uptime. Please note that SpringSource is not making any uptime guarantees but rather depending on our providers to give you an idea of what to expect. At this time we are solely dependent on S3, but have plans in the near future to have a hot backup based on another system with transparent fail-over.
I would point out that if you are really worried about access to these repositories (and I strongly encourage you to be if you're building business-critical software) you should maintain your own corporate repositories inside of your network. Both Ivy and Maven support this already and since the SpringSource Enterprise Bundle Repository are standard implementations of both it should work transparently.
David Savage says:
Added on May 13th, 2008 at 5:32 amWould it be possible to provide an example of how to configure either ivy or maven to provide a backup?
Having done a little experimentation it seems both ivy and maven cache bundles that you have downloaded in the past. However these are bundles you explicitly referenced. For enterprise deployments it seems more appropriate to cache all bundles locally and then update on demand based on internal rules.
Derek Baum says:
Added on May 13th, 2008 at 10:13 amHi,
I don't have permission to view the JIRA you reference http://issuetracker.springsource.com/browse/PLATFORM-35
Thanks,
Derek
[quote comment="106013"]@Sebastian, using your Maven2 cache as a repository search path is actually a great idea and has been discussed on the SpringSource Application Platform forums. The issue is that right now the platform can\'t support arbitrarily deep search paths like Maven uses for groupIds (interestingly since Ivy has flat orgs it will work). However, Andy tells me that he\'s got ANT-style paths on the docket for this week. When he gets those in, I\'m going to blog about exactly what you\'ll need to do to use your Maven cache as a repository with the next release.
There\'s also a JIRA issue open for it: http://issuetracker.springsource.com/browse/PLATFORM-35/quote
Andy Wilkinson says:
Added on May 13th, 2008 at 2:54 pmHi Derek,
You have to sign-up: http://issuetracker.springsource.com/secure/Signup!default.jspa
Regards,
Andy
David Savage says:
Added on May 14th, 2008 at 5:21 amHmmm I can't read that page either, I've signed up to Jira and logged in but I get the following message when I follow the link (On my preferences page it says I am a member of jira-users):
PERMISSION VIOLATION
ERROR
It seems that you have tried to perform an operation which you are not permitted to perform.
If you think this message is wrong, please consult your administrators about getting the necessary permissions.
David Savage says:
Added on May 14th, 2008 at 5:40 amOut of interest I started to dig around to figure out how to do a local mirror of content from the spring repository using Ivy.
However it seems the spring repository is not enabled to allow ivy to do wild card searches which I think most enterprises would need. The alternative is specifically listing every version which is used in the enterprise but this would get unmanageable very quickly.
What I did (possibly some daft mistake):
build.xml:
Relatively new to Ivy but docs suggest this should this work?
Adding some debug lead me to the fact that the spring repository does not allow directory browsing on folders which ivy needs to do wild card matching. Compare the output from the maven repository used by the ivy biblio resolver and spring:
http://repo1.maven.org/maven2/
http://repository.springsource.com/ivy/bundles/release/
Is this just a case of overzealous admins turning off directory browsing by default?
David Savage says:
Added on May 14th, 2008 at 5:44 amSorry previous post mangled, need to escape xml characters!!
Does this work >test<
David Savage says:
Added on May 14th, 2008 at 5:50 amGreat, ok escaped this time:
<ivysettings>
<settings defaultCache=".ivycache" defaultResolver="spring" checkUpToDate="false" />
<property name="repository.dir" value="/opt/ivy-repository"/>
<resolvers>
<chain name="spring">
<url name="com.springsource.repository.bundles.release">
<ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<url name="com.springsource.repository.bundles.external">
<ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
</chain>
<filesystem name="internal">
<ivy pattern="${repository.dir}/[module]/ivy-[revision].xml" />
<artifact pattern="${repository.dir}/[module]/[artifact]-[revision].[ext]" />
</filesystem>
</resolvers>
</ivysettings>
build.xml
<target name="install" description="–> install dependencies with ivy">
<ivy-install from="spring" to="internal" organisation="org.springframework" module="org.springframework.*" revision="*" matcher="glob" />
</target>
Ben Hale (blog author) says:
Added on May 14th, 2008 at 9:27 am[quote post="332"]However it seems the spring repository is not enabled to allow ivy to do wild card searches which I think most enterprises would need. The alternative is specifically listing every version which is used in the enterprise but this would get unmanageable very quickly.[/quote]
[quote post="332"]Is this just a case of overzealous admins turning off directory browsing by default?[/quote]
Actually no it's not a case of overzealous admins
In fact, it's a byproduct of the fact we back the repository with Amazon S3 (http://s3.amazonaws.com). Since S3 is a REST service with a flat namespace there's no actual directory structure to look for. It turns out that with the judicious use of slashes it can appear to be a directory structure when it's not actually.
In general, it's never recommended that you try and replicate large swaths of these repositories internally. I'll point at a Maven tool called maven-proxy (http://maven-proxy.codehaus.org/) for the moment but the same thing applies to Ivy. To quote:
[quote]It is quicker than mirroring the entire of ibiblio (not to consider the waste)[/quote]
What you'd really want to do is have a proxy so that when you requested a specific type it'd try and service that from a local cache. If it couldn't it would then try and download from the primary repository and cache locally for all future requests. A standard HTTP proxy can do this, but obviously a dedicated proxy like maven-proxy is ideal.
Andy Wilkinson says:
Added on May 15th, 2008 at 11:25 am[quote comment="106142"]Hmmm I can't read that page either, I've signed up to Jira and logged in but I get the following message when I follow the link (On my preferences page it says I am a member of jira-users)[/quote]
I've added you to the Platform beta users group, which should give you the access you need. Let me know if it doesn't do the trick.
Graham Jenson says:
Added on May 22nd, 2008 at 5:48 pmThe idea of a run-time downloading of bundles from a repository brings up complex issues. Although a lot of discussion has been on “Why run-time?� I think it is a good idea but have some reservations.
What is the starting state for the newly downloaded bundle? Is it run? This may be needed if it supplies services that are required, implementations from that package, but if not then it takes up resources for no reason. If it is installed but not started then possible services that are needed will not be in the registry, the bundle that required the package now has satisfied dependencies but can make no use of it.
Also there are issues of selection, if there are multiple different bundles supplying the same package, this could be possible because they may be sharing libraries packages, how will they be selected. I am aware of the Import-Library header, but there will be bundles that may be required that are not using this.
Finally if it is detected that in the current context running older versions of bundles, will they be automatically updated if they will “fit� and not cause other dependency issues? I.E. a new build of an apache bundle is released with the exact dependencies, will SpringSource update my older version from the repository.
Thank you
Graham
Andy Wilkinson says:
Added on May 30th, 2008 at 7:08 amWe haven't thought in detail about the semantics of remote provisioning at runtime as it's still just an idea at this stage. Our main goal in this area is to simplify developers' lives, with runtime remote provisioning being one possible solution.
As Christian mentioned above another solution will be available in a forthcoming update to the Platform Tools. This update will integrate with the Enterprise Bundle Repository so that developers can browse for and download new dependencies quickly and easily to a Platform installation.
Talking in more general terms, a bundle will only ever be installed into Equinox by the Platform's provisioning support when it's needed, i.e. it's required to satisfy another bundle's dependencies. As of beta5, the Platform will start any bundles that it installs during dependency satisfaction so that their services can be made available.