May 13, 2013

jclouds at Gluecon 2013

Everett Toews is hitting the road, getting some hands on keyboards, and really teaching developers how to use jclouds. He’ll be giving a workshop titled “Solve the Cross-Cloud Conundrum with jclouds” at Gluecon this year. The session is on May 22 from 10:30 to 11:30 am in Breakout 2.

Gluecon

What is Gluecon? In their own words, “Cloud, Mobile, APIs, Big Data — all of the converging, important trends in technology today share one thing in common: developers. Developers are the vanguard. Developers are building in the cloud, building mobile applications, utilizing and building APIs, and working with big data. At the end of the day, developers are the core.” This session will get developers doing what they do best. The workshop includes an overview of jclouds, its terminology, and its community then it quickly gets down to development. First, we’ll install jclouds for use on the command line during the workshop. Then we’ll work through an example of using jclouds to start virtual machines in multiple clouds. Rackspace will be offering free cloud trial accounts to the attendees of this session. It’s their way of helping you get started in the cloud and say thanks for coming out.

Conclusion

The reason to use jclouds is to control your cloud infrastructure with Java. In this workshop Everett be showing you exactly how to do that.

April 28, 2013

jclouds 1.6.0 released!

jclouds 1.6 represents 7 months of refactoring to essentially “reset” the project 4 years after its founding.  The goal was to do less magic and be easier to code and troubleshoot.

No doubt, we’ve added new apis, including ones from OpenStack, Rackspace, AWS, DynECT, and UltraDNS, so if any of that interests you, move on to details in the release notes.

Most exciting is our move to the Apache Incubator.  Watch out as details will unfold this week as we progress to our next level of open source!

In the mean time, update your jclouds to version 1.6.0 and let us know how it works for you.

December 12, 2012

jclouds 1.5.4 “Mistletoe” released

jclouds community is an international group with over 100 contributors since we started in early 2009.  The idea of holidays vary, but we decided “Mistletoe” as an appropriate codename for jclouds 1.5.4.  Here’s why.

Chef and Karaf

We’ve noticed an attraction between systems that use chef and those that use karaf.  For example, it isn’t uncommon to have a deployment pipeline that incorporates the two.  Since jclouds has subproject for both chef and karaf, it seemed natural to cozy them together.  Thanks to Ioannis and ignasi, karaf and chef are now a couple!

For example, the following syntax can be used in karaf directly, or via the jclouds cli to start a node in any supported cloud and bootstrap chef recipes.

karaf@root>node-create --adminAccess --recipe chef/java::openjdk karaf
 

What else is in 1.5.4?

We’ve a number of other highlights in jclouds 1.5.4.  Many thanks to the code contributors jclouds, jclouds-chef, jclouds-karaf, and jclouds-cli for putting them together. Also to our review team for helping ensure quality: Andrew G and Matt for code, and Becca on docs.

Rackspace Cloud Load Balancers

Everett modernized support for “rackspace-cloudloadbalancers-us” and uk is now available.  This includes syntactic sugar such as auto-pagination and easy transforms.

FluentIterable<HostAndPort> sockets = clb.getLoadBalancerApiForZone(zone)
.list().concat().transform(converter);

Nova Server Diagnostics

Some implementations of Nova have diagnostic information available.  Thanks to Leander, you can now discover this capability at runtime.

Optional<Map> diagnosticInfo = novaApi.getServerApiForZone("az-1.region-a.geo-1")
    .getDiagnostics(serverId);

S3 Multi-Object delete

Maginatics have BlobStore containers with over a billion objects in them.  S3 containers (buckets) can now be cleared with 1/1000 the requests, thanks to Andrei's additional support of S3 multi-delete.

DeleteResult result = s3Api.deleteObjects(container, keys);
assertEquals(result.size(), keys.size());

What’s next

We are working on 1.6, an effort more efficient and resilient.  One tool we are evaluating for this is Netflix Hystrix.  We are also working on 1.5.x releases, which are likely to include google compute engine and virtualization support.  If interested in helping us out, hop onto IRC freenode #jclouds or join our dev group.

November 17, 2012

jclouds 1.5.3 out the door

Released on 2012-11-14, jclouds 1.5.3 includes minor fixes, and a few important updates:

  • new openstack-cinder and rackspace-cloudblockstorage-us/uk providers
  • add new Asia Pacific (Sydney) Region [ap-southeast-2]
  • new TagApi for ec2, in a revised syntax similar to openstack-nova
  • handle network failures in large container (1M+) deletes in blobstore
  • jclouds-cli and jclouds-karaf now provide a more intuitive file-based interface for reading and writing blobs.

Many thanks to the contributors in this release of jclouds, jclouds-chef, jclouds-karaf, and jclouds-cli. Also many thanks for the diligence of our review team: Andrew G and Matt for code, and Becca on docs.

Moreover, we’re indebted to BuildHive for checking over 500 pull requests to date! On a build side, we currently validate both JDK6 and JDK7 on each pull, thanks to Andrew P.

Please update to jcloud 1.5.3 and check the new features. Particularly, try things out using JDK7. If you have questions, ping our user group or tag your question with “jclouds” on stackoverflow.

We are working on 1.6 now, which includes a lot of cleanup and mechanisms to make less api calls when starting groups of machines. If interested in helping us out, hop onto IRC freenode #jclouds or join our dev group.

November 9, 2012

500 jclouds builds on BuildHive and counting…

At jclouds, we’ve been running our Jenkins continuous integration jobs in CloudBees’ DEV@cloud for a while now (CloudBees has a FOSS programme). We also have an active and ever-increasing contributor community, which amongst others means…lots of pull requests.

So we were very interested to hear about BuildHive, another CloudBees initiative that integrates with your GitHub account to automatically trigger build jobs for pull requests, and quickly signed up.

We did not regret it. BuildHive has served as a crucial initial filter to allow us to concentrate precious reviewing time on working changes and to remain confident, as pull requests are updated, that we are not going to break the build.

Of the last 100 builds, we’ve had 7 with errors, 2 with warnings, and 1 incomplete, so a roughly 10% efficiency gain in terms of focussing pull request review time. In future, we may also use BuildHive to provide “style feedback” and similar metrics for pull requests - most likely some feature suggestions for CloudBees to come here!

Thanks, BuildHive!

September 28, 2012

jclouds 1.5 is out!

Over 6 months in the making, and just in time for JavaOne, jclouds 1.5 is formally released. 

Many of you have been awaiting (or helping build) massive new support for OpenStack based clouds. There’s even more, such as our new Jenkins Plugin and command-line utility.

I encourage you to check out our release notes and give the newly minted version 1.5.1 a whirl.  If you haven’t seen our new website, check it out and let Becca know how you like it!

For those of you looking to meetup, you can find us at the Jenkins User ConferenceJavaOne, meetup at Abiquo during VMworld EU, the OpenStack Summit, Java/Cloud2Days, and RWX.  You can also meet us on irc freenode #jclouds and our google group.

Thanks to the many who made this happen, especially our community!  Here’s to 1.6!

July 16, 2012

fluency for paginated api lists

The problem of large result sets is something cloud providers want you to have. For example, what better sign of cloud life is there than Amazon S3 having a trillion objects stored so far. The way most cloud apis deal with listing your million (or even thousand) things is to paginate the response.

For example, you get a first page with maybe 500 records in it, and a “marker” you can use to get the next.

Sounds pretty straightforward, and most jclouds abstractions do this dance for you.

For example, our BlobStore api has an interface PageSet which holds a bunch of results and the underlying marker.  This also deals with the fact that sometimes “marker" is called "nextToken" or other unnecessarily different names :)

3 years since the birth of PageSet, you can imagine folks could get a better idea of what they *really* want.

Here’s a few comments:

  • Marker isn’t necessarily a String! - Andrew Gaul notes that especially in native implementations, it isn’t the case that Marker will always be a String. Imagine you are making an in-memory store. It is very likely the marker will be an Object in this case.
  • Set is the wrong type for results! - Tim Peierls notes that uniqueness isn’t a fundamental concern of API results.  In fact, it can get in the way of streaming.
  • Iterating through a bunch of pages is monkey-work! - Jeremy Whitlock found dancing through each page to get a complete view of metrics was tiring, and made a CloudWatch function for that.
  • Ensure it is possible to opt-out! - Toni Batchelli notes that iterating across network calls can lead to inconsistent state.  Particularly clojure users will want the option to manually control pagination.

This feedback underscored jclouds general concern to make things easier, yet still allow control. The status was tracked in issue 1011 and our jclouds-dev google group.

Through several iterations  and many thanks to Tim P for the design, we have a new type: PagedIterable, which extends Guava’s fantastic FluentIterable.

Here are a few examples of how it can be used:

lazy advance through all your metrics:

FluentIterable<Metric> allMetrics = cloudwatch.getMetricApi().list().concat();

advance only until we find the load balancer we want:

Optional<LoadBalancer> firstInterestingLoadBalancer = elb
   .getLoadBalancerApi().list()
   .concat()
   .firstMatch(isInterestingLB());

get only the first page of database instances

IterableWithMarker<Instance> firstPage = rds.getInstanceApi().list().get(0);

The above syntax is being worked through relevant apis.  In order to try it out, grab jclouds 1.5.0-beta.7 (releasing today), and use any of the following methods:

Other apis and abstractions will be caught up while we finish the 1.5 release.

Many thanks to guava for the base class, jclouds folks who participated in the design, as well the airlift guys who’s feedback helped solidify the idea.  If you are interested in participating, please reach out on irc freenode #jclouds or @jclouds on twitter!

May 2, 2012

New toys in 1.5.0-alpha.6

We are very near beta, I promise!  In the mean time, we just cut a new codebase with a bunch of cool new toys.  Here’s a few.

Thanks to Adam Lowe, we are getting even deeper into OpenStack with more Keystone support than ever. Setup your code to pull org.jclouds.labs/openstack-keystone, and you can do stuff like this:

      ContextBuilder contextBuilder = ContextBuilder.newBuilder("openstack-keystone");
      RestContext keystone = contextBuilder
         .credentials("tenantId:user", "password")
         .endpoint("https://keystone:35357")
         .build();

      for (String regionId : keystone.getApi().getConfiguredRegions()) {
         AdminClient adminClient = keystone.getApi().getAdminClientForRegion(regionId);
         for (Tenant tenant : adminClient.listTenants()) {
            // ...
         }
      }

Also pro, is our new Amazon CloudWatch support from Jeremy Whitlock. This is our first complete renovation of an AWS api to have the same look/feel as our new OpenStack stuff.  Just add a dependency on org.jclouds.providers/aws-cloudwatch and you can do this!

      ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-cloudwatch");
      RestContext cloudwatch = contextBuilder
         .credentials("accessKey", "secretKey")
         .build();

      for (String regionId : cloudwatch.getApi().getConfiguredRegions()) {
         MetricClient metricClient = cloudwatch.getApi().getMetricClientForRegion(regionId);
         for (Metric metric : metricClient.listMetrics()) {
            // ...
         }
      }

And for the jenkins users, we also have an api for remote job and computer control, at org.jclouds.labs/jenkins!

      ContextBuilder contextBuilder = ContextBuilder.newBuilder("jenkins");
      RestContext localhost = contextBuilder.build();
      
      Node master = localhost.getApi().getMaster();
      localhost.getJobClient().createFromXML("newJob", xmlAsString);

This is especially helpful with the new jclouds-plugin, which uses jclouds to spin up new slaves and publish artifacts to BlobStore. Tons more in there, too.

Definitely play around, and let us know how it works!

March 23, 2012

Take a peek at vCloud Director 1.5, OpenStack, and VirtualBox

The jclouds team have been working very hard lately, particularly on a few new apis.  We’ve decided to cage them no longer and cut jclouds 1.5.0-alpha.1:

Most notably, we’ve added the openstack-nova api, and three new providers, all of which discovered via OpenStack Keystone v2.0:

  • trystack-nova
  • hpcloud-compute
  • hpcloud-objectstorage

Here’s how to boot up a new machine and add your login using the new hpcloud-compute provider in clojure:

(use 'org.jclouds.compute2)
(import 'org.jclouds.scriptbuilder.statements.login.AdminAccess)
(def compute
(compute-service "hpcloud-compute" "tenantId:ACCESSKEY" "SECRETKEY"
:slf4j :sshj)) (create-node compute "test"
(build-template compute { :run-script (AdminAccess/standard) } ))

Here’s an example of how to do the same on TryStack, authenticating w/ user & pass as opposed to key, via our java example:

java -jar target/compute-basics-jar-with-dependencies.jar \
trystack-nova tenantId:user password mygroup add

In the group org.jclouds.labs, you’ll find two more new and notable members of the jclouds family:

  • vcloud-director: supports 250 user and admin operations defined in the vCloud Director 1.5 ReST API
  • virtualbox: start a group of vms on your laptop, provided an iso location for ubuntu

You can try out virtualbox like any other api.  For example, you can use the clojure above, only changing how you create the connection slightly:

(def compute (compute-service "virtualbox" "administrator" "12345" :sshj :slf4j))

The code in labs will certainly change before we release a beta, but feel free to check them out.  Meanwhile, you can try them out and give us feedback on #jclouds irc freenode or jclouds-dev google group!

Finally, many thanks to the dozen contributors who’s work is in this alpha, and particularly HP and VMware for sponsoring substantial effort.

Oh, and don’t forget to clean up your nodes :)

(destroy-nodes-matching compute (constantly true))
March 9, 2012

EC2 gets rid of that “if instancetype.is32bit” statement

2 weeks ago, I started a bit of a rant on EC2’s 32bit-only vms.

32bit vms make me angry

hey, ! deprecate m1.small for a new 64bit t1.small

any chance of a future blog titled: m1.small is dead. long live t1.small!

This occurred on freenode and twitter, mainly due to pent-up frustration, set off by needing a separate if statement in our JDK installer just to accommodate the lame 32bit m1.small instance.

I’m not (quite) self-absorbed enough to think Jeff Barr’s latest announcement was in response to this.  Even if I was, my mention never reached him! (my tweet went to the less interested jeffbar, yeah the only one ‘r’ one)  Nevertheless, our prayers were answered, and reported by the real @jeffbar:

EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client

Here’s the jist: Instead of deprecating the old m1.small, they updated it to support 64bit images (and also the c1.medium).  To sweeten the deal, they also threw in a new m1.medium size, which is roughly a 2x m1.small.

Now, all users producing images don’t have to make a 32bit option just cause they need more ram than t1.micro.  In fact, many will probably drop the maintenance entirely.

For jclouds users, action is simple, update to version 1.3.2 which removes the restriction of m1.small’s only being able to use 32bit amis.