Papyrus UML support in Hawk

by tevirselrahc at January 19, 2018 03:00 PM

mondo project The Mondo Project‘s mondo-hawk now supports my models!

You can read more about this on their announcement page!

Note that they only index my semantic model (i.e., the “.uml” files) and not the diagrams, which makes sense in their context given that diagrams are not semantic elements.

Still, this is so cool!


by tevirselrahc at January 19, 2018 03:00 PM

Papyrus has layers (like parfait?)

by tevirselrahc at January 17, 2018 04:17 PM

The Papyrus layers functionality has been in incubation for a while, and some may be wondering what it’s all about:

The Layers mechanism allows to build different views of an underlying UML diagrams by applying selections rules as well as graphical transformation operators.

You can read the description of this function in the Papyrus Wiki, and you can also see it in action in this YouTube video:

 


by tevirselrahc at January 17, 2018 04:17 PM

KubeCon CloudNativeCon 2017 Takeaways + 2018 Predictions

by Chris Aniszczyk at January 16, 2018 04:48 AM

It was a crazy 2017 for me with 300,000 miles of business travel, but it was all worth it to experience every major cloud provider adopt Kubernetes in some fashion and grow our community to 14 projects total! Also, it was amazing to help host 4000+ people in Austin for KubeCon/CloudNativeCon, where it actually snowed!

I’d like to share some personal take aways I had from the conference (of course with accompanying tweets) that will serve as predictions for 2018:

Exciting Times for Boring Container Infrastructure!

One of the themes from the conference was that the Kubernetes community was working hard to make infrastructure boring. In my humble opinion, Kubernetes becomes something like “POSIX of the cloud” or “Linux of the Cloud” where Kubernetes is solidifying kernel space but the excitement should be in user space.

The Open Container Initiative (OCI) community also held a meeting where it celebrated its v1.0 release to make containers a bit more boring and standardized.

In 2018, look for the boring infrastructure pattern to continue, the OCI community is planning to make distribution a bit more boring via a proposed distribution API specification. I also predict that some of the specialized/boutique cloud providers who haven’t offered Kubernetes integration will do so finally in 2018.

CNCF + KubeCon and CloudNativeCon: Home of Open Infrastructure

CNCF has a community of independently governed projects, as of today which there are 14 of covering all parts of cloud native. There’s Prometheus which integrates beautifully with Kubernetes but also brings modern monitoring practices to environments outside of cloud native land too! There’s Envoy which is a cloud native edge and proxy, that integrates with Kubernetes through projects like Contour or Istio, however, Envoy can be used in any environment where a reverse proxy is used. gRPC is a universal RPC framework that can help you build services that run on Kubernetes or any environment for that matter! There are many other CNCF projects that have use cases outside of just purely a cloud native environment and we will see more of that usage grow over time to help companies in transition to a cloud native world.

In 2018, look for CNCF conferences continue to grow, expand locations (hello China) and truly become the main event for open source infrastructure. In Austin it was incredible to have talks and people from the Ansible to Eclipse to JVM to Openstack to Zephyr communities (and more). I can’t think of any other event that brings together open source infrastructure across all layers of the cloud native landscape.

Moving up the Stack: 2018 is Year of Service Meshes

Service meshes are fairly a new concept (I highly recommend this blog post by Matt Klein if you’re new to the concept) and will become the middleware of the cloud native world. In CNCF, we currently host Envoy and linkerd which helped poineer this space. In 2018, I expect more service mesh related projects to be open sourced along with more participation from traditional networking vendors. We will also see some of the projects in this space to mature with real production usage.

Cloud Native AI + Machine Learning: Kubeflow

In 2018, ML focused workloads and projects will find ways to integrate with Kubernetes to help scale and encourage portability of infrastructure. Just take a look at the kubeflow project which aims to make ML with Kubernetes easy, portable and scalable. Note, this doesn’t mean that AI/ML folks will have to become Kubernetes experts, all this means is that Kubernetes will be powering more AI/ML workloads (and potentially even sharing their existing cloud native infrastructure). I expect more startups to form in this space (see RiseML as an example), look to see a “cloud native” AI movement that focuses on portability of workloads.

Developer Experience Focus and Cloud Native Tooling

One of my favorite keynotes from KubeCon was Brendan Burns speaking about metaparticle.io, a standard library for cloud native applications. I completely agree with his premise that we need to democratize distributed systems development. Not everyone developer needs to know about Kubernetes the same way not every developer needs to understand POSIX. In 2018, we are going to see an explosion of open source “cloud native languages” that will offer multiple approaches to democratizing distributed systems development.

Also in 2018, I expect us to see growth in cloud native development environments (IDEs) to provide better developer experience. As an example, for those that were wondering why there was an Eclipse Foundation booth at KubeCon, they were demoing a technology called Eclipse Che which is a cloud native IDE framework (your workspace is composed of docker/container images). Che is a framework that helps you build Cloud Native IDEs too, for example, OpenShift.io is OpenShift integrated with Che to provide you a fully blown online development experience.

Finally in 2018, I expect the developer experience of installing Kubernetes applications improved, including the underlying technology for doing so. For example, the Service Catalog work and websites like kubeapps.com showcase what is possible in making it easier for people to install Kubernetes app/integrations, we’ll see this grow significantly in 2018. Also I predict that the Helm community will grow faster than it has before.

Diversity and Inclusion

One of my favorite take aways from the conference was the focus on diversity and inclusion within our community:

We (thank you amazing diversity committee) raised $250,000 and helped over 100 diversity scholarship recipients attend KubeCon/CloudNativeCon in Austin. In 2018, I predict and truly hope some other event will match or beat this.

Anyways, after a crazy 2017, I can’t wait to grow our communities in 2018.

 

 


by Chris Aniszczyk at January 16, 2018 04:48 AM

EE4J Code Arrives

by Mike Milinkovich at January 15, 2018 04:51 PM

Last week the EE4J project achieved an important milestone when the source code for the API and reference implementation of JSON-P JSR-374 project was pushed by Dmitry Kornilov into its GitHub repository in the EE4J organization. This is the first project of the initial nine proposed to reach this stage.

This may seem like a small step in a very large process, but it is a concrete demonstration of the commitment to move forward with the migration of Java EE to the Eclipse Foundation. The Oracle team and the Eclipse Foundation staff had a ton of work to do to make this possible. This is definitely one of those cases where the visible code contributions are just the visible tip of an iceberg’s worth of effort.

Here are just a few examples of the work that went on to get to this stage:

  • The names of the projects such as Glassfish represent important trademarks in the industry. Oracle transferred ownership of these project names to the Eclipse Foundation so that they can be held and protected for the community.
  • The EMO staff reviewed the projects proposals, ran the project creation review, provisioned the repositories and set up the committer lists.
  • The Oracle team packaged up the source code and updated the file headers to reflect the new EPL-2.0 licensing.
  • The EMO IP staff scanned the code and ensured that all was well before approving it for initial check-in.

Now that the collective team has run through this process with JSON-P we will be working to get the remaining eight initial projects pushed out as quickly as possible. Hopefully by the end of this month. Meanwhile, more projects will be proposed and we will be migrating a steady stream of Java EE projects into EE4J.

Exciting times!


by Mike Milinkovich at January 15, 2018 04:51 PM

Eclipse IoT Day Santa Clara 2018 | Call for Papers

January 15, 2018 02:15 PM

We're happy to announce the Eclipse IoT Day Santa Clara, taking place May 14, in co-location with IoT World 2018.

January 15, 2018 02:15 PM

Machine Learning Formatting with Xtext: Part II

by Holger Schill (schill@itemis.com) at January 12, 2018 11:12 AM

Hi folks! Some of you might have seen my earlier blogpost on Machine Learning Formatting. It was more or less meant as an advertiser for the Eclipse Democamp Munich. As promised, here comes the follow up blogpost with more details.

Codebuff is a project started by Terrence Parr. The project aims the problem that writing a codeformatter is hard and will not fit all needs. Therefore people want to customize it in every way since formatting is a very individual thing.

Codebuff tries to format code by taking already formatted examples and learning out of that how to format code based on the same grammar. To do that it needs the corresponding lexer and parser. The lexer is used to get the tokens and the parser to create an abstract syntax tree out of the tokens. Based on that information Codebuff tries to mimics the behavior of a developer. Sounds simple, but how does that work?  First of all code consists of tokens.

tokens.png


Depending on tokens and the location a developer would normally make a decision to introduce or delete whitespaces. A whitespace could be a simple space, a newline along with an indentation or alignment. Another valid decision could also be to do nothing. To make a decision you need certain criteria. Codebuff works with 21criterias, so called features that are taken into account. The following figure shows the 5 most obvious features.

Bildschirmfoto 2017-12-19 um 16.26.00.png


Based on those 21 
features Codebuff analyses the given examples and computes token properties.

Bildschirmfoto 2017-12-21 um 14.56.14.png


When training is done all token properties are stored as vectors in a matrix along with the whitespace information.

Matrix.png


From this point on we should start to speak about machine learning. What we want to achieve is finding a match in the computed matrix of token properties and get the whitespace to inject into the given document at a certain position.

First of all we have to compute where to look at in the examples. For that reason we look at our current position in the document and compute the token property (vector) like we did it in the training without storing it to the matrix.

Vector.png


Now that we have that vector we need an algorithm to find a match in the matrix and apply the right whitespace to the document.

Format.png


For doing that Codebuff uses the simplest machine learning algorithm that exists – the 
K-nearest neighbors algorithmThe algorithm uses a function to compare a token property to the entries of the existing training-matrix. To do that the a constant named k needs to be specified to define how many nearest neighbors should be used. Mostly k is computed once by taking the root of the number of examples in the corpus, but for Codebuff k is fixed to 11.

Bildschirmfoto 2018-01-03 um 16.19.08.png


As shown above, experiments with different languages showed that the median error rate is nearly constant from k = 11 on, so taking this as a constant makes sense for performance reasons. 

As we now know the value of k, we can start to find most similar token properties to the one we just computed at the location to format.

KNN.png 

Out of that the algorithm uses a function to weigh the best whitespace. In Codebuff it takes simply the whitespace that appears most often – very simple but efficient. Codebuff's theoretical background is explained in more depth here and you can find an entertaining talk by Terence on YouTube.

Looking at the way Codebuff works you might ask yourself how many example files you need to get good results? The answer is pretty simple: after a lot of experiments the Codebuff guys found out that from 10 example files on the results will not get better nor worst. The next figure shows for some different languages that after 10 files the error rate stays around the same.

Bildschirmfoto 2017-12-21 um 15.18.06.png


So – what has Codebuff to do with Xtext?

Now as we know a bit of the theoretical background around Codebuff you may wonder what all that stuff has to do with Xtext.

In Xtext you can create various DSLs for very different domains. For each language you need to specify a formatter. Luckily Xtext comes with two different APIs. The older one relies on tokens only, that means you can define how your DSL should be formatted on a lexical level. The newer API relies on the semantic model in addition. Depending on the state of the abstract syntax tree you can treat things different. Cool – but customization of your formatter is not coming out of the box and we know that formatting is an individual thing. So you would end up in a lot of effort to write a reasonable formatter and make customizations possible. Even when you did your best you will always find a special case that someone wants to treat differently. Codebuff sounds like a nice idea to solve that problem...

To marry Xtext and Codebuff together you don't need to do much. Ok – Xtext relies on AntLR 3 and bringing AntLR 4 on the classpath will take you into hell, but there is a pragmatic way around that.

DISCLAIMER: From this point on we are talking about a prototype – not ready for production!

First we wrote a so-called GeneratorFragment. To understand this you would have to understand how Xtext works under the covers, but for now let's just stay with the fact that Xtext generates it's DSL specific infrastructure out of the Xtext grammar and a GeneratorFragment is one piece of the overall generator.

To come back to our issue – the GeneratorFragment needs to generate an AntLR 4 grammar. Out of that we need to generate the AntLR 4 lexer and parser. To do that we need the AntLR 4 generator but we are not allowed to bring that stuff on the classpath. Let's assume we managed to do that – how could we compile the generated code without AntLR 4 on the classpath? Even when we managed that we are not allowed to bring Codebuff on the classpath as it carries AntLR 4 under the covers. Even when we managed that – does Codebuff have an API that fits for us? The documentation just deals with a call of a main method with a bunch of parameters....

Ok – step by step.

1. Generate the grammar

The major difference between AntLR 3 and 4 is that AntLR 4 does not allow syntactic predicates anymore since it handles left recursion now directly. AntLR 3 is not able to do that and for that reason grammars need to be left refactored. By doing that, syntactic predicates guide the parser on it's way to get rid of ambiguities and make decisions. The other thing is that the syntax has changed a bit from AntLR 3 to 4. Beside that Codebuff has a special need. Whitespaces and comments need to go to the hidden channel.

2. Generate lexer and parser

To achieve that we are downloading the necessary jar on demand once, like we already do it in Xtext for AntLR 3. To invoke the so called Tool of AntLR 4 we created a URLClassLoader that points to the jar. As Xtext uses AntLR 3 we need to leave the parent classloader empty to avoid confusions. After that we need to use reflection to call the right API.

As a fun fact we observed that the AntLR 4 Tool has two methods to process a grammar and both produce lexer and parser - but they are not equal. After fighting with that situation we found out that the method process is not the right choice as Codebuff got stuck with that the generated sources. ProcessGrammarsOnCommandLine produced the right result since it reorders the tokens by calling  sortGrammarByTokenVocab and then called process.

Of course the generated classes are stored in a separated folder that is no source folder. Otherwise we would end up in compile errors in the project. ;-)

3. Compile lexer and parser

We cannot simply compile the lexer and parser like the rest of the code since AntLR 4 is not on the classpath. To achieve compiling we used the javax.tools.JavaCompiler directly. To use it we need to bring referenced classes into a special classloader. For this use-case the AntLR 4 jar is enough. After that we need to handle in the files to compile - that's it. Of course the compiled class files are stored in a separated folder as well. ;-)

4. Use Codebuff with the compiled classes without bringing it on the classpath

To achieve that we used the same trick as for AntLR 4 and downloaded Codebuff on demand once. So we ended up in two jars, AntLR 4 and Codebuff, a grammar and compiled lexer and parser. Now we need to combine all those things.

Codebuff does not really have an API to use it somewhere else then the commandline. Normally the Tool class needs a bunch of information including a destination file.

Bildschirmfoto 2018-01-04 um 10.48.02.png


The formatter API in Xtext does not write to a file directly. It needs to set the formatted text as a string. Because of that the given approach does not work for use. We digged a bit in the code and yes there is a way that is slightly complicated and as you know we have to do everything through reflection ;-)

To explain it in a simple way we will just mention the steps without showing the ugly reflective code.

  1. We need to instantiate a org.antlr.codebuff.misc.LangDescriptor that holds all necessary information about the language. That includes the folder of the examples, a regex to identify examples, the compiled lexer and parser, the root rule of the grammar, the amount of spaces that make an indentation and the rule for comments.
  2. We need to load all examples to invoke the train method that fills the matrix with token properties.
  3. We need to parse the document that we want to format.
  4. We need to instantiate the formatter and fill in the necessary information including the training result.
  5.  We need to invoke the formatter to get a string back that represents the formatted document.


Sounds simple but we have to admit that it was pain. If Codebuff would have a more user friendly API, it would have been much simpler – we are planning to create a pullrequest to Codebuff to make it easier.

On Xtext side the story is simple – we need to bind a org.eclipse.xtext.ui.editor.formatting.IContentFormatterFactory to create our org.eclipse.jface.text.formatter.IContentFormatter that has only one method that gets the document that holds all necessary information and we can directly set the new formatted text.

The only tricky part that is left is classloading. We need to have Codebuff, AntLR4 and the compiled lexer and parser in a classloader. Of course a URLClassLoader solves the problem as we already downloaded the necessary jars and compiled the lexer and parser.

As a wrap up we can say that Codebuff creates a nice way to get rid of an old problem – but it's not perfect. For the future of Xtext it would be nice to have something like that included. Currently we are looking into alternative approaches and enhancements of the existing Codebuff codebase. Stay tuned.


by Holger Schill (schill@itemis.com) at January 12, 2018 11:12 AM

Announcing Orion 17

by Mike Rennie at January 11, 2018 06:52 PM

We are pleased to announce the seventeenth release of Orion, “Your IDE in the Cloud”. You can run it now at OrionHub, from NPM or download the server to run your own instance locally.

Once again, thank you to all committers and contributors for your hard work this release. There were 86 bugs and enhancements fixed, across 248 commits from 13 authors!

This release was focussed entirely on making compatibility, stability and overall quality improvements to the Node.js server.


by Mike Rennie at January 11, 2018 06:52 PM

German TUV Sud Joins Eclipse openPASS Working Group

January 11, 2018 04:10 PM

On Jan 1, 2018 the German TUV Sud has joined as a new member of the Eclipse Foundation & Eclipse openPASS Working Group.

January 11, 2018 04:10 PM

SWT Spy – debugging SWT layouts

by Maximilian Koegel and Jonas Helming at January 10, 2018 01:28 PM

Do you have to debug an existing SWT view and you are not sure what elements are really on screen? Are you tired of manually coloring composites to find out what’s wrong in your layout?

It is time to remember an oldie but goodie: The SWT spy!

It is a fairly old tool, but still very useful, so let us have a quick look.

Spy can augments a running tool and provide valuable information to debug any SWT application. This can be the IDE in case you are a plugin developer or any RCP or Eclipse 4 application. You can get SWT spy from this location:

http://download.eclipse.org/eclipse/updates/4.7

Either add it to your target platform or directly install it into your IDE.

In case you install SWT spy into the IDE, you can open it by pressing:

SHIFT+CTRL+ALT+F9

If you would like to use SWT spy in a custom application, you need to include the bundle “org.eclipse.tools.layout.spy” into your product. To trigger the spy, there are two options:

  1. If you run on the compatibility layer (3.x), the key binding will also work
  2. If you are running on Eclipse 4, you can embed the following code to open the SWT spy window:
final LayoutSpyDialog popupDialog = new LayoutSpyDialog(shell);
popupDialog.open();

The SWT spy will open as a separate dialog, which is connected to the running application. It allows for the inspection of any SWT element with a focus on layout information. To navigate you can either browse the hierarchy or directly jump to an element. The hierarchy is shown as a list on the left side. By double clicking on an element within the current container you will walk down the SWT widget tree, the button on the top left corner will send you one level up. To jump to a specific element, click the button “Select Control” and hover over any element in the running application. A yellow border will help you with the selection. If you check the option “Show overlay”, SWT spy will draw a yellow border around the currently selected element.

The text field on the right side shows the layout data of the current parent composite. If you select an element on the left, you see its layout data in the text field below. This is very helpful to debug layouts. Furthermore, you can use existing layouts as a template. The SWT spy directly shows you how to create the layout data for a selected element, e.g.

GridDataFactory.fillDefaults()
    .grab(true, false)
    .align(SWT.FILL, SWT.BEGINNING)
    .minSize(0, 0)
    .hint(107, SWT.DEFAULT)

During our work on EMF Forms, we often develop or adapt renderers. The goal of EMF Forms is to ease the creation of forms, especially concerning the layout. However, before this can happen, you must first create a renderer which will take care of the layout for you. EMF Forms ships with default layouts, but you might want to adapt them.

When debugging the layout of a combination of renderers, it is often helpful to visualize all elements, which are on screen. Sometimes, elements are not visible (e.g. composites) or have the same background color as their parent. For this purpose, the SWT spy allows you to color controls. If you check the option “Color controls …”, the spy will assign different background colors to each level in your widgets tree. This looks ugly, but is helpful in finding hidden composites or to see how much space certain elements occupy.


by Maximilian Koegel and Jonas Helming at January 10, 2018 01:28 PM

RHAMT Eclipse Plugin 4.0.0.Final has been released!

by josteele at January 10, 2018 07:42 AM

We are happy to announce the latest release of the Red Hat Application Migration Toolkit (RHAMT) Eclipse Plugin.

Getting Started

It is now available through JBoss Central, and from the update site here.

What is RHAMT?

RHAMT is an automated application migration and assessment tool.

Example ways to RHAMT up your code:

  • Moving your application from WebLogic to EAP, or WebSphere to EAP

  • Version upgrade from Hibernate 3 to Hibernate 4, or EAP 6 to EAP 7

  • Change UI technologies from Seam 2 to pure JSF 2.

An example of how to run the RHAMT CLI:

$ ./rhamt-cli --input /path/to/jee-example-app-1.0.0.ear --output /path/to/output --source weblogic --target eap:7

The output is a report used to assess and prioritize migration and modernization efforts.

The RHAMT Eclipse Plugin - What does it do?

Consider an application migration comprised of thousands of files, with a myriad of small changes, not to mention the tediousness of switching between the report and your IDE. Who wants to be the engineer assigned to that task? :) Instead, this tooling marks the source files containing issues, making it easy to organize, search, and in many cases automatically fix issues using quick fixes.

Let me give you a quick walkthrough.

Ruleset Wizard

We now have quickstart template code generators.

Rueset Wizard

Rule Creation From Code

We have also added rule generators for selected snippets of code.

Rule Generation From Source

Ruleset Graphical Editor

Ruleset navigation and editing is faster and more intuitive thanks to the new graphical editor.

Graphical Editor

Ruleset View

We have created a view dedicated to the management of rulesets. Default rulesets shipped with RHAMT can now be opened, edited, and referenced while authoring your own custom rulesets.

Ruleset View

Run Configuration

The Eclipse plugin interacts with the RHAMT CLI process, thereby making it possible to specify command line options and custom rulesets.

Run Configuration

Ruleset Submission

Lastly, contribute your custom rulesets back to the community from within the IDE.

Ruleset Submission


You can find more detailed information here.

Our goal is to make the RHAMT tooling easy to use. We look forward to your feedback and comments!

Have fun!
John Steele
github/johnsteele


by josteele at January 10, 2018 07:42 AM

Hazelcast Joins the Eclipse Foundation

by Michael Redlich at January 09, 2018 02:00 PM

Hazelcast, a provider in open source In-Memory Data Grid (IMDG), recently joined the Eclipse Foundation to work with the other members of the Eclipse community to mainly focus on JCache, Eclipse MicroProfile, and EE4J. Greg Luck, CEO at Hazelcast, spoke to InfoQ about Hazelcast joining the Eclipse Foundation.

By Michael Redlich

by Michael Redlich at January 09, 2018 02:00 PM

Eclipse Collections 9.1 Released

by Donald Raab at January 05, 2018 07:39 PM

New Features, more languages, better symmetry and other improvements

Eclipse Collections is a collections framework for Java. It has optimized List, Set and Map implementations with a rich and fluent API. The library provides additional data structures not found in the JDK like Bags, Multimaps and BiMaps. The framework also provides primitive versions of Lists, Sets, Bags, Stacks and Maps with a rich and fluent API. There is support for both Mutable and Immutable versions of all containers in the library.

The Eclipse Collections community continues to grow. There were ten developers who contributed to the Eclipse Collections 9.1.0 release. I want to thank everyone who made a contribution. If this was your first contribution to an open source project, congratulations and welcome!

The release notes for Eclipse Collections 9.1.0 are here:

eclipse/eclipse-collections

Multi-language Website

In addition to code contributions, several developers worked on translating our Eclipse Collections Website to other spoken languages. This is an amazing example of how you can use more than just your coding skills to help open source projects. Thank you to both the contributors and reviewers!

Here are our website links:

A Chinese translation is in the works, and we will be looking to add other languages in the future. Please feel to submit an issue or pull request if you’d like to work on a translation to a spoken language you are fluent in.

The Eclipse Collections Katas

Both of the Eclipse Collections katas are now available in Reveal.js as of this release.

We also had our first ever contribution of a live video demonstration of the Eclipse Collections Pet Kata using Eclipse Oxygen presented in Brazilian Portuguese. Great work and thank you Leo!

https://www.youtube.com/watch?v=1TH0tdU_esQ

It would be great to have multi-language translations of our katas as well. Our katas are hosted now as simple markdown files which get translated to web based slides using Reveal.js. Feel free to submit a pull request if you’d like to translate the katas to another spoken language.

New Collectors

Several new Collector implementations were added to the Collectors2 utility class in this release. We continue to look to provide good integration between the Streams library and Eclipse Collections. Here’s an example of using the new countBy Collector with a Stream of String.

@Test
public void countBy()
{
Stream<String> stream = Stream.of(
"One", "Two", "two", "Three", "three", "tHrEe");

Bag<String> counts =
stream.collect(
Collectors2.countBy(String::toLowerCase));

Assert.assertEquals(1, counts.occurrencesOf("one"));
Assert.assertEquals(2, counts.occurrencesOf("two"));
Assert.assertEquals(3, counts.occurrencesOf("three"));
}

You can now consider using the countBy Collector as an alternative to using the groupingBy and counting alternative from the JDK shown below.

@Test
public void groupingByCounting()
{
Stream<String> stream = Stream.of(
"One", "Two", "two", "Three", "three", "tHrEe");

Map<String, Long> counts =
stream.collect(
Collectors.groupingBy(String::toLowerCase,
Collectors.counting()));

Assert.assertEquals(Long.valueOf(1), counts.get("one"));
Assert.assertEquals(Long.valueOf(2), counts.get("two"));
Assert.assertEquals(Long.valueOf(3), counts.get("three"));
}

There were also Collectors added for groupByEach and groupByUniqueKey on the Collectors2 class in this release.

Symmetric Sympathy Continues

You can now use zip with primitive Lists. There are two varieties of zip for primitive Lists.

  1. Zipping a primitive and an object List.
@Test
public void zipPrimitiveWithObject()
{
MutableList<IntObjectPair<String>> zipped =
IntLists.mutable.with(1, 2, 3)
.zip(Lists.mutable.with("one", "two", "three"));

List<IntObjectPair<String>> expected =
Lists.mutable.with(
PrimitiveTuples.pair(1, "one"),
PrimitiveTuples.pair(2, "two"),
PrimitiveTuples.pair(3, "three"));

Assert.assertEquals(expected, zipped);
}

2. Zipping two primitive Lists of the same type.

@Test
public void zipIntToInt()
{
MutableList<IntIntPair> zipped =
IntLists.mutable.with(1, 2, 3)
.zipInt(IntInterval.oneTo(3).asReversed());

List<IntIntPair> expected =
Lists.mutable.with(
PrimitiveTuples.pair(1, 3),
PrimitiveTuples.pair(2, 2),
PrimitiveTuples.pair(3, 1));

Assert.assertEquals(expected, zipped);
}

There is a zip<Type> method for each primitive type. Here is an example of the zipChar method available on CharAdapter, which is an ImmutableCharList. We can obtain a CharAdapter now simply by using the new Strings factory class.

@Test
public void zipCharToChar()
{
ImmutableList<CharCharPair> zipped =
Strings.asChars("hello")
.zipChar(Strings.asChars("hello").asReversed());

List<CharCharPair> expected =
Lists.mutable.with(
PrimitiveTuples.pair('h', 'o'),
PrimitiveTuples.pair('e', 'l'),
PrimitiveTuples.pair('l', 'l'),
PrimitiveTuples.pair('l', 'e'),
PrimitiveTuples.pair('o', 'h'));

Assert.assertEquals(expected, zipped);
}

Abandoning Symmetry for a better design

Sometimes providing good symmetry may not be the best solution to a problem. There has been a method called zipWithIndex available on Eclipse Collections object collections for a very long time. ZipWithIndex was added before we had primitive collections in Eclipse Collections, so its return type unfortunately boxes an Integer for the index. I did not want to add a primitive version of of the same API. Instead, I added a new method to both object and primitive Lists (Symmetry!) called collectWithIndex. CollectWithIndex can be used to implement zipWithIndex by collecting to a ObjectIntPair.

@Test
public void whereIsZipWithIndex()
{
MutableList<Pair<String, Integer>> original =
Lists.mutable.with("1", "2", "3")
.zipWithIndex();

MutableList<ObjectIntPair<String>> improved =
Lists.mutable.with("1", "2", "3")
.collectWithIndex(PrimitiveTuples::pair);

Assert.assertEquals(
original.get(0).getTwo().intValue(),
improved.get(0).getTwo());
Assert.assertEquals(
original.get(1).getTwo().intValue(),
improved.get(1).getTwo());
Assert.assertEquals(
original.get(2).getTwo().intValue(),
improved.get(2).getTwo());
}

Ideally, zipWithIndex would have returned ObjectIntPair, but this type wasn’t available when the API was originally added. Since we don’t like breaking backwards compatibility on existing methods unless there is a very compelling reason to, we’ll probably never change the return type for zipWithIndex. The collectWithIndex method can return any type you want, not just a Pair. I believe collectWithIndex will prove to be a more useful method than zipWithIndex in the long run.

And of there is more…

Have a look at some of the other features listed in the release notes. We look forward to seeing more contributors to Eclipse Collections in the future.

Happy New Year and have fun using Eclipse Collections 9.1 in your Java projects!

Eclipse Collections is open for contributions. If you like the library, you can let us know by starring it on GitHub.


by Donald Raab at January 05, 2018 07:39 PM

Time for a New Challenge

by Ian Skerrett at January 04, 2018 01:39 PM

The new year brings a big change; I will be leaving the Eclipse Foundation at the end of January. This has been a difficult decision to make but it is time to take on a new challenge.

I am extremely proud to have been part of the Eclipse Foundation and Eclipse community since the early days. I fondly remember the chaotic and exciting early years of setting up the Foundation and the explosive growth of projects, members and users. The Eclipse Foundation was one of the first open source foundations that showed how professional staff can help bring together companies and open source developers to create innovative open source communities. In some ways, the Eclipse Foundation became the template for many of the new open source foundations started in the last 5 years.

It is great to be leaving when the Foundation is gaining tremendous momentum. In particular, EE4J, Eclipse Microprofile, DeepLearning4J, OpenJ9 and other projects are all showing Eclipse is becoming the place for open source enterprise Java. The Eclipse IoT working group continues to grow and is certainly the largest IoT open source community in the industry. The Foundation is definitely on a roll.

Of course I will miss the amazing people at the Foundation and in the Eclipse community. I am privileged to have worked with so many individuals from many different organizations. One of the best parts of my role is being able to learn from some of the world’s best.

It is time to start looking for a new challenge. I am going to take some time to look for a new position. My passion is working with smart technologists to create and implement strategies to promote the market adoption of new technologies and solutions. Today there are some fascinating new technologies being introduced, like AI/ML, Blockchain/Distributed Ledgers, Cloud Native Computing, IoT, etc. It is an exciting time to work in the technology industry and I look forward to taking on a new challenge.

My last day with the Eclipse Foundation will be January 26, 2018. The technology industry is very small so I hope and expect to keep in touch with many people from the Eclipse community. I will continue to be active on my blog, Twitter and LinkedIn.



by Ian Skerrett at January 04, 2018 01:39 PM

Following the Community's Lead

by Doug Schaefer at January 03, 2018 09:35 PM

I posted this tweet a couple of weeks ago and thought I'd explain it a bit more.

I wrote it shortly after posting the final bits for the CDT 9.4 release and was starting to think about my plans for 2018. And certainly the current state of Eclipse tools projects including the CDT has been weighing on my mind.

When QNX started the CDT project, we embraced open source as a means of sharing in a common effort to build a great C/C++ IDE. We didn't have a huge team and we knew that many platform vendors were in a similar situation. By working together, we all benefit. And for many years it worked and we now have the embedded industry's most popular IDE that we all should be proud of.

But times they are a changing. As I blogged last year around this time, web technologies are becoming the platform of choice, not just for web, but with the help of Electron, for the desktop as well. We can not discount the popularity of Visual Studio Code and the ecosystem that has quickly developed around it. And with shared technologies like language servers, the door is being opened to choice like we've never had before.

So as I was inferring from my tweet, it's time to take stock of where we are and where we are going. I was encouraged with the discussions I had at EclipseCon around the future of CDT. I've done what I can to help keep things rolling but at the end of the day the community leads the way and I need to be honest with myself and my customers and go where they go.


by Doug Schaefer at January 03, 2018 09:35 PM

The 4am Jamestown-Scotland ferry and other optimization strategies

by Donald Raab at January 03, 2018 05:16 AM

When performance is important, so is understanding your available options.

Shortcuts sometimes aren’t.

Happy New Year!

I thought I would start out 2018 with a performance optimization story from 2017.

Takeaways from this blog

  • Java Iteration Pattern Optimization Strategies
  • A few Eclipse Collections and Java Stream Iteration Pattern options.
  • Recommendations at the end

A shortcut with a twist

On January 2nd 2017, I sat with my family in our Honda Pilot on a pier at 3:30am for a half hour waiting for the 4am Jamestown-Scotland ferry to arrive. I had come to the literal end of the road on a shortcut that wasn’t exactly as I had expected. I decided to take the shorter distance route on my car’s Nav System to avoid having to go north on Interstate 95 only then to have to go south to get to Williamsburg, Virginia. I’ve gotten stuck in bumper to bumper traffic in Virginia on Route 95 on late night rides coming back from Florida a few times before. When we got to the end of the road on our shorter route, the Nav System indicated the next turn was to get on the ferry (see picture above).

I was willing to take slower local roads, especially since it was early in the morning and there would be no traffic on them. We discovered too late that the path that our chosen path included a ferry ride. At this point, we only had two options. We could wait for the ferry and hope it was running, or turn around and add another 3 to 4 hours to our trip. A classic Hobson’s Choice. We waited for the ferry. It turned out to be a fun experience once we parked our car on the ferry, but I would have preferred an alternative at 4am after driving 14 hours.

“Two roads diverged in a wood…” — Robert Frost

I certainly took the one less traveled by. I did learn a new route that I didn’t know before for getting to Williamsburg from Orlando, as well as the planning required to optimize that route with the ferry schedule.

What does this trip have to do with Eclipse Collections, you may ask? Well, the path I took was the Serial (one lane Colonial era roads), Lazy (ferry does the work until you get to the dock), and Boxed (your car is literally boxed in on the ferry by other cars) — just one of many options you can choose with Eclipse Collections and Java Streams.

“Premature optimization is the root of all evil” — Donald Knuth

Readability should be prioritized above performance when writing code. However, it helps to know what your available performance optimization options are, before you discover last minute your only option is stopping and waiting for the next ferry. You may actually be able to achieve better performance without sacrificing readability. In fact, there may be options you were unaware of previously that improve both readability and performance.

There is a set of Iteration Pattern Optimization Strategies that I believe all developers should become aware of so they can appropriately tune their code for the best performance.

Don’t guess when optimizing code. First prove you have a problem that needs to be fixed. Then benchmark any solutions you think may help to prove that they actually do.

Travelers Beware: You can lose many hours of your life measuring performance optimization benefits. The tests I have run below take 45–50 minutes to run each time. I had to run them several times along with unit tests to validate that the results were the same across all similar tests. When you see the charts, you may be at first compelled by the graphs in terms of wanting to change your code to be more “optimal”. Optimal may not equate to noticeably faster in terms of your application’s overall performance. Each of these tests take at most hundreds of milliseconds to run. They are all “fast”, because they are all in memory. The optimal solutions may only accumulate savings over a large number of executions. If you happen to see a more readable solution you were not aware of here, go for that one.

Iteration Pattern Optimization Strategies

Do you know how to leverage all of these strategies separately and together to increase performance without sacrificing readability?

  • Eager — executes immediately with potential optimizations specific to each algorithm and data structure. Eager algorithms are as close to a hand coded for-loop as you will get, so they are easy to understand and debug. I prefer eager as the default option for iterating over collections. It is the simplest and usually most succinct and readable solution available. I consider every other solution a potential optimization, which may prove pre-mature.
  • Primitive — If you can avoid boxing primitives, you can reduce memory cost and potentially increase performance. I always use primitive collections and algorithms when I can.
  • Lazy — executes only when a terminal operation is called. Optimizations include reducing the amount of memory required and total computation when multiple operation are executed. Short-circuiting effects can really help performance when run lazily. I prefer lazy as soon as I am executing multiple operations that would result in temporary collections being created.
  • Parallel — It costs more to run in parallel. You need the right data size, algorithm and multiple cores. If you have all of these, you may benefit from running in parallel. Measure it to prove it.

Eager vs. Lazy — Understanding how they work

Let’s take a list of five integers and perform a filter, map, and reduce set of operations both eagerly and lazily.

@Test
public void eagerVsLazy()
{
long eagerSum = Lists.mutable.with(1, 2, 3, 4, 5)
.tap(i -> System.out.println("eager select: " + i))
.select(i -> i % 2 == 0)
.tap(i -> System.out.println("eager collect: " + i))
.collectInt(i -> i * 2)
.tap(i -> System.out.println("eager sum: " + i))
.sum();
System.out.println(eagerSum);

long lazySum = Lists.mutable.with(1, 2, 3, 4, 5)
.asLazy()
.tap(i -> System.out.println("lazy select: " + i))
.select(i -> i % 2 == 0)
.tap(i -> System.out.println("lazy collect: " + i))
.collectInt(i -> i * 2)
.tap(i -> System.out.println("lazy sum: " + i))
.sum();
System.out.println(lazySum);

Assert.assertEquals(eagerSum, lazySum);
}

Except for the additional call to asLazy in the lazy example, the code should look identical. The printed results are as follows:

eager select: 1
eager select: 2
eager select: 3
eager select: 4
eager select: 5
eager collect: 2
eager collect: 4
eager sum: 4
eager sum: 8
12
lazy select: 1
lazy select: 2
lazy collect: 2
lazy sum: 4
lazy select: 3
lazy select: 4
lazy collect: 4
lazy sum: 8
lazy select: 5
12

Notice how the order of execution changes on the lambdas in the lazy case. In the eager case, two additional lists are created as intermediate results during the execution. A List of Integer with two Integers (2, 4) and then an IntList with two ints (4, 8) are created before the final call to sum. In the lazy case, there are no intermediate collections created. This results in less garbage being generated. This is why I prefer lazy execution when there are multiple operations involved. If there was a single operation involved, then I would default to using the eager solution.

If we look at the serial Stream solution, it’s execution order will be the same as the lazy Eclipse Collections solution.

@Test
public void stream()
{
int streamSum = Lists.mutable.with(1, 2, 3, 4, 5)
.stream()
.peek(i -> System.out.println("stream filter: "+ i))
.filter(i -> i % 2 == 0)
.peek(i -> System.out.println("stream map: "+ i))
.mapToInt(i -> i * 2)
.peek(i -> System.out.println("stream sum: "+ i))
.sum();
System.out.println(streamSum);
}

Here is the output:

stream filter: 1
stream filter: 2
stream map: 2
stream sum: 4
stream filter: 3
stream filter: 4
stream map: 4
stream sum: 8
stream filter: 5
12

Lazy + Parallel = Harder to Follow

Using Eclipse Collections lazy parallel with a batch size of one so we can see the results for a very small list.

@Test
public void parallel()
{
long parallelSum = Lists.mutable.with(1, 2, 3, 4, 5)
.asParallel(Executors.newWorkStealingPool(), 1)
.select(i -> {
System.out.println("parallel select: " + i);
return i % 2 == 0;
})
.collect(i -> {
System.out.println("parallel collect: " + i);
return i * 2;
})
.sumOfInt(i -> {
System.out.println("parallel sum: " + i);
return i;
});
System.out.println(parallelSum);
}
Run 1:
parallel select: 2
parallel select: 1
parallel select: 4
parallel collect: 4
parallel select: 3
sum: 8
parallel select: 5
parallel collect: 2
sum: 4
12
Run 2:
parallel select: 1
parallel select: 3
parallel select: 2
parallel select: 5
parallel select: 4
parallel collect: 2
parallel collect: 4
parallel sum: 4
parallel sum: 8
12
Run 3:
parallel select: 4
parallel select: 2
parallel collect: 2
parallel select: 5
parallel select: 3
parallel select: 1
parallel sum: 4
parallel collect: 4
parallel sum: 8
12

The result is consistent between runs, but the order of execution of lambdas is not guaranteed nor consistent.

Using parallel Streams:

@Test
public void parallelStream()
{
int streamSum = Interval.oneTo(5).toList()
.parallelStream()
.peek(i -> System.out.println("stream filter: "+ i))
.filter(i -> i % 2 == 0)
.peek(i -> System.out.println("stream map: "+ i))
.mapToInt(i -> i * 2)
.peek(i -> System.out.println("stream sum: "+ i))
.sum();
System.out.println(streamSum);
}
Run 1:
stream filter: 4
stream filter: 1
stream map: 4
stream filter: 2
stream sum: 8
stream filter: 3
stream filter: 5
stream map: 2
stream sum: 4
12
Run 2:
stream filter: 5
stream filter: 1
stream filter: 3
stream filter: 2
stream filter: 4
stream map: 2
stream map: 4
stream sum: 4
stream sum: 8
12
Run 3:
stream filter: 2
stream filter: 4
stream map: 2
stream map: 4
stream sum: 8
stream filter: 1
stream filter: 3
stream filter: 5
stream sum: 4
12

Measure, Execute and Repeat.

I am going to show different options and their performance characteristics for a set of use cases using a million randomly generated integers stored in Lists. These are not likely to be the use cases you will encounter in production code, but they should hopefully illustrate some options you may not have been aware of next time you find a bottleneck you were not expecting in your basic Java data structures and algorithms. I will demonstrate the performance differences between using object and primitive lists, eager and lazy APIs, with both serial and parallel execution, with four different use cases.

In each use case, I share what I observed — expected and unexpected. I only observed. I have not dug into the why the results were what they were. “The why” perhaps is a topic for another blog.

Use Cases — Filter, Map, Reduce, and Filter/Map/Reduce

1. Filter even integers into a List
2. Multiply the integers by 2 and storing the result in a List
3. Sum all the integers into a long
4. Filter/Map/Reduce (Filter Evens, Multiply x 2, Sum into long)

The Data — 1,000,000 Integers

private List<Integer> jdkList;
private MutableList<Integer> ecList;
private IntList ecPrimitiveList;
private ExecutorService executorService;
@Setup
public void setup()
{
PrimitiveIterator.OfInt intGenerator =
new Random(1L).ints(-1000, 1000).iterator();
this.ecList =
FastList.newWithNValues(1_000_000, intGenerator::nextInt);
this.jdkList = new ArrayList<>(1_000_000);
this.jdkList.addAll(this.ecList);
this.ecPrimitiveList =
this.ecList.collectInt(i -> i, new IntArrayList(1_000_000));
this.executorService = Executors.newWorkStealingPool();
}

Hardware

I will be using a MacPro with the following hardware specs to measure the benchmarks:

Processor Name: 12-Core Intel Xeon E5
Processor Speed: 2.7 GHz
Number of Processors: 1
Total Number of Cores: 12
L2 Cache (per Core): 256 KB
L3 Cache: 30 MB
Memory: 64 GB

Software

To illustrate the different options that are available for these particular use cases, I will be using JDK 1.8.0_152 with Eclipse Collections and Streams.

Benchmarking

I am using JMH version 1.19 as the benchmark harness for my tests. I am running 30 warmup iterations, and 20 measurement iterations with 2 forks. I am using Mode.Throughput with the tests so they are easy to read. The numbers are in Operations per Second. The bigger the number, the better the result.

public static void main(String[] args) throws RunnerException
{
Options options = new OptionsBuilder()
.include(".*" + IntListJMHTest.class.getSimpleName() + ".*")
.forks(2)
.mode(Mode.Throughput)
.timeUnit(TimeUnit.SECONDS)
.warmupIterations(30)
.build();
new Runner(options).run();
}

I will highlight in dark green the best overall result in the run. I will highlight in light green the best serial execution result. Where I use EC in a label in the chart it stands for a solution using Eclipse Collections. Where I used JDK, the solution uses a standard JDK approach.

Filter even integers

Filter even numbers from a List of 1,000,000 Integers

Expected:

  • I expected ECParallelEager to perform better.
  • I expected primitive collections to outperform boxed collections.
  • I expected serial eager to outperform serial lazy.

Unexpected:

  • I did not expect parallel streams to perform this poorly.
@Benchmark
public MutableList<Integer> filterECBoxedEager()
{
return this.ecList.select(i -> i % 2 == 0);
}
@Benchmark
public MutableList<Integer> filterECBoxedLazy()
{
return this.ecList
.asLazy()
.select(i -> i % 2 == 0)
.toList();
}
@Benchmark
public MutableList<Integer> filterECParallelEager()
{
return ParallelIterate.select(
this.ecList,
i -> i % 2 == 0,
new CompositeFastList<>(),
false);
}
@Benchmark
public MutableList<Integer> filterECParallelLazy()
{
return this.ecList
.asParallel(this.executorService, 50_000)
.select(i -> i % 2 == 0)
.toList();
}
@Benchmark
public IntList filterECPrimitiveEager()
{
return this.ecPrimitiveList.select(i -> i % 2 == 0);
}
@Benchmark
public IntList filterECPrimitiveLazy()
{
return this.ecPrimitiveList
.asLazy()
.select(i -> i % 2 == 0)
.toList();
}
@Benchmark
public List<Integer> filterJDKBoxedParallelStream()
{
return this.jdkList
.parallelStream()
.filter(i -> i % 2 == 0)
.collect(Collectors.toList());
}
@Benchmark
public List<Integer> filterJDKBoxedStream()
{
return this.jdkList
.stream()
.filter(i -> i % 2 == 0)
.collect(Collectors.toList());
}

Map each integer x 2

Multiply times two, each integer in a List of 1,000,000 Integers

Expected:

  • I expected primitive collections to outperform boxed collections.
  • I expected serial eager to outperform serial lazy.

Unexpected:

  • I did not expected ECParallelLazy to perform so poorly.
  • I did not expect either Stream solutions to perform so poorly.
@Benchmark
public MutableList<Integer> mapECBoxedEager()
{
return this.ecList.collect(i -> i * 2);
}
@Benchmark
public MutableList<Integer> mapECBoxedLazy()
{
return this.ecList
.asLazy()
.collect(i -> i * 2)
.toList();
}
@Benchmark
public MutableList<Integer> mapECParallelEager()
{
return ParallelIterate.collect(
this.ecList, i -> i * 2,
new CompositeFastList<>(),
false);
}
@Benchmark
public MutableList<Integer> mapECParallelLazy()
{
return this.ecList
.asParallel(this.executorService, 50_000)
.collect(i -> i * 2)
.toList();
}
@Benchmark
public IntList mapECPrimitiveEager()
{
return this.ecPrimitiveList
.collectInt(i -> i * 2, IntLists.mutable.empty());
}
@Benchmark
public IntList mapECPrimitiveLazy()
{
return this.ecPrimitiveList
.asLazy()
.collectInt(i -> i * 2)
.toList();
}
@Benchmark
public List<Integer> mapJDKBoxedParallelStream()
{
return this.jdkList
.parallelStream()
.mapToInt(i -> i * 2)
.boxed()
.collect(Collectors.toList());
}
@Benchmark
public List<Integer> mapJDKBoxedStream()
{
return this.jdkList
.stream()
.mapToInt(i -> i * 2)
.boxed()
.collect(Collectors.toList());
}

Sum all integers

Sum 1,000,000 Integers

Expected:

  • I expected primitive collections to outperform boxed collections.
  • I expected little benefit from parallelization here. Summing ints is a very fast operation. I expected eager primitive to be faster than most of the parallel options.

Unexpected:

  • I did not expect serial streams to get crushed. There seems to have been an improvement made in Java 9. I ran the benchmarks again with Java 9 and this particular benchmark improved by ~7–8x.
@Benchmark
public long sumECBoxedEager()
{
return this.ecList.sumOfInt(Integer::intValue);
}
@Benchmark
public long sumECBoxedLazy()
{
return this.ecList
.asLazy()
.sumOfInt(Integer::intValue);
}
@Benchmark
public long sumECParallelEager()
{
return ParallelIterate.sumByInt(
this.ecList,
i -> Integer.valueOf(0),
Integer::intValue).get(0);
}
@Benchmark
public long sumECParallelLazy()
{
return this.ecList
.asParallel(this.executorService, 50_000)
.sumOfInt(Integer::intValue);
}
@Benchmark
public long sumECPrimitiveEager()
{
return this.ecPrimitiveList.sum();
}
@Benchmark
public long sumECPrimitiveLazy()
{
return this.ecPrimitiveList
.asLazy()
.sum();
}
@Benchmark
public long sumJDKBoxedParallelStream()
{
return this.jdkList
.parallelStream()
.mapToLong(Integer::longValue)
.sum();
}
@Benchmark
public long sumJDKBoxedStream()
{
return this.jdkList
.stream()
.mapToLong(Integer::longValue)
.sum();
}

Filter, Map, Sum

Filter even integers, multiply remaining x 2 and return their sum

Expected:

  • I expected lazy operations to outperform eager.
  • I expected primitive lazy would outperform all of the other serial operations.
  • I expected JDKBoxedParallelStream would perform well with this use case.

Unexpected:

  • I did not expect ECParallelEager to do as well as or better than ECParallelLazy, even though it was optimized.
  • I did not expect JDKBoxedParallelStream to do better than ECParallelLazy.
@Benchmark
public long filterMapSumECBoxedEager()
{
return this.ecList
.select(i -> i % 2 == 0)
.sumOfInt(i -> i * 2);
}
@Benchmark
public long filterMapSumECBoxedLazy()
{
return this.ecList
.asLazy()
.select(i -> i % 2 == 0)
.sumOfInt(i -> i * 2);
}
@Benchmark
public long filterMapSumECOptimizedParallelEager()
{
return ParallelIterate.sumByInt(
this.ecList,
i -> i % 2,
i -> i * 2).get(0);
}
@Benchmark
public long filterMapSumECOptimizedParallelLazy()
{
return this.ecList
.asParallel(this.executorService, 50_000)
.sumOfInt(i -> i % 2 == 0 ? i * 2 : 0);
}
@Benchmark
public long filterMapSumECParallelLazy()
{
return this.ecList
.asParallel(this.executorService, 50_000)
.select(i -> i % 2 == 0)
.sumOfInt(i -> i * 2);
}
@Benchmark
public long filterMapSumECPrimitiveEager()
{
return this.ecPrimitiveList
.select(i -> i % 2 == 0)
.collectInt(i -> i * 2, IntLists.mutable.empty())
.sum();
}
@Benchmark
public long filterMapSumECPrimitiveLazy()
{
return this.ecPrimitiveList
.asLazy()
.select(i -> i % 2 == 0)
.collectInt(i -> i * 2)
.sum();
}
@Benchmark
public long filterMapSumJDKBoxedParallelStream()
{
return this.jdkList
.parallelStream()
.filter(i -> i % 2 == 0)
.mapToLong(i -> (long) (i * 2))
.sum();
}
@Benchmark
public long filterMapSumJDKBoxedStream()
{
return this.jdkList
.stream()
.filter(i -> i % 2 == 0)
.mapToLong(i -> (long) (i * 2))
.sum();
}

Congratulations!

I hope you enjoyed the blog and learned some new things about Iteration Pattern Options and Optimization Strategies using Eclipse Collections and Java Streams. If your only tool is a hammer, everything else is a nail. Knowing your available options before you get started on your journey and adapting as needs arise is one of the keys to writing better and more responsive applications. This can also help you execute a less stressful trip from Orlando to Williamsburg, if ever that occasion happens to arise.

Recommendations

  • Prefer Primitives over Boxing.
  • Prefer Eager iteration for single or fused operations.
  • Prefer Lazy iteration for multi-step operations.
  • Prove it before going Parallel.
  • Try Eclipse Collections if you want more than Hobson’s Choice.

Eclipse Collections is open for contributions. If you like the library, you can let us know by starring it on GitHub.


by Donald Raab at January 03, 2018 05:16 AM

Analyzing Eclipse plug-in projects with Sonarqube

by Lorenzo Bettini at January 02, 2018 10:16 AM

In this tutorial I’m going to show how to analyze multiple Eclipse plug-in projects with Sonarqube. In particular, I’m going to focus on peculiarities that have to be taken care of due to the standard way Sonarqube analyzes sources and to the structure of typical Eclipse plug-in projects (concerning tests and code coverage).

The code of this example is available on Github: https://github.com/LorenzoBettini/tycho-sonarqube-example

This can be seen as a follow-up of my previous post on “Jacoco code coverage and report of multiple Eclipse plug-in projects. I’ll basically reuse almost the same structure of that example and a few things. The part of Jacoco report is not related to Sonarqube but I’ll leave it there.

The structure of the projects is as follows:

Each project’s code is tested in a specific .tests project. The code consists of simple Java classes doing nothing interesting, and tests just call that code.

The project example.tests.parent contains all the common configurations for test projects (and test report, please refer to my previous post on “Jacoco code coverage and report of multiple Eclipse plug-in projects for the details of this report project, which is not strictly required for Sonarqube).

This is its pom

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>example</groupId>
    <artifactId>example.parent</artifactId>
    <version>0.0.1-SNAPSHOT</version>
  </parent>

  <artifactId>example.tests.parent</artifactId>
  <packaging>pom</packaging>

  <properties>
    <!-- this will be overridden in jacoco profile with tycho.testArgLine, 
      which, in turn, will be set by Jacoco prepare-agent goal. This property can 
      then be used in argLine for custom tycho-surefire configuration, both when 
      Jacoco is used and when it is not used. -->
    <additionalTestArgLine></additionalTestArgLine>
    <jacoco-version>0.7.9</jacoco-version>
  </properties>

  <profiles>
    <profile>
      <id>jacoco</id>
      <activation>
        <activeByDefault>false</activeByDefault>
      </activation>
      <properties>
        <!-- This will be set by Jacoco prepare-agent goal -->
        <additionalTestArgLine>${tycho.testArgLine}</additionalTestArgLine>
      </properties>
      <build>
        <plugins>
          <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>${jacoco-version}</version>
            <configuration>
              <excludes>
                <exclude>**/plugin1/Main.class</exclude>
              </excludes>
            </configuration>
            <executions>
              <execution>
                <goals>
                  <goal>prepare-agent</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>

</project>

Note that this also shows a possible way of dealing with custom argLine for tycho-surefire configuration: tycho.testArgLine will be automatically set the jacoco:prepare-agent goal, with the path of jacoco agent (needed for code coverage); the property tycho.testArgLine is automatically used by tycho-surefire. But if you have a custom configuration of tycho-surefire with additional arguments you want to pass in argLine, you must be careful not to overwrite the value set by jacoco. If you simply refer to tycho.testArgLine in the custom tycho-surefire configuration’s argLine, it will work when the jacoco profile is active but it will fail when it is not active since that property will not exist. Don’t try to define it as an empty property by default, since when tycho-surefire runs it will use that empty value, ignoring the one set by jacoco:prepare-agent (argLine’s properties are resolved before jacoco:prepare-agent is executed). Instead, use another level of indirection: refer to a new property, e.g., additionalTestArgLine, which by default is empty. In the jacoco profile, set additionalTestArgLine referring to tycho.testArgLine (in that profile, that property is surely set by jacoco:prepare-agent). Then, in the custom argLine, refer to additionalTestArgLine. An example is shown in the project example.plugin2.tests pom:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>example</groupId>
    <artifactId>example.tests.parent</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <relativePath>../example.tests.parent</relativePath>
  </parent>

  <artifactId>example.plugin2.tests</artifactId>
  <packaging>eclipse-test-plugin</packaging>

  <build>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>tycho-surefire-plugin</artifactId>
        <version>${tycho-version}</version>
        <configuration>
          <!-- additionalTestArgLine is set in the parent project with the 
            Jacoco agent when the jacoco profile is active (and it will be empty when 
            jacoco profile is not active). This way, you can customize argLine without 
            overwriting the jacoco agent set by jacoco:prepare-agent. -->
          <argLine>${additionalTestArgLine} -DbuildingWithTycho=true</argLine>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

You can check that code coverage works as expected by running (it’s important to verify that jacoco has been configured correctly in your projects before running Sonarqube analysis: if it’s not working in Sonarqube then it’s something wrong in the configuration for Sonarqube, not in the jacoco configuration, as we’ll see in a minute):

mvn clean verify -Pjacoco

Mare sure that example.tests.report/target/site/jacoco-aggregate/index.html reports some code coverage (in this example, example.plugin1 has some code uncovered by intention).

Now I assume you already have Sonarqube installed.

Let’s run a first Sonarqube analysis with

mvn clean verify -Pjacoco sonar:sonar

This is the result:

So Unit Tests are correctly collected! What about Code Coverage? Something is shown, but if you click on that you see some bad surprises:

Code coverage only on tests (which is irrelevant) and no coverage for our SUT (Software Under Test) classes!

That’s because jacoco .exec files are by default generated in the target folder of the tests project, now when Sonarqube analyzes the projects:

  • it finds the jacoco.exec file when it analyzes a tests project but can only see the sources of the tests project (not the ones of the SUT)
  • when it analyzes a SUT project it cannot find any jacoco.exec file.

We could fix this by configuring the maven jacoco plugin to generate jacoco.exec in the SUT project, but then the aggregate report configuration should be updated accordingly (while it works out of the box with the defaults). Another way of fixing the problem is to use the Sonarqube maven property sonar.jacoco.reportPaths and “trick” Sonarqube like that (we do that in the parent pom properties section):

<!-- Always refer to the corresponding tests project (if it exists) otherwise 
      Sonarqube won't be able to collect code coverage. For example, when analyzing 
      project foo it wouldn't find code coverage information if it doesn't use 
      foo.tests jacoco.exec. -->
    <sonar.jacoco.reportPaths>
      ../${project.artifactId}.tests/target/jacoco.exec
    </sonar.jacoco.reportPaths>

This way, when it analyzes example.plugin1 it will use the jacoco.exec found in example.plugin1.tests project (if you follow the convention foo and foo.tests this works out of the box, otherwise, you have to list all the jacoco.exec paths in all the projects in that property, separated by comma).

Let’s run the analysis again:

OK, now code coverage is collected on the SUT classes as we wanted. Of course, now test classes appear as uncovered (remember, when it analyzes example.plugin1.tests it now searchs for jacoco.exec in example.plugin1.tests.tests, which does not even exist).

This leads us to another problem: test classes should be excluded from Sonarqube analysis. This works out of the box in standard Maven projects because source folders of SUT and source folders of test classes are separate and in the same project (that’s also why code coverage for pure Maven projects works out of the box in Sonarqube); this is not the case for Eclipse projects, where SUT and tests are in separate projects.

In fact, issues are reported also on test classes:

We can fix both problems by putting in the tests.parent pom properties these two Sonarqube properties (note the link to the Eclipse bug about this behavior)

<!-- Workaround for https://bugs.eclipse.org/bugs/show_bug.cgi?id=397015 -->
    <sonar.sources></sonar.sources>
    <sonar.tests>src</sonar.tests>

This will be inherited by our tests projects and for those projects, Sonarqube will not analyze test classes.

Run the analysis again and see the results: code coverage only on SUT and issues only on SUT (remember that in this example MyClass1 is not uncovered completely by intention):

You might be tempted to use the property sonar.skip set to true for test projects, but you will use JUnit test reports collection.

The final bit of customization is to exclude the Main.java class from code coverage. We have already configured the jacoco maven plugin to do so, but this won’t be picked up by Sonarqube (that configuration only tells jacoco to skip that class when it generates the HTML report).

We have to repeat that exclusion with a Sonarqube maven property, in the parent pom:

<!-- Example of skipping code coverage (comma separated Java files). -->
    <sonar.coverage.exclusions>
      **/plugin1/Main.java
    </sonar.coverage.exclusions>

Note that in the jacoco maven configuration we had excluded a .class file, while here we exclude Java files.

Run the analysis again and Main is not considered in code coverage:

Now you can have fun in fixing Sonarqube issues, which is out of the scope of this tutorial 🙂

This example is also analyzed from Travis using Sonarcloud (https://sonarcloud.io/dashboard?id=example%3Aexample.parent).

Hope you enjoyed this tutorial and Happy new year! 🙂

 

Be Sociable, Share!

by Lorenzo Bettini at January 02, 2018 10:16 AM

In Review — 2017

by Nikhil Nanivadekar at December 31, 2017 03:30 PM

Sunset on Miyajima (Dusk of 2017)

As the sun is about to set for 2017, I but can’t help reflect what an amazing year 2017 has been! For me personally 2017 was a year of many “firsts”. I have summarized few of the many awesome experiences I had in 2017.

Experiencing the Land of the Rising Sun: In February I visited Japan and was awestruck by the beauty, heritage, friendliness, hospitality, cleanliness and most importantly the delicious food. I do not speak Japanese, and at first I was a bit apprehensive of how the visit will turn out. Few of my Japanese friends taught me some important phrases and I ventured ahead. I am happy to say, it was an experience of the lifetime. The people in Japan were very friendly and accommodating. I tried to converse with them in a mixture of English, Hindi, Marathi, sign-language and most importantly tried hard to learn new bits and pieces of Japanese. One of the humbling experiences was in Hiroshima when I was trying to get to Miyajima. I got on the wrong bus and ended up at a completely different train station. Patiently the bus driver explained the route to me with just numbers and sign-language and I reached the ferry to Miyajima without any further issues. This was my first visit to Japan but definitely not the last one!

Devoxx US: In March I was one of the speakers at Devoxx US. The conference was filled with rich content, amazing speakers and gave numerous opportunities to network, share ideas and concepts. I co-presented Collections.compare and Robots for the Kid in Us talks. I was also one of the conductors for Devoxx4Kids workshop which took place before Devoxx. This was my first Devoxx experience and I enjoyed it a lot. I hope to have the opportunity to present at many more Devoxx-es in the future.

Eclipse Collections Project Lead: In April, I was handed over the reins of Eclipse Collections and made the active project lead. I have been a contributor to the framework for more than 2 years and it was an amazing feeling to be recognized as a project lead. I worked hard to continue the legacy of successful project leads like Hiroshi, Craig and Don. This year we designed the official Eclipse Collections project logo, released 9.0.0 major version which works fine with JDK 9, presented Eclipse Collections via numerous avenues. I have highlighted my experience leading Eclipse Collections in this blog.

GIDS 2017: In April, I traveled to India, my homeland and presented at the Great Indian Developer Summit (GIDS 2017). This was my first time at GIDS and I will highly recommend developers to participate in GIDS as a speaker or as an attendee. The conference organizers do a phenomenal job putting the event together. They take good care of not just the attendees but also the speakers. The 4-day event was sold out with standing room only for pretty much all presentations. Looking forward to continued participation at GIDS.

JavaOne: In September, I was happy to be a part of the speaker lineup for JavaOne and JavaOne4Kids. This was my third JavaOne (second as a speaker) and like every year it was an amazing week. I was delighted to reconnect with many friends, Java Gurus, developer advocates, awesome people from the community. I learnt a lot of new things with the main theme being JDK 9, advancing Eclipse Collections and open source participation. I also planned my UK, Ireland JUG tour during the tail end of JavaOne. JavaOne was definitely one the highlights of my year.

UK, Ireland JUG tour: As the year drew to an end, I visited UK, Ireland in November. I presented at various Java User Groups during my visit. In summer, I had visited Manchester and met one of my good friends Nick. Nick is also one of the organizers of Manchester Java Community (MJC) and floated the idea of me returning to Manchester to present at an MJC event. This was really the beginning of the tour planning. It was an amazing experience to travel to London, Belfast, Dublin, Edinburgh, Manchester, Birmingham and meet with the Java community and have a ton of amazing discussions. I have written my tour experience on this blog. I would definitely want to do a similar tour again.

Wrap it up in India: I decided what other better way to end an awesome year than to visit family and take a break for a couple of weeks. 2017 was definitely a year of many firsts, I hope to continue the streak in 2018 and wish that it is much more positively eventful with many firsts and seconds.

I would like to thank each and every one who was a part of my 2017 and hoping to create even more awesome memories in 2018.

Wishing everyone a Happy, Healthy and Safe New Year!

Sunrise on Crete (Dawn of 2018)

by Nikhil Nanivadekar at December 31, 2017 03:30 PM

Java annotation processing in Eclipse

December 30, 2017 05:00 AM

Java annotations provide metadata for Java code. Many developers use annotations provided by standard Java APIs (like @Override) or by frameworks (like @Test). Developers can define their own annotations and process them at runtime using reflection. Additionally, Java provides APIs for writing custom annotation processors that can process these annotations at compile time.

Eclipse provides support for hooking these processors into the compilation process. So when you edit code with these annotations, the processor can analyse the source files, do stuff, and report information back to you.

To give you an idea about how that works, I’ll use Contracts for Java as an example, a tool that implements a contract model similar to that of Eiffel in Java, based on annotations. For our setup, we’ll need Eclipse 4.7 (Oxygen, the 2017 release) or later.

To use Contracts for Java, create a Java project, and add the latest release to the build path:

Java project

Then add some code that uses contract annotations (copy and paste the code below into the src folder of your project):

import com.google.java.contract.Ensures;
import com.google.java.contract.Requires;
 
public class Contracts {
    public static void main(String[] args) {
        System.out.println(new Numbers().add(-10, 5));
    }
}
 
class Numbers {
    @Requires({ "c > 0", "b > 0" })
    @Ensures({ "result > a", "result > b" })
    int add(int a, int b) {
        return a - b;
    }
}

At this point, the project should not report any issues. Normal compilation of the Java source file works just fine. Our processor is not enabled for our annotations yet. To configure the annotation processing we go to Project > Properties > Java Compiler > Annotation Processing.

Select Enable project specific settings and add the following options that we want to pass to the annotation processor: set com.google.java.contract.classpath to %classpath% and com.google.java.contract.classoutput to %PROJECT.DIR%/.apt_generated (the location of your project’s generated source directory):

Project &gt; Properties &gt; Java Compiler &gt; Annotation Processing

These properties are used by the processor to compile the annotated files, and to create output files. The %classpath% placeholder is replaced with your Java project’s build path by Eclipse, so that the annotation processor can access all the libraries used in your project when compiling your code. The %PROJECT.DIR% placeholder is replaced with the path to your Java project.

Finally we add the processor Jar to Project > Properties > Java Compiler > Annotation Processing > Factory Path:

Project &gt; Properties &gt; Java Compiler &gt; Annotation Processing &gt; Factory Path

After confirming these changes, and running a full build, compilation issues in the annotations are now reported in the editor. Our code contains a precondition mentioning a variable called c. But the annotated method has no c parameter. This is reported as an error in the editor:

Java editor

After we fix the first precondition to a > 0, the code compiles, and we can run it (Run > Run as > Java application).

To see the files generated by the processor in the .apt_generated directory in Eclipse, you should disable filtering of .*resources in the Package Explorer’s view menu (the little triangle > Filters…):

View menu

Also make sure you have set up your workspace to refresh automatically in Preferences > General > Workspace > Refresh using native hooks or polling:

Preferences &gt; General &gt; Workspace

Besides the annotation processing at compile time, Contracts for Java also uses bytecode instrumentation for its runtime checks. To have the contracts checked at runtime, add -javaagent:cofoja.contracts.asm-1.3-20160207.jar to the VM arguments of your run configuration (go to Run > Run Configurations, activate the Arguments tab):

Run configuration

Now, when running, we are made aware of the violated precondition in our code, since we are passing -10 as a, which is not larger than 0:

Precondition error

After we fix the call violating the precondition to new Numbers().add(10, 5); we now see that our implementation of add does not fulfill the postcondition, since the result is not larger than a:

Postcondition error

After fixing our implementation to return a + b, not a - b, all contracts are fulfilled and the code now runs without errors.

Contracts for Java uses annotations in an interesting way and shows what can be achieved with annotation processing as a tool, and how Java annotation processors can integrate with Eclipse.

If you’re interested in creating your own processors, check out my test projects for bug 3412981 to get a basic setup where an annotation processor compiles the annotated Java source file.

For more information on these tools check out the Contracts for Java and JDT-APT project sites.


1 That bug came up in the comments of a previous post about using Contracts for Java in Eclipse.


December 30, 2017 05:00 AM

Remote Services without OSGi bundles

by Scott Lewis (noreply@blogger.com) at December 28, 2017 01:25 AM

Remote Services provides a dynamic, transport-independent, simple, modular way to expose micro services.   ECF has created a spec-compliant implementation along with a large and growing number of open and extensible distribution providers.   

Remote services are frequently useful for fog/edge use cases, where the communication transports (e.g. MQTT) may be different than those typically used in the cloud (e.g. http/https, jaxrs, enterprise messaging, etc).   

Typically, remote services are run on OSGi-based frameworks and apps such as Equinox, Felix, Karaf, Eclipse, and others, and indeed ECF's RSA implementation works very well in any of these environments.   

Perhaps less well known, however, is that remote services can be used in other inter-process environments...for example between Java and Python.  

It's also possible to use ECF remote services without an OSGi framework, i.e. running only as Java applications.   This repository has an example of using ECF remote services without an OSGi framework.   The projects are Java applications (no OSGi framework assumed), on both the remote service provider side, as well as the remote service consumer side.   The examples may be run within Eclipse by using the launch configs in the example projects.

Most of the benefits of OSGi Remote Services are still available...for example the ability to use multiple distribution providers for a service, the ability to remotely discover services and dynamically respond to network failure, and the ability to use the OSGi service registry for service dynamics, and service injection.   Also, the service definition, implementation, registration and lookup are exactly the same whether via an OSGi bundle or a Java application.   This allows services to be defined consistently across runtime environments in addition to cross-distribution mechanisms.

Please clone the ServiceRegistry github repository and give things a try!






by Scott Lewis (noreply@blogger.com) at December 28, 2017 01:25 AM

De-obfuscating Eclipse workers

by Andrey Loskutov (noreply@blogger.com) at December 23, 2017 03:53 PM

Imagine you are debugging Eclipse IDE, which does something long and boring, running multiple worker threads, and of course all of them have very clear and consistent names:  Worker-1, Worker-2, Worker-3, ..., Worker-101, Worker-102, ..., you got it, I hope.

What do they do, you may ask? They all work, what else:

Workers do work, what else we can see here?
Now we have Christmas time, and what happens at this time? Santa is coming to all good plug-in developers!

If you use latest 4.8 Eclipse SDK build (from the 4.8 builds download page), and enable "Preferences -> Java -> Debug -> Listen to thread name changes", you can finally see what all those busy workers above really do:
Workers de-obfuscated
Isn't this cool?

P.S.
For details see bug 223492 and bug 528808.

Right now the preference is not enabled by default, but we hope to enable it in 4.8 M5 after some more testing if there will be no performance related side effects for bigger workspaces.

Enjoy!


by Andrey Loskutov (noreply@blogger.com) at December 23, 2017 03:53 PM