8 common misconceptions on units and references systems (for software engineers)

There are plenty of things in math and physics (like special relativity or vector calculus) that seem hard but are in fact simple once you understand them. Then there are a other things, instead, that seem really simple, but they are actually hard.

I find that some software engineers and computer scientists have a superficial understanding of units and reference systems, and hold to some common but flawed ideas.

Units are symbols

In the simplest case, the unit is a symbol we stick after a number: 5 m, 3 s, 1 pound... But clearly that does not apply to everything, as in m/s for velocity or m/s² for acceleration. Units are expressions.

So, if you store a unit in a string, that’s fine. If you compare that string to compare the unit, that’s not fine: “m^2” is the same as “m*m”, but they do not have the same string representation.

Different quantities have different physical dimensions

Two units that refer to the same physical quantity must have the same physical dimension: distance is [L], velocity is [L][T^-1], and everybody learns to check their result using dimensional analysis. But the revers is not true: two units can have the same physical dimension and not refer to the same quantity. Energy density (energy per volume) and pressure (force per area) have the same physical dimension [M][L-1][T-2] but they may be not compatible. Summing the energy density of a battery with the pressure on the surface of such battery has very little physical meaning.

So, if you are writing classes for units to check that the number passed around your code have the right meaning, know that comparing the physical dimensions is necessary but not sufficient to make sure the input is valid. They actually have to refer to the same thing. For the same reason, you cannot simply convert everything that have the same physical dimension to the same unit: J/m^3 is not the same as Pa.

angles have a physical dimension of angles

Some people, carrying the previous misconception, feel that they must create another physical dimensions for angles, otherwise you would be able to sum angles to pure number. And that does not make sense! But as we have seen before, dimensional compatibility is necessary but not sufficient. So you can have quantities that have the same physical dimension but are not homogeneous.

An angle is the ratio between the circular arc within the two sides and the radius of such arch: [L] over [L] so it is a pure number. Angles are measured in units, such as radians and degrees. Pure number can have units. Among other units that dimensionless are the steradian (for solid angles) and the decibel (for ratios).

ANGLE times a LENGTH equal LENGTH

If you multiply an angle by a radius, you’ll get the length of the arc (according to the previous definition). So, you may conclude that angle times length equal length. But that’s true only if the length is the actual radius for that angle. For example: angle * radius * height gives me the area formed by the arc as the base and the height. Suppose I group angle * height: that’s not a length, because I am not getting the arc formed by the height and the angle. It’s angle times length. So, the result of angle times length depends on what you are doing.

Again, when you combine to quantities it matters what they are and how you are combining them. In general, you need to understand what’s going on. Force times length equal work only if that force and that length have a particular relationship between them. If you are trying to write a system that converts units “automatically”, you either need to make sure that it has enough information to understand the difference.

Knowing value and unit is enough to convert to a compatible unit

Again, this works in simple cases. What is 5 meters in millimeters? 5000. But if I asked you, what is 5 Kelvin in Celsius? That depends: is 5 K representing an absolute temperature or is it a difference between two temperatures? In the first case you will have -268.15 C while in the second you will have 5 C. 

Temperature is the only example I know, because each unit really defines a scale with a reference point.

The component of a vectors are all of the same unit (or quantity)

If you use a Cartesian coordinate system, you are most likely to express all the components in the same unit (e.g. [1,2,3] m). You could also express them in different units (e.g. [1m, 2cm, 3Km]), but conceptually you could choose to make them uniform without loss of generality. But you may choose a spherical coordinate system, in which case you cannot express all the components with the same unit because they represent different quantities. A length in a spherical coordinate system is [length, angle, angle].

Different coordinate are in general expressed by different units and/or quantities. The type of quantity is given by the choice of actual coordinates that are used to identify the points in the space. If you define a vector class that has an array and only one unit, know that you’ll be able to only work in special cases.

A point is a vector

When someone learn the Cartesian reference system, one of the first thing that he is taught is to think as the set of coordinates used to specify the location of a point as a vector. And you can imaging this by drawing an arrow that starts from the origin and ends on the point. This actually is not strictly true, and it causes a lot of confusion. If you had the good fortune to learn generalized coordinate system, your confusion has been cleared. If you didn’t, you are confused and you may not realize it.

Imagine you have an actual vector, like a momentum, a force or an electric field. Imagine that you now apply a translation to your coordinate system: how does that affect the vectors? It doesn’t: they still point in the same direction, have the same magnitude, and have the same components on the axis. They just happen to be located at a different point. But what happens to the coordinate of a point? It changes by the amount of the translation.

For a deep understanding, you’ll need to learn generalized coordinates system (which is one of those things that is simple once you understand it). For the time being: do not use the same class for Point and Vector!

Given the components of a vector, you know the vector

A vector is identified by its magnitude and direction, so if I gave you its components in one reference system you’d expect you now know everything about that vector. This is true if you are in a Cartesian reference, but only there. If you use spherical coordinates, the first component of the vector is the component along the radial direction. That changes from point to point. You need to know at which point the coordinates are defined, because the 3 vectors that make up the basis can, in general, be different from point to point. In a Cartesian reference, they are the same.

A vector is magnitude and a direction defined at a point (it’s a “local” object). If you change coordinates, all vectors transform the same, but their transformation depends on the point where they are defined. Operations among vectors (like scalar of vector product) should also be done between vectors at the same point. If you have a Vector class, and you do no keep track the Point where the vector is defined, you’ll be able to only work in special cases.

Conclusion

As you can see, there aren’t very simple “mechanic” rules, that can be applied without understanding the semantic of what you are doing. Whatever tool you are going to use (or write) to help to keep track of units, it can’t be a substitute for your (or your users’) understanding.

Lego museum

I finally sat down and tried to build something from scratch. Here is the result:

The two different colors for the floor is because I didn't have enough pieces. Many choices where forced because of the lack of pieces! It’s a modular building, you can see on the side where you would hook up other buildings.

Ground floor (or first floor for Americans)

On the right the entrance and a few paintings, in the middle a double staircase to the upper floor and on the left fossils (kids and teacher from a school trip).

First floor (or second floor for Americans)

Upstairs ancient weapons and armors, and some animals.

Moving to blogger

Since I’d like to start blogging about other interests of mine (like making music, physics, ice-cream/gelato making, Lego, …), I will be moving my main blog from Java.net to Blogger.

I will try a combination of Live Writer (for offline editing), Blogger (for publishing) and SyntaxHighlighter (to make code more readable). This should also make it a little bit easier to blog, which hopefully means I’ll do it more! We’ll see how it goes.

My new address will be http://gabrielecarcassi.blogspot.com. I will be cross-posting on Java.net for the posts that are relevant to Java.

Wish list for java array (or numeric collections)

If you have been doing numeric calculations in Java for some time, you'll have learned to both love and hate numeric arrays.

For the love part:

• they are safe. No garbage data. All access is guarded.
• they are fast. A simple loop to calculate the average of an array of 100,000 elements on JDK 1.7, Intel Core i7 Q840 1.87GHz, takes about 100,000 nanoseconds, or one nanosecond per element.

For the hate part:

• there are 6 number arrays (byte, short, int, long, float double). If you need to write a function that works with all, you either have to convert (expensive) or provide 6 implementations (can't use primitive with generics, and generics and arrays don't mix well anyway)
• you can't have immutable references. If you return an array from an object, you have the option of being efficient but unsafe (you return the array directly, thus exposing your internal state) or being safe but inefficient (you return a copy, even if 90% of the uses are reads)
• you can't give "views" of other arrays. To pass a sub-array you need to actually copy it. NIO buffers, for example, have to implement that functionality, like so many other have to in other contexts, including me.
• their are concrete classes. Or better, we don't have a superclass for primitive/number collections.

This props up various attempts, all incompatible, to patch these deficiencies. If you search in ohloh for DoubleList
http://code.ohloh.net/search?s=DoubleList&browser=Default&pp=0&fl=Java&m...
you'll find some of them.

Naturally, I am in that group too. I have my "solution", which can't really be a solution because it's not standard. So, if I have to use some java math library to do something, I still have to convert in and out. Yet, I'll go through some of the details, because some things do work, and it's the kind of things I'd like to see.

What we need are numeric collections. The current collections don't mesh well, for the usual reasons: they are collection of objects, not primitives; generics does not work with primitives; generics is invariant.

The basic thing one need is an iterator for external iteration (yes, internal iteration is nice, but it becomes difficult when you need to iterate over more than one collection at a time, possibly in different patterns). Following both the java.lang.Number and java.util.Iterator conventions, I have:

public interface IteratorNumber {
    
    boolean hasNext();
    float nextFloat();
    double nextDouble();
    byte nextByte();
    short nextShort();
    int nextInt();
    long nextLong();
}

In the same way that Iterator allows iteration of a collection, and Number allows to use a numeric value "without caring" with a single binding, this interface allows to iterate a numeric collection with a single binding. One can also image implementation of these that do a safe cast (i.e. throw an exception if the convertion overflows), or wrappers that do that. Again, similarly to java.util.Number and its sub-classes, I have:

public abstract class IteratorByte implements IteratorNumber {

    @Override
    public float nextFloat() {
        return (float) nextByte();
    }

    // Implement all except nextByte()

}

This accomplishes two purposes: when implementing I only have to implement one method, and allows the client to detect what type of numer it is (a byte instead of a double). So that, in case one _does_ want to have different bindings for the different types, one can still do it!

Some kind of mechanism like this is the basic minimum. One could think of using Iterator and Iterator, but they have the following problems:

• next().doubleValue() means there was an object created in between (the Number or Double). If the code can be inlined and escape analysis kicks in, that's not an issue. But, given that this is a basic interface, and you may have several implementations, this would not (currently) happen.
  
• given that generics are invariant, you'd have to always use Iterator instead of Iterator. And working with wildcards all over the places is not pretty.

On the other side, IteratorDouble as implemented before, could easily implement Iterator too. The Double next() would only have one implementation (in the IteratorDouble superclass), which would refer to the nextDouble() call, so it could be inlined more easily, escape analysis may kick in and so on.

Regardless of the details: I think there is a need for a numeric iterator! :-)

Apart from the iterator, we need some basic collections. With lack of imagination (which is often enemy of good design), I have a CollectionNumber, which only supports sequential access, a ListNumber, which support random/indexed access. These have abstract implementations in CollectionDouble and ListDouble, again so that most of the implementation is done and so that a consumer can tell which type is in the collection.

Lastly, there is a set of ArrayDouble, ArrayFloat, ... which are just wrappers around arrays. And here's the punchline: if you code to these _directly_, through the magic of inlining, you suffer _no_ performance loss. Zero. Which means: using these set of interfaces, I am losing _nothing_. I can still provide 6 different bindings to ArrayXxx, which would all perform in the same way like I'd do with plain arrays. But, if performance is not that criticial, I _can_ code to ListNumber or CollectionNumber, and cover all cases in one. In fact, if only a couple of concrete classes are instanced, you'll also suffer no performance loss.

Again, regardless of the details: we need something like this.

I want to be able to make the decision to code to the general case first. Once I see that performance is an issues, I can take, say, the ArrayDouble case and give a second implementation for that, simply by switching on the type of the class, and then rewriting the same method with a different parameter type (the rest stays the same). In fact, it'd be nice if the VM did that for me, but that's another story.

The other nice thing about the wrapper array class is that I can disable the write. It's "read-only", not as good as immutable, but in many cases it works the same.

You can also create a "safe copy wrapper", which if accessed in read only, does not copy, but at the first write, would make the copy of the array.

These few things cover at least the problems I mentioned before. The solution I showed I think fits already the patterns established by java.lang.Number and the Collection classes. So at least does not feel "foreign".

pvmanager: a framework to deal with live data (part 2)

I'll continue the overview of the design, focusing on the fluent API.

We left with some basic spec for the object that do the live processing of the data. A user will need to create a whole set for each pipeline, plus the set of instructions to give to the data sources (what data to read, where to put it and who to notify). We need to construct and expression language, which allows to:

• define each part of the pipeline
• allow to construct an expression by reusing, mixing and matching different parts
• allow different users to add their own pieces without requiring modification of the framework

Expression classes

At the end of the day, we'll want something like:

PVReader reader = PVManager.read(<read expression>) 

For example:

// Read the channel named "my channel" at maximum 10 Hz (i.e. will skip the extra)
PVReader<Object> reader = PVManager.read(channel("my channel")).maxRate(ofHertz(10));

// Queue and read all the new values for channel "my channel" at maximum 1 Hz
// (i.e. all notification will arrive, for batch processing)
PVReader<List<Object>> reader = PVManager
    .read(newValuesOf(channel("my channel")))
    .maxRate(ofHertz(1));

// Read the channel named "my channel" at maximum 1 Hz (i.e. will skip the extra)
PVReader<Map<String, List<Object>>> reader = PVManager.read(
    mapOf(newValuesOf(channels("channel1", "channel2", "channel3")))).maxRate(ofHertz(1));

In other word, we need to say "read this", and we need a way to say what this is. We may be tempted to have one Expression class. But if you look closely, some expressions (like channel) will get processed at the data source rate, some other (like mapOf) will get processed at the desired rate, and some (like newValuesOf) will convert a source rate expression to a desired rate expression (i.e. will implement one of those Collectors - queues, caches and the like).

So, we really need at least 2 kinds: a DesiredRateExpression and a SourceRateExpression. So we can define, for example:

public static SourceRateExpression<Object>
    channel(String name) {...}

public static <T> DesiredRateExpression<List<T>>
    newValuesOf(SourceRateExpression<T> exp) {...}

But if you look closely, again, some expressions (like channels - note the plural) actually give us a list of expressions. So we need extra 2: DesiredRateExpressionList and SourceRateExpressionList. So we can define:

public static SourceRateExpressionList<Object>
     channels(String... names) {...}

public static <T> DesiredRateExpression<Map<String, T>>
    mapOf(DesiredRateExpressionList<T> expressions) {...}

And so on. If we also want to support writable expressions, we will need a WriteExpression and a WriteExpressionList. If we want the same expressions to be read and write, we will need a DesiredRateReadWriteExpression and SourceRateReadWriteExpression. So, 5 single expression (source, desired, write + 2 combinations), and 5 lists (one for each).

This is where Java is pain, though: since we don't have multiple inheritance, these all need to be interfaces. But, since we still need to provide implementations, we need provide an extra 10 classes. Moreover, interfaces cannot have package private members, which means we may have to expose more things than we strictly need. So, the org.epics.pvmanager.expression has 10 interfaces plus 10 implementation classes. Kind of a pain. Fortunately, this is only a burden when you are implementing new expression, not when you are using the API.

When you are using it, you get an expression fully checked: if an operator needs a expression read at the source rate, that's the only thing it can get; if it requires a list, it can get both a list or a single expression (which is a list of one); if it needs a number, it gets a number, and so on. Any user can create his own operators, mix them with the others, and the power of the API is the combination of all the different operators.