Victor Laskin's Blog

Encapsulation of asynchronous behaviour in distributed system of scripted autotests

Victor Laskin — Fri, 06 May 2016 07:01:02 +0000

This is an example how to create continuation-like monadic behaviour in C++11 (encapsulate async execution) which is used for specific example: writing asynchronous scripts for system of distributed autotests.

I needed to add autotests into continuous integration of client-side multiplatform product. Basic way to build autotesting system is to grab one of well-known frameworks (especially if we are talking about web solutions) and test product locally. Local testing environment simulates clicks buttons inside UI and gets the results right away. This is simple and good way except the case when you have truly cross-platform solution (win, mac, web, iOS, android, etc), custom rich UI framework and you want to perform tests on various kinds of systems at same time.

So I came up with remote distributed scheme for autotests. And inside this post there are some ways to make this scheme shine.

This post contains 3 main points:

what is distributed system of autotests and why the one would need it (advantages / disadvantages)
how to implement asynchronous scenarios of distributed autotests using continuation monadic style in C++11/14 (how to encapsulate asynchronous execution and do this the most compact way)
how to integrate ChaiScript scripting into such system to make it more flexible

PART 1 – DISTRIBUTED SYSTEM OF AUTOTESTS

If you are interested only in ways of encapsulation of async interfaces in C++ you could skip this chapter and move on to PART 2.

Let’s build server which will control clients via sending events through separate web-socket connection. It is like performing tests of car when you use remote control. Next diagram shows main structure differences:

Let’s discuss advantages at first:

Cover a lot of devices

In such scheme we could launch clients at any device (as we don’t need to setup local testing environment any more). Also it’s possible to run tests on clients under debugger and see what happens.

Write once – run on device park

We could launch tests on nightly builds using all available range of devices. When new test is written it will work on all devices (except case when test uses some platform specific aspect like push notifications, etc). When new device is added into the zoo there is almost zero overhead on setup of testing environment.

Analyze results

We could export statistics from all tests into one server side storage and analyze all variety of test cases inside one analytical system. In my case it’s ElasticSearch + Kibana, but this could be any tools/storages you like to work with. Also the one could use autotests as profiling system: you could compare performance results measured during same tests (using different devices or different versions of target application).

Fast write, Fast run

Once again you need to write test only once to support all variety of platforms, operating systems, devices, etc. This is huge gain in time. Also when there is no need in custom setup of testing environment on device we could analyse more cases or spend more time writing new tests. Developers could run tests right on their builds without no need to upload them into some testing environment.

Finally – More stable CI

Performing tests on large zoo of devices, analysing not only green lights from tests but fine performance measurements, increased speed of test development and autotests integration – all this leads to more stable continuous integration.

Disadvantages

Are there any disadvantages? First one is that you have to spend some time implementing such system. There are no boxed solutions as far as I know. But actually this is not the main problem. On application side it’s enough to create simple interface for accepting external commands received through socket. It could be only UI control commands (like click at x,y coords or is this element visible requests), or it could be some complex interaction with inner application model. Any way this is simple request-answer interface, which could be extended with periodic performance reports, sending last chains of inner events or even passing crash log reports to server.

What really is the problem: how we should write tests when test target is separated from test itself – how to write asynchronious tests?

For example, instead of general way when you deal with local tests and need to push some interface button you just need to call some method like button1.click() and thats all. But when you are dealing with distributed scheme you need to send some message to client – “push button1” and wait for answer. And we need to take into account that client might not even answer or our ui button might be hidden already and message could fail to execute.

To solve this first idea is to implement test on server side as finite state machine (link). Let’s assume test has a state which could be represented as some simple enumeration (or item from some set). Each state has listeners for client’s answers. And by each async operation we jump from one state to another. For example, for “click button” case we have two states – first: we send message and switch to second state, second: if we receive answer that button was successfully clicked we move on to next action.

If we have simple test of linear continuous actions we could just keep state as int and increment it performing each ‘step’ from current state to next one.

STEP("start",
      send("tests.ui.click element", {{"id", bLogin}});
      state++;
     );
 
STEP("tests.ui.click element",
     if (answer->isOk())
     {
         state++;
         // .... next action here ....
     } else 
     { 
         // try something else
     }
     );

Everything could be implemented in such manner, but the problem is that if your tests are not so trivial the code will become unreadable mess of state jumps. Also there will be a problem to extract functionality into separate methods because state logic overlaps each other – finishing handler for last action inside some function block should move on to next state which is outside of scope. You might look for ways to organise this structure and solve such ‘overlap’ problems but there is a better solution to solve this at once.

PART 2 – C++14 CORE FOR WRITING ASYNCHRONOUS TESTS

IT’S POSSIBLE TO WRITE ASYNCHRONOUS CODE IN SYNCHRONIOUS MANNER

In ideal world I just want to keep writing the code of my test using old ‘synchronous‘ way as it is more simple, readable, etc. But I want it to work async way. To achieve this we need all power of C++11/14 because this will look like continuation monad in C++. What are continuations you could read here.

I want to write code using general sequential notation – but implicitly this code will produce a lot of functional handlers and totally encapsulate whole state machine inside. Yes, this is possible using modern C++!

Warm up example – simple login test:

// THIS IS ASYNCHRONIOUS!
    
client->getCurrentView()->assertValue("Start view:", "login");
auto tbEmail = client->getElement("login", "tbEmail")->assertOk();
client->setTextInTextbox(tbEmail, email);
auto tbPsw = client->getElement("login", "tbPassword")->assertOk();
client->setTextInTextbox(tbPsw, password);
auto bLogin = client->getElement("login", "bLogin");
bLogin->assertOk();
client->clickElement(bLogin);
    
client->getCurrentView()->makePeriodic(300)->successOnValue("main");

This seems like normal local test but in reality under the hood this code creates state machine with message handlers which will be executed asynchroniously. Not only this is much shorter than defining states and handles manually, but also this is general way of doing things which is familiar for any auto tester.

The problem is that not all things could be transferred totally unchanged. Cycles, conditions and so on can not be written in such clean way except we introduce some preprocessing or DSL to make changes into asynchronous form. But let’s go step by step.

IMPLEMENTATION DETAILS

First big help here is auto keyword. When we execute request method like getElement in previous example the result is actually not a value, but a future. And as we need as compact syntax as possible and a lot of additional customisations, here will fit custom implementation. (There are a lot of ways to implement this – so any specific ways are optional and you could tune them as you like)

Here is part of such asynchronous value which is not yet acquired:

template 
class ATRequestValue : public std::enable_shared_from_this> {
public:
    
    std::shared_ptr value;        // main value - is empty at start
    
    ATTestState state;               // state of test when request has to be made

    std::function request;                       // make request
    std::function response;     // parse response
    MVMessage answer;                                       // client's answer [optional]
    //....

Main fields here: value – pointer to value which will be filled when answer from client will be received by server. state – corresponding state of test. request – function which will send actual request to client. response – function with fill parse answer from server.

And so getElement function in previous code piece is not actually performing some actins but creates such ATRequestValue and fills it with corresponding request/response handlers as lamda functions:

auto getElement(const ValueProvider& prefix, const ValueProvider& name){
        return addRequest(std::make_shared(test,
            "tests.ui.get element",
            [=](ATIdPtr value){
                 test->send("tests.ui.get element", {{"prefix", prefix()}, {"name", name()}}, value->delay);
            },
            [=](MVMessage msg){
                 return std::make_shared(toID(msg->getString("id")));
            }
        ));
}

In this example we try to find some UI element by name and some prefix (which is also string), and result is some identifier of type id.

Function addRequest is something like this:

/// Add custom request (async stage) into chain
    template
    T addRequest(T request)
    {
        // add item to current scope
        scopes.currentScope->addItem(request->name, [=](){
            jumpTo(request); 
            return true;
        });
        
        // add to list of requests
        requests.push_back(make_at_variant(request));
        return request;
    }

Before we discuss scopes, jumpTo, etc – let’s first talk why arguments of getElement are not just strings, but ValueProvider. We could have kept them as strings if all parameters had been predefined before test. We can’t do that because in reality the most of inputs are computed depending on previous results. Thats why we need value providers – functors which will return (provide) some value of predefined type.

// Provide value through evaluation of inner function
template 
class ValueProvider {
private:
    std::function f;
public:
    ValueProvider(const std::function& f) : f(f) {}
    ValueProvider(T value) : f([value](){ return value; }) {}
    
    // construct from any functor
    template 
    ValueProvider(F f) : f([f](){ return f(); }) {}
    
    // get operator
    T operator()() const { return f(); }
};

Now we could pass lamda as parameter to getElement which will be executed later – for example:

auto bLogin = client->getElement("login", [](){ 
    // here could be complex logic
    return "bLog"s + "in"s; 
});

Now let’s go back to place where getElement is called. Details on how exactly we form request and read response are not so important – what matters is how to make a chain of such calls. This is the place where we need scopes, elements, requests and jumps between them.

Let’s look at first simple case when we have only one scope (simple test with no branches or cycles). In such case we just need basic jump from one request to another (and gather them inside one scope’s storage to keep them alive).

template 
void jumpTo(std::shared_ptr> value)
{
    test->state = value->state;
    test->state++; // just increment state - simple case
    value->request(value);
}

In simple case to perform jump we need just increment test’s state and perform request of next RequestValue. To finish the magic we just need to assemble() things together into one state machine.

// Final assemble
void assemble(){     
    ATTestState s = 1;
    for (auto request : requests)
         addHandlersForRequest(s, request);
}

/// Main constructor of stages
template 
void addHandlersForRequest(ATTestState& state, std::shared_ptr> value)
{
     value->state = state;
     
     // go to next stage   
     state++;
        
     // add answer
     test->actions[state].set(value->name, [value, this](MVMessage msg){
            value->answer = msg;
            value->value = value->response(msg);
           
            scopes.runNextItemInScope();
        });
        
}

I’m skipping here the variadic part – ATRequestValue could contain different types of values and so the list of requests should be able to hold variadic types – but may be this will be the topic of separate post.

In trivial case when we have only one scope running next item could be pretty simple:

bool runNextItemInScope(){
    scope->position++;
    scope->children[scope->position].run();
}

And this enough to implement simple sequential chains without any branching. But we need more.

SCOPES AND BRANCHING

First problem arises when we want to add first conditional branch into our test’s logic. When we write something like:

auto text = getElementText(editBox1);
if (text.get() == "5"){
    auto element = getElement("main", "button1");
    client->clickElement(element);
}

This will not work because condition will be executed synchronously and text will not yet be received. So this code will crash. The solution is to create async_if implementation which will accept predicate and evaluate it at right moment when the text will be received. Also we need scopes here. They could be implemented as simple lists of run functions, but with some additional enter / exit handlers.

/// Item inside scope
class ATScopeItem {
public:
    string name;                        /// for debugging
    std::function run;          /// run item - returns false if there is nothing to run
};

// type of scope
enum class ATScopeType { normal, cycle };

/// Async scope
class ATScope {
public:
    ATScopeType scopeType;
    ATScope() : position(-1), 
       scopeType(ATScopeType::normal), 
       enter([this](){ position = -1; return !isEmpty(); }), 
       exit([](){ return false; }) {}

    vector children;             ///< the list of scope items
    
    int position;                             ///< current position in scope
    
    std::function enter;              ///< on scope enter
    std::function exit;               ///< on scope exit
    
    
    void addItem(string name, std::function item){
        children.push_back({name, item});
    }
    
    bool isEmpty() { return (children.size() == 0); }
};

Optional stuff here – names for scope elements (for debugging). Also there are scope types – we will need them later.

Enter and exit functions return boolean values – when true is returned it means we had some asynchronious action called inside and should wait for result. If false is returned we could continue execution and move to next item in scope without waiting.

Also we need some structure which will contain all scopes and organise movement between them. We could use some tree or stack here.

// Stack of scopes
class ATScopeStack {
public:
    
    ATScopePtr currentScope;  ///< pointer to current scope

    // main push/pop operations 
    void pushScope(ATScopePtr scope) { scopes.push(scope); currentScope = scope; }
    void pop(){ scopes.pop(); currentScope = scopes.top(); }
    
    bool enterScope(ATScopePtr scope){
        scope->position = -1;
        if (currentScope != scope)
            pushScope(scope);
        if (scope->enter())
            return runNextItemInScope();
        return false;
    }
    
    bool isRootScope(){ return (scopes.size() == 1); }
    
    // call run for next element (recursive, could go through stack of scopes)
    bool runNextItemInScope(){
        
        auto scope = currentScope;
        
        while (true)
        {
            // if we have more items in current scope
            if (scope->position < (int)scope->children.size() - 1)
            {
                scope->position++;
                if (!scope->children[scope->position].run())
                {
                    continue;
                }
                else
                {
                    return true;
                }
            }
            else
            {
                if (scope->exit())
                    return true; // exit() called some block
                
                if (isRootScope())
                    return false; // do nothing
                
                pop();
                scope = currentScope;
            }
        }
        return false;
    }
    
    std::vector allScopes;     ///< all scopes as list
private:
    std::stack scopes;    ///< current state of scope stack
};

Function enterScope here also returns boolean so it could stop synchronous execution and wait for next result in asynchronous chain. Now our main function runNextItemInScope which goes to next item becomes slightly more complicated – it’s a cycle which will call elements in current scope one by one until any of them signal that we have to wait by returning true from it’s run(). When there are no more items in scope we pop upper scope from scope stack and continue execution there using same pattern. And finally when scope has no parent (stack has only 1 item) we just stop.

It’s great that in such scheme we could implement not only “if” branch but cycles too. But let’s start with async_if:

auto async_if(std::function condition, std::function thenBody, std::function elseBody){
        
        // then
        auto thenScope = make();
        scopes.pushScope(thenScope);
        scopes.allScopes.push_back(thenScope);
        thenBody();
        scopes.pop();
        
        // else
        auto elseScope = make();
        scopes.pushScope(elseScope);
        scopes.allScopes.push_back(elseScope);
        elseBody();
        scopes.pop();
        
        // create scope item
        scopes.currentScope->addItem("async_if", [=](){
            if (condition())
                return scopes.enterScope(thenScope);
            else
                return scopes.enterScope(elseScope);
        });
    }

So this function actually executes both then/else branches right away! But this execution only creates functional handlers which will be called later. We pass both branches and predication as parameters here. Thanks to C++11’s lambda syntax it’s possible to write such code:

auto text = getElementText(editBox1);
async_if([=]()( return (text.get() == "5"); )
    [=](){
        // more async logic
        auto element = getElement("main", "button1");
        client->clickElement(element);
    },[=](){
        // do something else async way
    }
);

I have to admit this is not so pretty as simple if and this is very sad. Of course this is much better than writing async handlers straightforward way, but this still is not perfect. Optional way is to make some preprocessing eliminating boilerplate functional wrapping, or even make some DSL for autotests. Using additional scripting overlay may ease the pain a bit (will be discussed at part 3).

WHILE / CONTINUE

Same way we could define while cycle.

auto async_while(std::function condition, std::function body){
        // body
        auto scope = make();
        scope->scopeType = ATScopeType::cycle;
        scopes.pushScope(scope);
        scopes.allScopes.push_back(scope);
        body();
        scopes.pop();
        
        thenScope->exit = [=](){
            if (condition())
                return scopes.enterScope(scope);
            return false;
        };
        
        // create scope item
        scopes.currentScope->addItem("async_while", [=](){
            if (condition())
                return this->scopes.enterScope(scope);
            return false;
        });
    }

Here we set custom exit procedure which checks condition and reenters scope if it was not satisfied. Usage example could be the following:

async_while([](){ return true; }, [](){
    client->clickElement(someButton).setDelay(2000);
});

We could also introduce continue operator. This is the place where we need to know the type of scope. Logic is simple – we go up through scope stack until we find cycle scope, and then we reenter that scope.

auto async_continue(){
        // create scope item
        scopes.currentScope->addItem("async_continue", [=](){
            
            // we go back to first cycle scope
            while (scopes.currentScope->scopeType != ATScopeType::cycle)
            {
                scopes.pop();
                
                if (scopes.isRootScope())
                    return false; // do nothing
            }
            
            return scopes.enterScope(scopes.currentScope);
        });
}

Now we could have any kind of nested structure like async_continue inside async_if which is inside async_while.

FOREACH

Async foreach is a bit tricky as you have to iterate over something which is not yet obtained. But whole trick is that you have to use data provider instead of data itself. Provider is a function which gives array – and you could access it by index or iterator. There are a lot of freedom here so I put here only basic start example for some vector and indexed access:

template 
auto async_foreach(ValueProvider> listProvider, std::function)> body){
        
        // we create new provider for element of list
        std::shared_ptr index = make_shared(0);
        ValueProvider elementProvider([=](){
            return listProvider()[*index];
        });
        
        // body
        auto scope = make();
        scope->scopeType = ATScopeType::cycle;
        scopes.pushScope(scope);
        scopes.allScopes.push_back(scope);
        body(elementProvider);
        scopes.pop();
        
        scope->enter = [=](){
            thenScope->position = -1;
            if (listProvider().size() == 0) return false;
            *index = 0;
            return !scope->isEmpty();
        };
        
        scope->exit = [=](){
            *index += 1;
            if ((*index >= listProvider().size()) || (scope->isEmpty()))
                return false;
            scope->position = -1;
            return scopes.runNextItemInScope();
        };
        
        // create scope item
        scopes.currentScope->addItem("async_foreach", [=](){
            return this->scopes.enterScope(scope);
        });
        
    }

Here we redefine both enter/exit handlers to create an iterator. Cycle body is called with parameter which is a value provider once again.

Usage example:

global tabs = getElementChildren(barWithTabs);
 async_foreach(
      provideVectorId([](){ return tabs.get(); }),
      [](auto x){
           auto tab = checkElement(x);
           touchElement(tab, 5.0, 5.0).setDelay(1000);
      });

This sample clicks all tabs inside some menu bar.

INSERT ANY SYNCHRONIOUS ACTION INTO CHAIN – ASYNC_DO

There are a lot of cases when between asynchronous requests we need to perform some synchronous actions (like printing something to log). Of course it cannot be written general way because in that case it will be executed at test’s assemble time when we only set handlers and yet have no data. As one way to solve this we add additional function – async_do.

auto async_do(std::function body){
        
        // create scope item
        scopes.currentScope->addItem("async_do", [=](){
            body();
            return false;
        });
         
}

So we just wrap functional body into scope element and insert it into scope. We could introduce here a lot more of helper functions such as async_set or async_copy. They could also assign the result to some variable or do something else.

Usage example of async_do could be found in part 3.

EXTENDING ASYNC VALUES

Second way to perform synchronous actions is to insert some functional checks right inside request values.

First one to add is special assert which check that async request went well:

bTotalPrice = getElement("", "bTotalPrice").assertOk("Something went wrong");

// or we could check some value
auto text = getElementText(bTotalPrice).assertNotEqual("$0", "Balance should not be equal to zero in this test");

// or we could even check some functional condition
text.assertCondition("Balance should be in USD", [](auto x){
    return (x.get().find("$") != string::npos); 
});

// or we could just do something after value was received
text.andDo([](){ log("Data was received"); });

To implement such functions we only need to create array of functional checks inside ATRequestValue and add such methods as:

/// Assert that value is equal to given value
Ptr assertValue(string message, T shouldBeValue){
        checks.emplace_back([this, shouldBeValue, message](){
            if (*value != shouldBeValue) 
                 test->fail(message + " expected: " + request_value_to_string(shouldBeValue) + " got: " + request_value_to_string(*value));
        });
        return this->shared_from_this();
}

I use fluent interface pattern here – this is optional of course.

You can find usage examples of such checks in part 3.

Also we could add function waitFor(interval, condition) which will make periodic requests until provided functional condition will be fulfilled.

I think you already got a lot of ideas how to expand this approach. So let’s move on to the last chapter.

PART 3 – INTEGRATION OF SCRIPTING LANGUAGE INSIDE C++

Final step to make our life sweet enough is addition of some scripting language. Advantages?

there will be ability to add new tests without rebuilding server (actually we even could not restart it)
we could write new tests faster
whole thing becomes more flexible
automation testers feel more comfortable as it’s not C++ (yes, that’s a plus )

Here comes our new hero – ChaiScript. At first I was thinking about lua, but after reading ChaiScript’s docs I decided that language has more that enough flexibility to cover autotests needs. Main advantage of ChaiScript is that it is made for C++ integration and so the one could expect very fast way to add the scripting into the app. ChaiScript is relatively new but mature enough. It has relatively good documentation.

It took me only one day to integrate it into testing environment.

Whole ‘scripting part’ implementation at the moment is just 300 lines of code. And it gave the ability to write all asynchronous tests completely inside separate script files.

Base syntax of ChaiScript is very similar to C++ or JS. In simple case there are only two points the one should change to make C++ test work as script:

// Lambda functions have different declaration
[](){ ... }  

// is replaced with 
fun(){ ... }   

// And -> operator is replaced with .
value.get() // not value->get()

Actually syntax for lambdas is even a bit better as too much square brackets may make things less readable.

Example of test:

// This is script

def start(){

    getCurrentView().assertValue("Start view:", "main");

    async_while(fun(){ return true; }, fun(){

        auto plotter = getElement("main", "plotter").assertOk("Can't find main plotter");
        
        // replot if plot button is visible
        auto bPlot = getElement(plotter, "bPlot");
        global isPlotBtnVisible = getElementVisibility(bPlot);
        async_if(fun(){ return isPlotBtnVisible.get(); },
                 fun(){ clickElement(bPlot); }, 
                 fun(){});

        // select min or max value (random) by clicking on it
        auto bExtremum = getElement(plotter, provide(fun(){ return (random(2) == 0) ? "bMax" : "bMin"; })).assertOk("Can't find min/max button");
        clickElement(bExtremum);

        // get selected value
        auto bSelectedValue = getElement(plotter, "bValue").assertOk("Can't find selected value");
        global valueText = getElementText(bSelectedValue).logValue("Value:").assertCondition("Not zero:", fun(x){ return (x.get().size() > 0); });
        async_do(fun(){
            // here we strip $ sign, convert text into double and print it to log
            auto b = replaceInString(valueText.get(), "\$", "");
            auto balance = to_double(b);
            log("Balance: " + to_string(balance));
        });

        // wait 3 secs
        getCurrentView().setDelay(3000).assertValue("Current view is still:", "main");
    });

};

This is test which every 3 seconds selects min or max value on some plotter, performs some checks and prints it to log. I hope now you can feel the benefits of encapsulation of async requests.

As for interface implementation between scripting and C++ – it is pretty simple. Not only you could export functions and object methods to ChaiScript, but you could also export custom type conversions, complex types and so on. I will not provide implementation details here as it will make the post too big, but you could get the idea from this cheatsheet.

One minor disadvantage: as far as I know there is no default parameter capture modes for lambda functions inside ChaiScript at the moment. If you work with a lot of small functions this could be an improvement to have syntax like fun[=](){ …. } which is not available at the moment. Or even make is as default behaviour. I hope @lefticus will add this in future.

Anyway, ChaiScript seems like a nice solution for me at the moment.

SHORT CONCLUSION

Encapsulation of async execution using modern C++ gives ability to create distributed solutions. This post shown how such technique could be used for creation of custom system of distributed autotests. Scripting could be added into such systems to increase flexibility.

Functional lenses for C++14

Victor Laskin — Thu, 03 Mar 2016 12:09:38 +0000

This post is about one more functional programming idea – LENSES. As you could guess it could be implemented using modern C++.

Concept of lenses from functional programming is actually pretty simple – lens is just pair of getter and setter, which are separated from target object. The main power of lenses is that you could combine them into chain to save a lot of work when you access or modify parts of complex nested structures. The analogy is the following – you can’t get much profit from one lens, but if you combine the bunch of lenses you could make telescope or microscope, etc.

Also there is a big profit when you work with immutable data. In that case a setter is a function which creates a copy of object with modified part. To modify nested structure you could produce very large piece of boilerplate code. But as you will see the lenses could solve this problem brilliantly.

LENS

So a lens is just a pair of functions (getter and setter):

Getter(Object) -> Field + Setter(Object, NewField) -> NewObject

There are several ways to express this using C++ – there is nice presentation here: slides . Another implementation could be found on github – link. And to finish the list there is nice post at Bartosz’s programming cafe – link.

Here is my draft implementation and couple of examples.

// Functional lens: G(T)->F + S(T,F)->T
template ::result >,
typename F = std::decay_t< typename function_traits::result >
>
class Lens {
private:
    G getter;
    S setter;
public:
    // Constructor - provide getter and setter
    Lens(G&& g, S&& s) : getter(std::forward(g)), setter(std::forward(s)) {}
            
    // getter
    F operator()(T holder) const { return getter(holder); };
            
    // setter
    T set(T holder, F&& value) const { return setter(holder, std::forward(value)); }
    T set(T holder, const F& value) const { return setter(holder, value); }
};

~~To construct a lens we could use the following builder:~~

// Lens builder
template 
auto make_lens(G&& g, S&& s){
    return Lens(std::forward(g), std::forward(s));
}

~~As example of immutable data let’s grab the example from previous post – about the way to create immutable structures in C++11:~~

class ScheduleItemData : public IImmutable {
    public:
        const int id;
        const time_t start;
        const time_t finish;
        SERIALIZE_JSON(ScheduleItemData, id, start, finish);
};
        
        
using ScheduleItem = ScheduleItemData::Ptr;
        
        
class EventData : public IImmutable {
public:
    const int id;
    const bool isPublic;
    const string title;
    const double rating;
    const vector schedule;
    const vector tags;
            
    SERIALIZE_JSON(EventData, id, isPublic, title, rating, schedule, tags);
};
        
        
using Event = EventData::Ptr;

This structure represents some event which has a name, rating, some schedule of time intervals and some integer tags. Let’s start with trivial lens created manually – accessor for title field of event.

auto lensTitle = make_lens([](Event e){ return e->title; }, [](Event e, string title){ return e->set_title(title); });

// Sample event:            
Event event = EventData(136, true, "Nice event", 4.88, {ScheduleItemData(1,1111,2222), ScheduleItemData(2,3333,4444)}, {45,323,55});

LOG << lensTitle(event) << NL;
// Nice event
            
auto correctedEvent = lensTitle.set(event, "Very nice event");
LOG << correctedEvent->title << NL;
// Very nice event

~~Note: we even created object after the lens.~~

We could insert constructors for such trivial lenses inside main macro which performs immutability magic! So instead of boilerplate creation of lenses we just need to call static method of immutable class:
auto lensTitle = EventData::lens_title();
So all lens-constructors for basic fields (in current example: isPublic, title, rating, start, finish, etc) are generated automatically.

COMPOSITION

Ok, now main magic – composition of lenses. To make such composition we could make special function (call it zoom for example), or we could use infix operator like the one inside slides. Or we could just use piping again to compose things (post about piping) – so we only need to overload “|” operator.
// Composition of lenses template inline auto operator|(Lens lens1, Lens lens2) { return make_lens([=](T1 holder)->F2{ return lens2(lens1(holder)); }, [=](T1 holder, F2 value)->T1{ return lens1.set(holder, lens2.set(lens1(holder), value)); }); }
Simple!

Let’s imagine we need to modify start time of one item inside our event’s schedule. There is the obvious problem – how to deal with arrays? To solve this we need custom lens which work with lists and focus on specific element of the list. The obvious lens might focus on element inside given vector by index, but to make this more error-proof we could focus on item by id. For example:
// Lens: look at element with specific id in list template ().front())> > auto lens_list_item_by_id(ID id){ return make_lens( [=](C list){ return list | ffind >> [=](auto el){ return (el->id == id); }; }, [=](C list, T value){ return list | filter >> [=](auto el){ return (el->id != id); } | fappend >> value; }); }
I used functional syntax here, but it could be easily rewritten using any regular syntax as you might guess. So consider it as just as example how it might look.

Now we are ready for final example:
auto lens = EventData::lens_schedule() | lens_list_item_by_id>(1) | ScheduleItemData::lens_start(); auto correctedEvent = lens.set(event, 222); LOG << lens(correctedEvent) << NL; // 222 // Print whole structure LOG << correctedEvent->toJSON() << NL; // {"id":136,"isPublic":true,"title":"Nice event \"3,4\"","rating":4.88,"schedule":[{"id":2,"start":3333,"finish":4444},{"id":1,"start":222,"finish":2222}],"tags":[45,323,55]}
We form final lens in one line and then apply it to correct schedule item with id = 1. This could be extended to any kind of complex nested immutable structures. Also you could create any custom lenses as we did to access vector element. It could even have some inner state.

Any way – lenses are brilliant way to modify parts of serious immutable data.

OUT OF FOCUS

There is one minor problem which you probably had experienced when tried to make some photos – sometimes you can’t get object part into focus. The same thing will happen if in previous example you will try to modify schedule item using wrong id. If such id does not exists inside schedule whole chain of lenses will crash. Let’s invent some protection for such case.

Modern camera will not shoot if autofocus system is not locked on something – so let’s add method bool hasFocus(…) to lens. I added this as third must-have parameter to lens constructor but there are probably many other ways.

So lens_item_by_id() and composition operator will be modified accordingly:
// Lens: look at element with specific id in list template ().front())> > auto lens_list_item_by_id(ID id){ return make_lens( [=](C list){ return list | ffind >> [=](auto el){ return (el->id == id); }; }, [=](C list, T value){ return list | filter >> [=](auto el){ return (el->id != id); } | fappend >> value; }, [=](C list){ return list | fcontains >> [=](auto el){ return (el->id == id); }; } ); } // Composition of lenses template inline auto operator|(Lens lens1, Lens lens2) { return make_lens([=](T1 holder)->F2{ return lens2(lens1(holder)); }, [=](T1 holder, F2 value)->T1{ return lens1.set(holder, lens2.set(lens1(holder), value)); }, [=](T1 holder)->bool{ if (!lens1.hasFocus(holder)) return false; return lens2.hasFocus(lens1(holder)); }); }
Now every lenses composition could be checked before run:
auto defocusedLens = EventData::lens_schedule() | lens_list_item_by_id>(10) | ScheduleItemData::lens_start(); LOG << "Is focused?: " << defocusedLens.hasFocus(event) << NL; // FALSE
Here we tried to focus on element with id 10, which was not in the list. If we provide other object which contains such schedule item – the lens will get focus.

This is nice way to avoid a lot of nullptr checks in real cases.

Focus check could be also implemented through some monadic execution of such chain, but this is another topic.

ADAPTIVE OPTICS

Instead of providing a variable to lens constructor we could use a function as predicate parameter. This functional parameter even could be a mutable function or some external dependence.

For example we could write generic find method as lens:
// Lens: find element in array by predicate template ().front())> > auto lens_find(const F& f){ return make_lens( [=](C list){ return fn_findone(list, f); }, [=](C list, T value){ return list | filter >> [f](auto el){ return !f(el); } | fappend >> value; }, [=](C list){ return list | fcontains >> f; } ); }
And use it in the same scheme:
auto sameLens = EventData::lens_schedule() | lens_find>([](auto item){ return (item->id == 1); }) | ScheduleItemData::lens_start(); if (sameLens.hasFocus(event)) LOG << "Value: " << sameLen2(event) << NL; // Value: 1111
Method hasFocus here could still be used to get rid of nullptr checks and in some cases this could be used as solution to replace MayBe monad (for a price – we need to traverse the chain twice). Also we could extend lens with additional parameters in both get/set methods and constructors and get more complex designs.

CONCLUSION

Lenses are nice way to write access/modifications of complex structures.

They have a lot of flexibility to tune.

Auto generation of lens-constructors for basic immutable fields could save a lot of time.

Separation of accessors / modifiers / data is a way to simplicity.

Functional pipelines are suitable way to construct composition of lenses.

Additional method to check focus could replace nullptr checks.

Introducing functional parameters to lens constructors gives additional flexibility.

Named tuple for C++

Victor Laskin — Sat, 02 Jan 2016 17:50:07 +0000

When I use std::pair or std::tuple I always feel sad about using std::get<> to get results. Because this is not so readable, not so maintainable and a bit ugly. Even when you use first/second it’s easy to forget what was first argument of std::pair and so on. When you use tuple to pass more than 3 arguments situation becomes even worse. New standard has nothing in box to solve this, but we can fix this writing our own solution using C++14.

At MeetingCPP 2015 conference I attended nice presentation by Manu Sánchez and found great solution there – named tuple. After some playtime I made my own implementation draft which you can find in this post.

Ok, once again – what’s else wrong with std::tuple?

Look at standard example from cppreference.com:
std::tuple get_student(int id) { if (id == 0) return std::make_tuple(3.8, 'A', "Lisa Simpson"); if (id == 1) return std::make_tuple(2.9, 'C', "Milhouse Van Houten"); if (id == 2) return std::make_tuple(1.7, 'D', "Ralph Wiggum"); throw std::invalid_argument("id"); } int main() { auto student0 = get_student(0); std::cout << "ID: 0, " << "GPA: " << std::get<0>(student0) << ", " << "grade: " << std::get<1>(student0) << ", " << "name: " << std::get<2>(student0) << '\n'; }
What if You by mistake switch ‘GPA’ and ‘grade’ inside your function? Nothing will happen to inform You that something went wrong. Information will still be passed to std::make_tuple, and auto-converted to std::cout. Except the output will be totally wrong… Is there some way to avoid this situation? A lot of you now are thinking about making custom struct with named fields, than and passing typed result as this struct. This is good solution and there are a lot of cases when this will be actually better, but … this is not so simple.

But what if you know that this temporary structure is rather limited in size and will be used just ‘here’ to pass results (and so creating of additional class to pass result will be overhead) – there is nice solution from python – named tuple.

C++14 is powerful enough to make our own named tuple. The only problem is that final syntax could be different depending on our own choice. I have chosen the following:
auto student0 = make_named_tuple(param("GPA") = 3.8, param("grade") = 'A', param("name") = "Lisa Simpson"); auto gpa = student0[param("GPA")]; auto grade = student0[param("grade")];
Creation: Instead of make_tuple() here we have make_named_tuple() and param() is creating named parameter which is set in place.

Access: To access data I use square bracers and same named parameter inside.

There are a lot of other ways to declare this – you are free to use your own syntax. It will be better then std::get<> anyway.

IMPLEMENTATION of tagged tuple

The main trick here is compile time string hash. It does not matter which specific implementation You choose for it – I just grabbed the one from Manu’s gist:
namespace foonathan { namespace string_id { namespace detail { using hash_type = std::uint64_t; constexpr hash_type fnv_basis = 14695981039346656037ull; constexpr hash_type fnv_prime = 109951162821ull; // FNV-1a 64 bit hash constexpr hash_type sid_hash(const char *str, hash_type hash = fnv_basis) noexcept { return *str ? sid_hash(str + 1, (hash ^ *str) * fnv_prime) : hash; } } } } // foonathan::string_id::detail
Next is simple class for named param:
/// Named parameter (could be empty!) template struct named_param : public std::tuple...> { using hash = Hash; ///< key named_param(Ts&&... ts) : std::tuple...>(std::forward(ts)...){ }; ///< constructor template named_param operator=(P&& p){ return named_param(std::forward
(p)); }; }; template using make_named_param = named_param;
Nothing special as You could see – i just store optional value inside inner tuple. So parameter could be empty and be used as static search key or it can contain some value and be passed to tuple construction function.

Now main tuple struct. To make it work with Visual Studio’s compiler I came up with this:
/// Named tuple is just tuple of named params template struct named_tuple : public std::tuple { template named_tuple(Args&&... args) : std::tuple(std::forward(args)...) {} static const std::size_t error = -1; template constexpr typename std::enable_if::type static get_element_index() { return error; } template constexpr typename std::enable_if::type static get_element_index() { using elementType = typename std::tuple_element>::type; return (std::is_same::value) ? I : get_element_index(); } template const auto& get() const { constexpr std::size_t index = get_element_index<0, Hash>(); static_assert((index != error), "Wrong named tuple key"); auto& param = (std::get< index >(static_cast&>(*this))); return std::get<0>( param ); } template const auto& operator[](NP&& param) { return get(); } };
And finally make_named_tuple() and param():
template auto make_named_tuple(Args&&... args) { return named_tuple(std::forward(args)...); } #define param(x) make_named_param< std::integral_constant >{}
Yes, that’s all!

CONCLUSION

So we now have rather compact way to create named tuples in C++. This has to improve readability and simplicity of any code dealing with std::tuples and std::pairs.

And what is even more important – this scheme is more error proof. When you change places of arguments in tuple – nothing happens – everything will still work. If you make mistake in parameter name – you will get compile error right away. If during project lifetime You decide to refactor your tuple structure you will get all places where you need to change access functions, because until you do this compiler will not produce any executable. Sweet.

Source code: gist

C++14: Transducers

Victor Laskin — Tue, 15 Dec 2015 18:30:27 +0000

Transducers – what are they? This is term from Clojure world. There is a classic video from inventor of this term Rich Hickey, and here is some manual page from closure’s docs. But I’ll try to explain all this the simplest possible way so you don’t have to watch all that stuff.

Also I will show my way to implement transducers using C++14 and a lot of examples how to use them.

The idea is simple – if You have sequential process and each inner step could be represented as functional composition of reducing functions – we could make ‘composer tools’ for it (high order functions) and call them transducers.

Perfect analogy from real world – baggage queue in airport. Each bag should go through several operations – check bag, wrap bag, get weight, pass bag, etc. Each iteration could be represented as some reducing function. And transducer is the way to make composition of operations – transduce one reducing function into another (producing desired composition).

So we can write chain of map, reduce, filter, limit, etc but in form of transducers. Instead of performing iterations each of them will produce composition of actions for single sequence element. And finally, when we execute whole functional chain there will be only one cycle through range.

There are already working libs for C++ available so you can play with transducers – one of them is open-source lib called Atria. Enjoy presentation about Atria from cppcon 2015.

But actually it’s not so complicated to implement your own version of transducers. My main aim was to integrate them into my working functional data processing scheme (which is described here). And also create the best possible syntax sugar to make life sweet.

Notice My implementation is a bit different from atria’s one – the difference will be explained below.

USAGE EXAMPLES

Let’s start with basic examples (which were at my slides from MeetingCPP 2015). Even if you are not familiar with basic ideas of functional processing the samples will still be quite readable.

First example is just composition of filter and map for primitive types. I took integers only as example, of course it’s not limited to them. The following example takes only even numbers from vector of ints and than adds one to each of them. Generally this could be done writing one or two circles with if condition inside, which would not be so readable. But we could use transducer here:
vector input{1,2,3,4,5,6,7}; auto comp = tr | tfilter([](int x){ return (x % 2 == 0); }) | tmap([](int x){ return x+1; }); auto result = into(vector(), comp, output_rf, input); // 3 5 7
Second line creates transducer comp (which can be reusable), and last line is performing actual iteration. Output_rf is standard reducing function which just adds new element into provided list. Function into takes four arguments – first is initial state of accumulator, second one is composite transducer, third is initial reducing function and the last one is input range.

We can write same operation as one line:
auto result = input | into_vector << (tr | tfilter([](int x){ return (x % 2 == 0); }) | tmap([](auto x){ return x+1; })); // 3 5 7
Here we switched into with into_vector, which assumes that we accumulating items into empty vector. Once again, there is only one cycle inside this chain. As You can see this line is quite readable.

We could have multiple inputs:
// multiple inputs vector input1{1,2,3,4}; vector input2{6,7,8,9}; auto result = into_vector(tr | tmap([](int x, int y){ return x + y; }), input1, input2); // 7 9 11 13
Also we could have any range as input:
// any range as source auto result = into_vector(tr | tfilter([](int x){ return (x % 5 == 0);}), ints(100)); // 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95
Any class which has begin()/end() interface could be used here.

Another one-liner to show that accumulator could be just one integer which could be used in place:
LOG << "Count: " << into(0, tr | tfilter(is_even), tr_count, ints(2000)) << NL; // Count: 1000
When you need to stop iterating due to some reason it’s possible to extend our transducers to make early termination. There are several ways to do so – this will be discussed further. Now the most obvious example – limit count of elements inside final accumulator.

Also we don’t have to put anything into result storage – here we just perform some function for each element using nullptr as accumulator.
vector{1,2,3,4,5,6,7} | into << nullptr << (tr | tfilter(is_even) | tenumerate() | tlimit(2)) << tr_each([](int n, int x){ LOG << n << ":" << x << NL; }) ; // 0:2 // 1:4
As transducers could deal with any sequence of data (like UI events, network events, etc) it would be nice to have ability to resume iteration process.
auto enumerateStrings = (tr | tenumerate() | tmap([](int n, string s){ return s + " " + SS::toString(n); }))(output_rf); auto result = fn_tr_reduce(vector(), enumerateStrings, vector{"a","b","c"}); // a 0 b 1 c 2 fn_tr_reduce_more(result, enumerateStrings, vector{"e","d"}); // a 0 b 1 c 2 e 3 d 4
As You can see in this example enumeration of strings was continued from right index. Sweet.

If You are not familiar with functional data processing concept at all, such examples might seem not so obvious. But trust me – after on hour playing with this syntax, everything will be perfectly readable.

Enough of examples – let’s proceed to possible implementation details:

IMPLEMENTATION

What do we need to implement transducers? Actually main essence could be written as several lines using C++14!
auto tr_map = [] (auto fn) { return [=] (auto step) { return [=] (auto& out, auto&& ...ins) { return step(out, fn(std::forward(ins)...)); }; }; };
Function which returns function which returns another function… looks totally crazy.

But this is not only problem – Microsoft’s compiler produced internal compiler error on this part of code. So I peeked at Atria implementation and rewrote this code, converting each lambda into templated functor! Instead of several lines I got this (as I need cross-platform solution):
namespace fn_detail { template struct tr_transducer { std::tuple params; tr_transducer(Ts ...ts) : params(std::make_tuple(ts...)){} template auto operator() (RF&& step) const { return this->make(std::forward(step), std::make_index_sequence()); } template auto make(RF step, std::index_sequence) const -> typename G::template apply { return { std::forward(step), std::get(params)... }; } }; struct tr_map_gen { template struct apply { ReducingFnT step; MappingT mapping; template bool operator() (StateT& out, InputTs&& ...ins) { return step(out, mapping(std::forward(ins)...)); } }; }; } template auto tr_map(T f){ return fn_detail::tr_transducer(f); }
If You are using only Clang you can stay with previous implementation and be happy.

This implementation already contains several assumptions. First it contains the way I choose to implement early termination. Each step() returns bool which is false when we need to terminate. Just that simple. And accumulator is passed by reference to step function.

This could be implemented by wrapping result state itself into wrapper struct and this way is described here – video.

Also I want to compose my transducers using functional chain, which was described here. But there is one important difference – when we compose reducing functions, the order of application is reversed. So we need to create reversed functional chain.
/// The inversed chain of functors ... is actualy just a tuple of functors template class fn_chain_reversed { private: const std::tuple functions; template inline typename std::enable_if(functions)(std::declval())) >::type call(Arg arg) const { return std::get(functions)(std::forward(arg)); } template struct final_type : final_type(functions)(std::declval())) > {}; template struct final_type<0, I, Arg> { using type = decltype(std::get(functions)(std::declval())); }; template inline typename std::enable_if::type >::type call(Arg arg) const { return this->call(std::get(functions)(std::forward(arg))); } static const bool isFunctionalChain = true; public: fn_chain_reversed() : functions(std::tuple<>()) {} fn_chain_reversed(std::tuple functions) : functions(functions) {} // add function into chain (inversed order) template< typename F > inline auto add(const F& f) const -> fn_chain_reversed { return fn_chain_reversed(std::tuple_cat(std::make_tuple(f), functions)); } // call whole functional chain template inline auto operator()(Arg arg) const -> decltype(this->call<0,Arg>(arg)) { return call<0>(std::forward(arg)); } };
And to support piping into such chain we need to overload operator like this:
template inline auto operator|(fn_chain_reversed && transducer, F&& rf) -> decltype(transducer.add(rf)) { return transducer.add(std::forward(rf)); }
One more thing – if we have general functional piping overload (more details here), this overload will produce errors during compilation as compiler will not know what to do – just call your chain using provided argument or push new element into chain. To solve this ambiguous situation we just need to add some type traits and SFINAE:
template struct fn_isNotFunctionalChain{ static const bool value = true; }; template<> struct fn_isNotFunctionalChain< fn_chain_reversed<> >{ static const bool value = false; }; //... template::value> > auto operator|(const F& f, T&& param) -> decltype(f(param)) { return f(std::forward(param)); }
Ok, let’s implement main iteration now:
template auto fn_accum_impl(std::index_sequence, RF&& step, A& out, Ranges... ranges) { auto firsts = std::make_tuple(std::begin(ranges)...); auto lasts = std::make_tuple(std::end(ranges)...); bool ok = true; while ((fn_detail::tuple_all_neq(firsts, lasts)) && (ok)) { ok = step(out, std::forward< decltype(*std::begin(ranges)) >(*std::get(firsts))...); fn_detail::noop(++std::get(firsts)...); } }
Only complication here is the need to support several input ranges. So we pack iterators into tuple and increment them all as one step. Also we need additional function to compare two tuples to know when to stop (which I grabbed from atria):
template struct tuple_all_neq_t { template bool operator()(Tuple1T&& t1, Tuple2T&& t2) { return std::get(std::forward(t1)) != std::get(std::forward(t2)) && tuple_all_neq_t{} ( std::forward(t1), std::forward(t2)); } }; template struct tuple_all_neq_t { template bool operator()(Tuple1T&&, Tuple2T&&) { return true; } }; template bool tuple_all_neq(Tuple1T&& t1, Tuple2T&& t2) { constexpr auto size1 = std::tuple_size< std::decay_t >::value; constexpr auto size2 = std::tuple_size< std::decay_t >::value; using impl_t = tuple_all_neq_t<0u, (size1 > size2 ? size2 : size1)>; return impl_t{} ( std::forward(t1), std::forward(t2)); }
All what’s left is to create several more convenient interfaces to call main iteration cycle. into and into_vector are probably the most obvious ways to use transducers from first examples, but they all just call main iteration function – fn_accum_impl.
template auto fn_tr_transduce(C init, T&& transducer, RF&& reducingFunction, Ins... ins) { C out = init; fn_accum_impl(std::make_index_sequence{}, transducer(reducingFunction), out, (std::forward(ins))...); return std::move(out); }; template auto fn_into_vector(RF step, C input, Ins... ins) { return fn_tr_transduce(std::vector>(), step, [] (auto& out, auto input) { out.push_back(input); return true; }, std::forward(input), (std::forward(ins))...); }; template auto fn_tr_reduce(C&& init, T&& step, Ins... ins) { C&& out = std::forward(init); fn_accum_impl(std::make_index_sequence{}, std::forward(step), out, (std::forward(ins))...); return std::move(out); }; #define tr fn_chain_reversed<>() fn_make_universal(tr_reduce, fn_tr_reduce); fn_make_universal(into, fn_tr_transduce); fn_make_universal(into_vector, fn_into_vector);
The macro fn_make_universal converts functor into ‘universal’ functor which supports piping and currrying (that topic is described here).

Now we can just add a bit more of sample transducers (filter, enumerate, limit, each):
struct tr_filter_gen { template struct apply { ReducingFnT step; FilterT pred; template bool operator() (StateT& out, InputTs&& ...ins) { if (pred(std::forward(ins)...)) return step(out, std::forward(ins)...); else return true; } }; }; struct tr_enumerate_gen { template struct apply { ReducingFnT step; int n; template bool operator() (StateT& out, InputTs&& ...ins) { return step(out, n++, std::forward(ins)...); } }; }; struct tr_limit_gen { template struct apply { ReducingFnT step; int n; int limit; template bool operator() (StateT& out, InputTs&& ...ins) { return (n++ > limit) ? false : step(out, std::forward(ins)...); } }; }; template struct tr_each_impl{ T step; template bool operator() (StateT& out, InputTs&& ...ins) { step(ins...); return true; } }; template auto tr_filter(T pred) { return fn_detail::tr_transducer(pred); } static auto tr_enumerate(int n = 0) { return fn_detail::tr_transducer(n); } static auto tr_limit(int limit){ return fn_detail::tr_transducer(1, limit); } template auto tr_each(T step){ return fn_detail::tr_each_impl{step}; }; const auto tr_count = [](auto&s, auto ...ins) { s++; return true; }; fn_make_universal(tmap, tr_map); fn_make_universal(tfilter, tr_filter); fn_make_universal(teach, tr_each); fn_make_universal(tlimit, tr_limit); fn_make_universal(tenumerate, tr_enumerate);
You could implement any custom transducer using this implementation as sketch.

CONCLUSION

Transducers combine long functional data processing chains into one iteration cycle. This is crucial when you need to process large amount of data or have sequence of realtime events.

Transducers are quite readable and simple to use (when you are familiar with primitive functional data processing chains). You need to write core of transducers only once (or You can grab one of libraries which already have suitable implementation).

I hope the benefits are obvious from samples. As one of ways to simplicity transducers were included into my presentation at MeetingCPP 2015 about fighting complexity in modern C++ (slides).

Immutable serialisable data structures in C++11

Victor Laskin — Mon, 29 Jun 2015 11:18:03 +0000

As C++ is very flexible language there are a lot of ways to construct immutable-like data structures from functional programming style. For example, this nice presentation by Kelvin Henney contains one of the ways. Here I will present my way which is slightly different.

At the same time my approach also solves another problem of serialisation of such immutable structures. I use JSON format at the moment, but the approach is not limited to this format only.

Unlike my previous posts I will start from the results this time.

I suppose you are familiar with general concept of immutability. If you are not – please read some minimal introduction to immutable data concept in functional programming.

FIRST PART – USAGE

ATM I have structure declarations simular to this:
class EventData : public IImmutable { public: const int id; const string title; const double rating; SERIALIZE_JSON(EventData, id, title, rating); }; using Event = EventData::Ptr;
Data fields of class are declared as const. And that’s what should be done to be sure that data will not be changed. Note, that I don’t change fields to functions, I don’t hide them below some getters or doing some other tricky stuff. I just mark fields const.

Next – SERIALIZE_JSON. This is macro which contains all the magic. Unfortunately we can’t at the moment achieve introspection without macro declaration. So this is black magic again. I promise next post will not contain macros

The last step is sharing immutable data through smart pointer to decrease data copy overhead introduced by functional-style data processing. And I use such pointers implicitly. For example: plain data object is named as EventData and pointer to such data is named just Event. This is arguable moment – it’s not necessary to follow this notation.

About the price of using shared_ptr there is nice video from NDC conference – The real price of Shared Pointers in C++ by Nicolai M. Josuttis.

Before presenting some usage examples let’s make data a bit more realistic:
class ScheduleItemData : public IImmutable { public: const time_t start; const time_t finish; SERIALIZE_JSON(ScheduleItemData, start, finish); }; using ScheduleItem = ScheduleItemData::Ptr; class EventData : public IImmutable { public: const int id; const string title; const double rating; const vector schedule; const vector tags; SERIALIZE_JSON(EventData, id, title, rating, schedule, tags); }; using Event = EventData::Ptr;
This is description of some event which has id, title, rating, some schedule as pairs of start/finish linux-times, and vector of integer tags. All this just to show nested immutable structures and const vectors as parts of serialisable data.

Note that you can still add some methods into this class. Marking them const will be good idea.

IMMUTABILITY

Ok, it’s time for action! Structure creation is simple:
Event event = EventData(136, "Nice event", 4.88, {ScheduleItemData(1111,2222), ScheduleItemData(3333,4444)}, {45,323,55});
Using new C++ initialisation syntax we can not only form immutable structure the simple way, but also declare all nested collections. All constructors are generated automatically by same black-magic macro.

Important: immutable data does not have empty constructor! You can only create ‘filled’ data state. This is good feature as now it’s very problematic to get corrupted unfully-constructed state. It’s all or nothing. And of course you still could have empty shared_ptr which could contain no immutable data at all.

When you add new field into data structure all places where data was created explicitly will stop to compile. It might seem as bad, but actually this is very good restriction. Now you can’t forget to modify your object construction according new design.

As you can guess all fields could be accessed general way. But any modification will be prevented by compiler.

To modify immutable data we need to create new copy of data with modified field. So it’s not modification, but the construction of new object. Same macro is generating all such constructors:

Note: type is auto-derived using decltype from C++11.

To get the idea how to use such immutable data in functional way using C++11 you can read several posts: post1 post2 post3 (and, probably, more are coming).

SERIALIZATION

Just two methods: toJSON() / fromJSON() are enough to handle all serialisation/deserialisation needs:
// serialisation string json = event->toJSON(); // deserialisation Event eventCopy = EventData::fromJSON(json);
Output:
{"id":136,"title":"Nice event","rating":4.880000,"schedule":[{"start":1111,"finish":2222},{"start":3333,"finish":4444}],"tags":[45,323,55]}
Sweet and simple. And note that we can (de)serialise nested structures / arrays.

Serialisation part is optional – if you only need immutability you don’t have to include this part. Implementation of immutability is not using any serialisation implicitly.

SECOND PART – IMPLEMENTATION

Under the hood there is fusion of macro-magic and C++11 features like decltype. I’ll try to express the main ideas how it was implemented instead of just copying of whole code. If you not trying to implement the given approach yourself you could even skip this and believe me that it works.

The next part will be a bit ugly. Be sure you are 16+ before reading this.

Also I had some additional limitation which absence could make life a bit more easy – i could not use constexpr. I use immutable data inside cross-platform solutions and one of my targets is Windows desktop. I don’t know why Microsoft Visual Studio 2015 still does not like constexpr but it’s support is not yet completed. So don’t be surprised why i’m using couple of old-school macro technics to make functional stuff work in more comfort way.

At first we need macro which applies macro to each of macro parameter.
#define SERIALIZE_PRIVATE_DUP1(M,NAME,A) M(NAME,A) #define SERIALIZE_PRIVATE_DUP2(M,NAME,A,B) M(NAME,A) M(NAME,B) #define SERIALIZE_PRIVATE_DUP3(M,NAME,A,B,C) M(NAME,A) SERIALIZE_PRIVATE_DUP2(M,NAME,B,C) #define SERIALIZE_PRIVATE_DUP4(M,NAME,A,B,C,D) M(NAME,A) SERIALIZE_PRIVATE_DUP3(M,NAME,B,C,D) #define SERIALIZE_PRIVATE_DUP5(M,NAME,A,B,C,D,E) M(NAME,A) SERIALIZE_PRIVATE_DUP4(M,NAME,B,C,D,E) #define SERIALIZE_PRIVATE_DUP6(M,NAME,A,B,C,D,E,F) M(NAME,A) SERIALIZE_PRIVATE_DUP5(M,NAME,B,C,D,E,F) #define SERIALIZE_PRIVATE_DUP7(M,NAME,A,B,C,D,E,F,G) M(NAME,A) SERIALIZE_PRIVATE_DUP6(M,NAME,B,C,D,E,F,G) #define SERIALIZE_PRIVATE_DUP8(M,NAME,A,B,C,D,E,F,G,H) M(NAME,A) SERIALIZE_PRIVATE_DUP7(M,NAME,B,C,D,E,F,G,H) #define SERIALIZE_PRIVATE_DUP9(M,NAME,A,B,C,D,E,F,G,H,I) M(NAME,A) SERIALIZE_PRIVATE_DUP8(M,NAME,B,C,D,E,F,G,H,I) #define SERIALIZE_PRIVATE_DUP10(ME,NAME,A,B,C,D,E,F,G,H,I,K) ME(NAME,A) SERIALIZE_PRIVATE_DUP9(ME,NAME,B,C,D,E,F,G,H,I,K) #define SERIALIZE_PRIVATE_DUP11(ME,NAME,A,B,C,D,E,F,G,H,I,K,L) ME(NAME,A) SERIALIZE_PRIVATE_DUP10(ME,NAME,B,C,D,E,F,G,H,I,K,L) #define SERIALIZE_PRIVATE_DUP12(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M) ME(NAME,A) SERIALIZE_PRIVATE_DUP11(ME,NAME,B,C,D,E,F,G,H,I,K,L,M) #define SERIALIZE_PRIVATE_DUP13(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N) ME(NAME,A) SERIALIZE_PRIVATE_DUP12(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N) #define SERIALIZE_PRIVATE_DUP14(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O) ME(NAME,A) SERIALIZE_PRIVATE_DUP13(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O) #define SERIALIZE_PRIVATE_DUP15(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P) ME(NAME,A) SERIALIZE_PRIVATE_DUP14(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P) #define SERIALIZE_PRIVATE_DUP16(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R) ME(NAME,A) SERIALIZE_PRIVATE_DUP15(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R) #define SERIALIZE_PRIVATE_DUP17(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S) ME(NAME,A) SERIALIZE_PRIVATE_DUP16(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S) #define SERIALIZE_PRIVATE_DUP18(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T) ME(NAME,A) SERIALIZE_PRIVATE_DUP17(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T) #define SERIALIZE_PRIVATE_DUP19(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,Q) ME(NAME,A) SERIALIZE_PRIVATE_DUP18(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,Q) #define SERIALIZE_PRIVATE_DUP20(ME,NAME,A,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,Q,Y) ME(NAME,A) SERIALIZE_PRIVATE_DUP19(ME,NAME,B,C,D,E,F,G,H,I,K,L,M,N,O,P,R,S,T,Q,Y) #define SERIALIZE_PRIVATE_EXPAND(x) x #define SERIALIZE_PRIVATE_DUPCALL(N,M,NAME,...) SERIALIZE_PRIVATE_DUP ## N (M,NAME,__VA_ARGS__) // counter of macro arguments + actual call #define SERIALIZE_PRIVATE_VA_NARGS_IMPL(_1,_2,_3,_4,_5,_6,_7,_8,_9,_10, _11,_12,_13,_14,_15,_16,_17,_18,_19,_20, N, ...) N #define SERIALIZE_PRIVATE_VA_NARGS(...) SERIALIZE_PRIVATE_EXPAND(SERIALIZE_PRIVATE_VA_NARGS_IMPL(__VA_ARGS__, 20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2,1)) #define SERIALIZE_PRIVATE_VARARG_IMPL2(M,NAME,base, count, ...) SERIALIZE_PRIVATE_EXPAND(base##count(M,NAME,__VA_ARGS__)) #define SERIALIZE_PRIVATE_VARARG_IMPL(M,NAME,base, count, ...) SERIALIZE_PRIVATE_EXPAND(SERIALIZE_PRIVATE_VARARG_IMPL2(M,NAME,base, count, __VA_ARGS__)) #define SERIALIZE_PRIVATE_VARARG(M,NAME,base, ...) SERIALIZE_PRIVATE_EXPAND(SERIALIZE_PRIVATE_VARARG_IMPL(M,NAME, base, SERIALIZE_PRIVATE_VA_NARGS(__VA_ARGS__), __VA_ARGS__)) #define SERIALIZE_PRIVATE_DUPAUTO(M,NAME,...) SERIALIZE_PRIVATE_EXPAND(SERIALIZE_PRIVATE_VARARG(M,NAME,SERIALIZE_PRIVATE_DUP, __VA_ARGS__))
This is very ugly, but it works (clang, gcc, VS). As you might guess this specific version could handle up to 20 arguments only.

I will split the whole macro to blocks and to improve readability I will skip ‘\’ sign here which is at the end of each line in multiline macro. Also in each block ‘helper’ macros will be placed below to show content together.

Main constructor:
// Main constructor NAME(SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLEDECL,NAME,__VA_ARGS__) int finisher = 0) : SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLEPARAM,NAME,__VA_ARGS__)IImmutable(){} #define SERIALIZE_PRIVATE_CTORIMMUTABLEDECL(NAME,VAL) decltype(VAL) VAL, #define SERIALIZE_PRIVATE_CTORIMMUTABLEPARAM(NAME,VAL) VAL(VAL),
NAME – is name of class. VA_ARGS is the list of all fields. So macro is expanded in something like this – NAME(decltype(a) a, decltype(b) b,int finisher=0) : a(a), b(b), IImmutable(){} . Here IImmutable is some base class (optional), which could be empty. Also I use finisher to solve last comma problem – may be there is some more clean solution for this.

So other part of magic here is using decltype from C++11.

Additional copy constructors:
// copy constructors NAME(NAME&& other) noexcept : SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLECOPY,NAME,__VA_ARGS__)IImmutable() {} NAME(const NAME& other) noexcept : SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLECOPY,NAME,__VA_ARGS__)IImmutable(){} #define SERIALIZE_PRIVATE_CTORIMMUTABLECOPY(NAME,VAL) VAL(other.VAL),
Note that we can’t move immutable data. Move semantics modify the source object and immutable objects can’t be modified.

The last constructor is from std::tuple. We need it for modification / serialisation methods.
// Constructor from tuple NAME(std::tuple< SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLEDECLTYPENONCONST,NAME,__VA_ARGS__) int> vars) : SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORFROMTUPLE,NAME,__VA_ARGS__)IImmutable() {} #define SERIALIZE_PRIVATE_CTORIMMUTABLEDECLTYPENONCONST(NAME,VAL) typename std::remove_const::type, #define SERIALIZE_PRIVATE_CTORFROMTUPLE(NAME,VAL) VAL(std::get(vars)),
The tricky part here is that to construct from tuple we need to know index of each field inside tuple. To get such index we need C++14’s constexpr with no C++11 limitations (increment inside constexpr). Or we can use __COUNTER__ macro to create additional static fields which contain indexes.
// Index of each field inside class: static const int _index_offset = __COUNTER__; SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_FIELDINDEX,NAME,__VA_ARGS__) #define SERIALIZE_PRIVATE_FIELDINDEX(NAME,VAL) static const int _index_of_##VAL = __COUNTER__ - _index_offset - 1;
Note that __COUNTER__ macro is global so we need to generate additional offset field to store initial value and solve the difference for each index.

Generation of ‘set_’ methods:
// Generation of set_ methods SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CLONEANDSET2,NAME,__VA_ARGS__) // Convert data into std::tuple std::tuple< SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLEDECLTYPENONCONST,NAME,__VA_ARGS__) int> toTuple() const noexcept { return make_tuple(SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_CTORIMMUTABLEVAL,NAME,__VA_ARGS__) 0); } #define SERIALIZE_PRIVATE_CLONEANDSET2(NAME,VAL) NAME::Ptr set_##VAL(decltype(VAL) VAL) const noexcept { auto t = toTuple(); std::get(t) = VAL; return std::make_shared(NAME(t)); }
Unfortunately tuple also has finisher as int type to solve ‘comma’ problem. May be i’ll find a way to make it possible without it.

To make shorter declaration of holding pointers:
// Short smart-pointer declaration typedef std::shared_ptr Ptr;
Additional overload of compare operator:
// compare operator overload bool operator== (const NAME& other) const noexcept { SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_COMPAREIMMUTABLE,NAME,__VA_ARGS__) return true; return false; } #define SERIALIZE_PRIVATE_COMPAREIMMUTABLE(NAME,VAL) if (other.VAL==VAL)
This overload could be changed according to business logic. For example you could compare only id fields.

Whole serialisation part of macro is quite short:
// JSON serialisation is done using my own old lib string toJSON() const noexcept { JSON::MVJSONWriter w; w.begin(); SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_APPENDTOJSON,NAME,__VA_ARGS__) w.end(); return w.result; } static NAME fromJSON(string json) { JSON::MVJSONReader reader(json); return fromJSON(reader.root); } static NAME fromJSON(JSON::MVJSONNode* node) { return NAME( make_tuple(SERIALIZE_PRIVATE_DUPAUTO(SERIALIZE_PRIVATE_FROMJSON,NAME,__VA_ARGS__) 0)); } #define SERIALIZE_PRIVATE_APPENDTOJSON(NAME,VAL) w.add(#VAL,VAL); #define SERIALIZE_PRIVATE_FROMJSON(NAME,VAL) node->getValue(#VAL),
Here I use my old lib for parsing JSON (described here). But you can use any JSON decoder you like. The only restriction is that you have to write wrappers so your lib could read/write values through single entry point. Here I will show an example implementation how to distinguish vector and non vector types.

JSON writer is using simple overloading / template specialisation:
template< typename T > inline string toString(const T& value); template< typename T > inline string toString(const vector& value);
Couple of implementations (for example):
template< typename T > inline string MVJSONWriter::toString(const T& value) { // default overload is for string-like types! return "\"" + value + "\""; } template< typename T > inline string MVJSONWriter::toString(const vector& value) { string result = "["; for (auto item : value) result += ((result != "[") ? "," : "") + item->toJSON(); result += "]"; return result; } template<> inline string MVJSONWriter::toString(const vector& value) { string result = "["; for (auto item : value) result += ((result != "[") ? "," : "") + std::to_string(item); result += "]"; return result; }
So all type variations are hidden inside JSON writing/reading section.

Reader has a bit more complicated form of overloading because we have to overload only output type. To make this possible we have some dummy parameter, std::enable_if and simple type trait for vector detection.
template struct is_vector { static const bool value = false; }; template struct is_vector< std::vector > { static const bool value = true; }; template struct is_vector< const std::vector > { static const bool value = true; }; // inside reader class: template inline T getValue(const string& name, typename enable_if::value, T>::type* = nullptr); template inline T getValue(const string& name, typename enable_if::value, T>::type* = nullptr);
Some overloads:
template inline T MVJSONNode::getValue(const string& name, typename enable_if::value, T>::type*) { MVJSONValue* value = getField(name); if (value == NULL) return ""; return value->stringValue; } template inline T MVJSONNode::getValue(const string& name, typename enable_if::value, T>::type*) { typename std::remove_const::type result; MVJSONValue* value = getField(name); if (value == NULL) return result; for (auto item : value->arrayValue) { result.push_back(remove_pointer().at(0).get())>::type::fromJSON(item->objValue)); } return result; } template<> inline const vector MVJSONNode::getValue(const string& name, typename enable_if>::value, const vector>::type*) { vector result; MVJSONValue* value = getField(name); if (value == NULL) return result; for (auto item : value->arrayValue) { result.push_back((int)item->intValue); } return result; }
So you need to specify overloads for all your trivial types and containers. In practice this is not so compact but anyway you should do it only once.

CONCLUSION

Once again, all this ugly implementation detail is written only once and is located on low level of architecture. Upper business layer just uses this utility as black box and should not be aware of inner implementation.

Anyway the main idea of this post was to show how compact declaration of immutable data could be in C++11, and that you don’t have to write more boilerplate code for each business class declaration.

Any comments are welcome.

UPDATE: I put some source code on github as 2 gists:

My old JSON lib – https://gist.github.com/VictorLaskin/1fb078d7f4ac78857f48

Declaration – https://gist.github.com/VictorLaskin/48d1336e8b6eea16414b

Please, treat this code like just an example.

Separating constraints, iterations and data (C++11)

Victor Laskin — Sun, 17 May 2015 21:37:45 +0000

Two recent posts in Bartosz’s programming cafe describe nice application of list monad to solve the following puzzle:

Each letter corresponds to single digit. There are a lot of methods to solve this. Bartosz is using list monad which is very simular to list comprehension methods which were described here. While this maybe not the fastest way to solve this specific puzzle, his approach shows the ways to solve large cluster of similar but smaller problems which we meet at basic “production” level. SEND+MORE problem may be is not so good example to show power of list monad because of one very important problem i want to discuss in this post.

Let’s rephrase the puzzle – we have 8 different variables and have find all combinations which satisfy some constraints.

Straightforward solution – form all possible combinations of variables and filter them using constraint conditions. To form such combinations we use general iteration over the list of possible values.

The problem: when the list of values is not small enough or count of variables is greater than 3 we could face performance problem as iterations become too long.

Size of SEND+MORE puzzle is close enough to meet this problem but still modern cpu could handle it straightforward way.

SLIGHTLY DIFFERENT WAY

The main idea is to separate iterations, data and constraints. And the most important: we need to split one global constraint into smaller constraints and apply them as early as possible.

And while doing that i want to maintain the simplicity of Bartosz’s solution.

To make it all work i will use some tools from my previous post . Main parts are currying of functions and functional piping.

DATA:
// using sInt = std::shared_ptr; // the list of possible values vector digits = {0,1,2,3,4,5,6,7,8,9}; // variables to find sInt s,e,n,d,m,o,r,y; // additional vars (described further) sInt r0,r1,r2,r3; // fill variables (0) for_each_argument_reference([](sInt& i){ make(i,0); }, s,e,n,d,m,o,r,y,r0,r1,r2,r3);
I use shared_ptr to access data here – that’s not important in this particular example. Even raw pointers could show the idea. The important part is the list of possible values called digits, and pointers to all variables.

Next – let’s define CONSTRAINTS:
// No constraint auto any = to_fn([](sInt x){ return true; });
Constraint is a function which returns “true” if given value has passed the condition. So constraint any gives green light to every integer. to_fn function here is just converting lambda into std::function.
// This is how we add numbers digit by digit: // 0 + d + e = y + r1 * 10 // r1 + n + r = e + r2 * 10 // r2 + e + o = n + r3 * 10 // r3 + s + m = o + m * 10 auto fn_constraint = to_fn([](sInt r0, sInt x, sInt y, sInt z, sInt r){ // r0 + x + y = x + r * 10 *r = *r0 + *x + *y - *z; if (*r == 0) return true; if (*r == 10) { *r = 1; return true; }; return false; }); const auto constraint = fn_to_universal(fn_constraint);
Instead of trying to invent some tricky constrains we go very simple and logical way – our constraint just defines how we add decimal numbers. Nothing more – nothing else. r0,r1,r2,r3 – are numbers which go on next column during addition.

Only one ‘not so nice’ step here is setting r through pointer. This is done to be able to use it during the following more deep constraints.

After definition i’m wrapping function into universal class which could handle currying and piping – see this post for details.

The last column of digits is an exception – so we have to define separate constraint for it:
auto fn_last_constraint = to_fn([](sInt r0, sInt x, sInt y, sInt z){ // r0 + x + y = x + y * 10 return (*y != 0) && (*r0 + *x + *y == *z + *y * 10); }); const auto last_constraint = fn_to_universal(fn_last_constraint);
Note that we are also checking for first number to be non zero.

So finally instead of one global constraint we have 4 smaller constrains and could apply them sooner to decrease amount of iterations.

ITERATIONS

Functional iterator is simple:
void fn_pick(sInt x, function constraint, function)> process, vector list) { for (auto item : list) { *x = item; if (constraint(x)) process(list | filter >> [&](int el){ return (el != item); }); } } fn_make_universal(pick, fn_pick);
This function is just picking all possible values from the list, apply constraint and if it was positive check, calls process method for reduced list of values (which does not contain picked value).

The last piece is the function to print the result:
auto printResult = [&](vector list){ printf("RESULT %i%i%i%i + %i%i%i%i = %i%i%i%i%i \n", *s,*e,*n,*d,*m,*o,*r,*e,*m,*o,*n,*e,*y); };
FINALLY
digits | pick << d << any << (pick << e << any << (pick << y << (constraint << r0 << d << e >> r1) << (pick << n << any << (pick << r << (constraint << r1 << n >> e >> r2) << (pick << o << (constraint << r2 << e >> n >> r3) << (pick << s << any << (pick << m << (last_constraint << r3 << s >> o ) << printResult ))))))); // RESULT 9567 + 1085 = 10652
Sorry that i’m using my ‘<<‘ notation for currying here – as this might be not ideal solution. I hope this will not prevent you to understand the idea of segregation. Of cause operation overloading could be changed to some another notation. Note that i use left and right currying together inside constraints.

This is compact enough to show the main idea – decomposing iterations and constraints.

My debug version is solving this puzle in 7ms.

PS. What I don’t like about this solution is using pointers too much. But we could change the design to pass data along functional chain without using pointers, but this could make the solution a bit more complicated. May be i will fix it later. Also i’m looking for the way to get rid of ‘)))))))’ stuff.

PS2: Whole puzzle solution together:
using sInt = std::shared_ptr; // ITERATIONS void fn_pick(sInt x, function constraint, function)> process, vector list){ for (auto item : list) { *x = item; if (constraint(x)) process(list | filter >> [&](int el){ return (el != item); }); } } fn_make_universal(pick, fn_pick); // DATA vector digits = {0,1,2,3,4,5,6,7,8,9}; sInt s,e,n,d,m,o,r,y; sInt r0,r1,r2,r3; for_each_argument_reference([](sInt& i){ make(i,0); }, s,e,n,d,m,o,r,y,r0,r1,r2,r3); // CONSTRAINTS auto any = to_fn([](sInt x){ return true; }); auto fn_constraint = to_fn([](sInt r0, sInt x, sInt y, sInt z, sInt r){ // r0 + x + y = x + r * 10 *r = *r0 + *x + *y - *z; if (*r == 0) return true; if (*r == 10) { *r = 1; return true; }; return false; }); const auto constraint = fn_to_universal(fn_constraint); auto fn_last_constraint = to_fn([](sInt r0, sInt x, sInt y, sInt z){ // r0 + x + y = x + y * 10 return (*y != 0) && (*r0 + *x + *y == *z + *y * 10); }); const auto last_constraint = fn_to_universal(fn_last_constraint); // print out the result auto printResult = [&](vector list){ printf("RESULT %i%i%i%i + %i%i%i%i = %i%i%i%i%i \n", *s,*e,*n,*d,*m,*o,*r,*e,*m,*o,*n,*e,*y); }; // ROCK&ROLL digits | pick << d << any << (pick << e << any << (pick << y << (constraint << r0 << d << e >> r1) << (pick << n << any << (pick << r << (constraint << r1 << n >> e >> r2) << (pick << o << (constraint << r2 << e >> n >> r3) << (pick << s << any << (pick << m << (last_constraint << r3 << s >> o ) << printResult )))))));
PS3. Bartosz’s programming cafe is a very good place to visit.

Templates as first-class citizens in C++11

Victor Laskin — Wed, 04 Mar 2015 15:22:29 +0000

C++11 treats functions as first-class citizens and this gives us ability to construct a lot of nice things as design patterns borrowed from functional languages. Meanwhile C++ has very powerful template metaprogramming. This post is aimed to make templated functions closer to first-class citizens, and to construct some simplicity and beauty which you can get from it.

Also here will be implementation for currying of such functions! If you don’t know what currying is – just remember std::bind.

And to make it shine we’ll add piping. (This article will improve some ideas from post about functional piping ). This step is optional and you can replace such piping with commas and function calls.

TEMPLATE TO SIMPLE FUNCTOR

Ok, let’s start from very simple template function.
template void print(T t) { cout << t << endl; }
I will use different styles of code colouring here to indicate difference between example code and inner implementation which is similar for all cases.

To pass it to some function we need to wrap it into functor:
class tfn_print { public: template auto operator()(Args&&... args) const ->decltype(print(std::forward(args)...)) { return print(std::forward(args)...); } }
Here operator() is overloaded to pass all arguments to print template. As we will modify this functor to cover all cases, the list of arguments is provided using variadic templates. Note: you can define all functions inside such wrappers from the start, but there is way to do it through macro:
#define make_citizen(X) class tfn_##X { public: template auto operator()(Args&&... args) const ->decltype(X(std::forward(args)...)) { return X(std::forward(args)...); } } make_citizen(print);
This small macro creates class from templated function.

Important note for macro haters. Yes, macros should be avoided when you can substitute them using variadic templates and other new stuff. But still there are some cases when you can’t do that. So when you have couple of very small obvious macro definitions which will make your code considerably smaller, more readable and maintainable – use it. Such cases are rare and when it happens C++ committee should look upon it and add some means into the language itself to fix it.

Example:
// testing print... { tfn_print print; print(5); print("hello"); }
We redefined print in smaller scope as instance of object. Now we can use it as function and pass as argument to another function. Note that the same function is used with arguments of different type – so no need to create distinct functors.

Next we will create some additional instruments to work with such ‘templated’ functors more effectively.

PASS TUPLE TO FUNCTION AS ARGUMENT LIST

As we will use such ability later – let’s write simple function which will expand std::tuple and feed results into the given function.
// apply tuple to function... namespace fn_detail { template struct int_sequence {}; template struct gen_int_sequence : gen_int_sequence {}; template struct gen_int_sequence<0, S...> { typedef int_sequence type; }; template inline auto fn_tuple_apply(int_sequence, const F& f, const std::tuple& params) -> decltype( f((std::get(params))...) ) { return f((std::get(params))...); } } template inline auto fn_tuple_apply(const F& f, const std::tuple& params) -> decltype( f(std::declval()...) ) { return fn_detail::fn_tuple_apply(typename fn_detail::gen_int_sequence::type(), f, params); }
In C++14 this could be done a bit shorter using std::integer_sequence, but at the moment I’m forced to use just C++11 ( to compile things for Android for example). The sequence of integer indices S is constructed through template recursion, and than is passed as additional argument for unpacking. This is just one of possible implementations.

Usage:
auto f = [](int x, int y, int z) { return x + y - z; }; auto params = make_tuple(1,2,3); auto res = fn_tuple_apply(f, params); print(res); // Result: 0
The next step is the idea that you can combine function input parameters into tuple using several steps instead of one std::make_tuple call.

TUPLE CONCATENATION

For concatenation of tuples C++11 has function named std::tuple_cat. To make things more compact we can overload << operator to add new parameter into tuple.
// tuple concatenation via << operator template tuple operator<<(const tuple & t, const NewArg& arg) { return std::tuple_cat(t, std::make_tuple(arg)); }
Usage:
auto list = make_tuple(1,4); auto res2 = fn_tuple_apply(f, list << 4); // f(1,4,4) print(res2); // Result: 1
FUNCTIONAL PIPELINE

See post about functional piping to get the idea (we use different implementation for piping here). Anyway, this step is optional and you can replace such piping with commas and function calls, but piping here looks very nice.

First simple overload will provide ability to pipe single argument into function:
// pipe single argument into function via | operator template auto operator|(T&& param, const F& f) -> decltype(f(param)) { return f(std::forward(param)); }
Usage:
// Return count of elements as templated operator template int count(const T& container) { return container.size(); } make_citizen(count); { tfn_count count; vector slist = {"one", "two", "three"}; slist | count | print; } // Result: 3
One new sample template function count() is just returning size() of provided collection. We provide the list of three strings into count function and after that pipe the result into print().

This is nice but application seems to be rather limited as functions tend to have more than one argument. Let’s solve this using some curry…

CURRYING

Currying in C++11 is usually made using std::bind, which is flexible and useful for a lot of cases. But we can’t use it for templates. So let’s create some wrapper class which will implement curry and provide the simplest way to work with it inside pipelines.

General way (like in haskell) is to curry from left to right argument – so call f(1,2,3) is equivalent to f(1)(2)(3). But in piping example everything is slightly different –f(1,2,3) is equivalent to 1 | f(2,3). In other words I want to curry both ways – left and right. Yes, std::bind gives ability to specify any order but this is for price of a bit longer syntax that it’s required in general cases. And haskell’s syntax is not so good also, imho. That’s because it’s hard for programmer to identify difference between currying and function call. So here i’m using different syntax (and this decision is optional) where you have separate operators for function call and argument curry.

The next class is called fn_universal as representation of some function which is polymorphic (in sense that it can accept arguments of different types) and can be used with currying and piping. May be this class will be extended further later. You might want to rename it to fn_curryed or something like it.

Of course, to hold arguments we will use tuples. The implementation:
// universal function / extended function wrapper ! template, typename TPackAfter = std::tuple<>> class fn_universal { private: F f; ///< main functor TPackAfter after; ///< curryed arguments TPackBefore before; ///< curryed arguments public: fn_universal(F && f) : f(std::forward(f)), after(std::tuple<>()), before(std::tuple<>()) {} fn_universal(const F & f, const TPackBefore & before, const TPackAfter & after) : f(f), after(after), before(before) {} template auto operator()(Args... args) const -> decltype( fn_tuple_apply(f, std::tuple_cat(before, make_tuple(args...), after)) ) { // execute via tuple return fn_tuple_apply(f, std::tuple_cat(before, make_tuple(std::forward(args)...), after)); } // curry template auto curry(T && param) const -> decltype(fn_universal(f, std::tuple_cat(before, std::make_tuple(param)), after)) { return fn_universal(f, std::tuple_cat(before, std::make_tuple(std::forward(param))), after); } template auto curry_right(T && param) const -> decltype(fn_universal(f, before, std::tuple_cat(after, std::make_tuple(param)))) { return fn_universal(f, before, std::tuple_cat(after, std::make_tuple(std::forward(param)))); } };
Note that class is immutable.

The main nice thing about this class: there is no restriction on types or count of arguments. All what is done here is combining provided arguments into two tuples – right and left parameters. And when time comes to execute function we just combine everything into single tuple and feed this tuple into function. So you can curry function which supports variadic number of arguments!

And to provide easy interface for currying I use the following overloads for operators:
// left curry by << operator template auto operator<<(const UF & f, Arg && arg) -> decltype(f.template curry(std::forward(arg))) { return f.template curry(std::forward(arg)); } // right curry by >> operator template auto operator>>(const UF & f, Arg && arg) -> decltype(f.template curry_right(std::forward(arg))) { return f.template curry_right(std::forward(arg)); }
Let’s add small builder helper. Also (very optional) I might add definition macro to make examples a bit shorter.
template auto fn_to_universal(F && f) -> fn_universal { return fn_universal(std::forward(f)); } #define make_universal(NAME, F) make_citizen(F); const auto NAME = fn_to_universal(tfn_##F());
This line is just defining new ‘universal’ function from given template function. You could change this using more convenient way.

EXAMPLES FOR CURRYING

Trivial examples:
// currying.... auto f = [](int x, int y, int z) { return x + y - z; }; auto uf = fn_to_universal(f); auto uf1 = uf << 1; auto uf2 = uf1 << 2 << 5; uf2() | print; // result: -2 // Piping: 1 | (uf << 4 << 6) | print; // 4+6-1 = 9 3 | (uf >> 6 >> 7) | print; // 3+6-7 = 2
Note the order of arguments. Not so complicated to read.

Now let’s write couple of template functions to get some realistic examples. This will look like functional operators. I even could mark these functions as inner implementation as they are so common, but to underline that you have the ability to control this behaviour yourself I leave it marked as sample:
// MAP template class C, typename F> auto fn_map(const C& container, const F& f) -> C()))> { using resultType = decltype(f(std::declval())); C result; for (const auto& item : container) result.push_back(f(item)); return result; } // REDUCE (FOLD) template class C, typename F> TResult fn_reduce(const C& container, const TResult& startValue, const F& f) { TResult result = startValue; for (const auto& item : container) result = f(result, item); return result; } // FILTER template class C, typename F> C fn_filter(const C& container, const F& f) { C result; for (const auto& item : container) if (f(item)) result.push_back(item); return result; } make_universal(fmap, fn_map); make_universal(reduce, fn_reduce); make_universal(filter, fn_filter);
It’s obvious that such primitives can be reused avoiding a lot of code duplication.

And one minor function – make sum of all arguments:
template T sum_impl(T arg, Args... args) { T result = arg; [&result](...){}((result += args, 0)...); return result; } make_universal(sum, sum_impl);
Here could be more trivial implementation and couple of overloads, but to show that you have no limitation with count of arguments I leave it like this.

Also we can modify print to handle any amount of arguments:
template void print(Args... args) { (void)(int[]){((cout << args), 0)...}; cout << endl; } auto uprint = fn_to_universal(print);
AND NOW: Let’s try this in action:
vector slist = {"one", "two", "three"}; // all strings as one slist | (reduce >> string("") >> sum) | (uprint << "All: "); // All: onetwothree // sum of elements of array vector{1,2,3} | (reduce >> 0 >> sum) | (uprint << "Sum: "); // Sum: 6 // count sum length of all strings in the list slist | (fmap >> count) | (reduce >> 0 >> sum) | (uprint << "Total: " >> " chars"); // Total: 11 chars
This is quite readable and functional-style code. Not the one we could expect from C++.

So the idea is that now we can write small universal templates and combine them in pipelines as we like. Looks great!

Consider small templates as small universal building blocks.

More examples…

Let’s assume we are working with some business data like collection of users.
template bool isNameEqualImpl(const T& obj, const TName& name) { return (obj->name == name); } make_universal(isName, isNameEqualImpl); template bool isIdEqualImpl(const T& obj, const TId& id) { return (obj->id == id); } make_universal(isId, isIdEqualImpl); template bool isNotNullImpl(const T& t) { return (t != nullptr); } make_universal(isNotNull, isNotNullImpl); template class C> T findOneImpl(const C& container, const F& f, const TKey& key) { for (const auto& item : container) if (f(item, key)) return item; return nullptr; } make_universal(ffind, findOneImpl);
First 3 methods are trivial – checking that name or id fields are equal to given values and that whole object is not null.

The last one is find method inside container (we assume here that business data items are presented as immutable structures inside smart pointers). Find receives one function as validate functor and key argument to pass to this validator. Let’s see how to use it:
// example data vector users {make(1, "John", 0), make(2, "Bob", 1), make(3, "Max", 1)}; users | (filter >> (isName >> "Bob")) | ucount | uprint; // 1 users | (filter >> (isId >> 13)) | ucount | uprint; // 0 vector{1,2,6} | (fmap >> (ffind << users << isId)) | (filter >> isNotNull) | ucount | uprint; // 2
Such examples do not require an explanation I suppose!

We can convert user names to xml like this:
string xmlWrapImpl(string name, string item) { return "<" + name + ">" + item + ""; } make_universal(xmlWrap, xmlWrapImpl); // Produce XML users | fmap >> [](User u){ return u->name; } | fmap >> (xmlWrap << "name") | reduce >> string("") >> sum | xmlWrap << "users" | print; // result: JohnBobMax
As you can see we can still use lambdas inside expressions.

FUNCTIONAL CHAINS

Now let’s add one more important element to proposed scheme. I want to be able to compose pipeline of functions, store it under some name and then call it multiple times as single function. In other words i want this:
auto countUsers = chain((fmap >> (ffind << users << isId)) | (filter >> isNotNull) | ucount); vector{1,2,6} | countUsers | (uprint << "count of users: ");
In fact this is simple functional composition (like the one discussed in appendix here ), and it can be implemented with ease in case of general functions. But when we are talking about templates things become a bit more complicated. But this is still possible!
// -------------------- chain of functors ---------------------> // The chain of functors ... is actualy just a tuple of functors template class fn_chain { private: const std::tuple functions; template inline typename std::enable_if(functions)(std::declval())) >::type call(Arg arg) const { return std::get(functions)(std::forward(arg)); } template struct final_type : final_type(functions)(std::declval())) > {}; template struct final_type<0, I, Arg> { using type = decltype(std::get(functions)(std::declval())); }; template inline typename std::enable_if::type >::type call(Arg arg) const { return this->call(std::get(functions)(std::forward(arg))); } public: fn_chain() {} fn_chain(std::tuple functions) : functions(functions) {} // add function into chain template< typename F > inline auto add(const F& f) const -> fn_chain { return fn_chain(std::tuple_cat(functions, std::make_tuple(f))); } // call whole functional chain template inline auto operator()(Arg arg) const -> decltype(this->call<0,Arg>(arg)) { return call<0>(std::forward(arg)); } };
How this works? When constructing such chain we just put functors into tuple. Tricky part comes when we need to call the function – we need to construct universal caller without knowing function specifications. This is possible using compile-time recursion and std::decltype/std::declval. Creating dummy arguments we go through whole chain recursively and detect final result type.

And to make it work under GCC i hit one of compiler bugs – inside call method recursion during type detection you need to add ‘this->‘ to make it compile (bug).

And to chain functors into the list we need to overload | operator again:
template inline auto operator|(fn_chain && chain, F&& f) -> decltype(chain.add(f)) { return chain.add(std::forward(f)); }
Now we can use this whole concept like:
// Use functional chain: auto f1 = [](int x){ return x+3; }; auto f2 = [](int x){ return x*2; }; auto f3 = [](int x) { return (double)x / 2.0; }; auto f4 = [](double x) { return SS::toString(x); }; auto f5 = [](string s) { return "Result: " + s; }; auto testChain = fn_chain<>() | f1 | f2 | f3 | f4 | f5; // execution: testChain(3) | print; auto countUsers = fn_chain<>() | (fmap >> (ffind << users << isId)) | (filter >> isNotNull) | ucount; vector{1,2,6} | countUsers | (uprint << "count of users: ");
Note that we use different types during chain execution.

So now we have the ability to store and combine pipelines as we like. Nice.

FINAL TOUCH – MONADIC PIPING (OPTIONAL)

This part is optional but it fits so well… because here will be very smooth shift to monads. Smooth and easy.

What if we change the direction of piping? Let’s pipe functions into data instead piping data into functions! Of cause, we can’t do this with raw data – we need some object to wrap it and call received functions for us. This object can do some additional manipulations with functions along the way. And yes, thats a kind of monad.

Why we need this kind of stuff? Because we can gain additional control of functional chain evaluation.

As example i’ll use maybe monad – which has very simple logic: if we have nullptr as output from any function in chain – we don’t execute further and report empty state. This is kind of implicit error protection.

Implementation will be a bit extended to support pointer and non-pointer types.

Also, this is not ‘true’ monad as true monadic bind operation requieres functions which return monad as result. But using the following implementation you can apply normal functions without any modifications.
// ------------------ maybe --------------------------> enum class maybe_state { normal, empty }; template typename std::enable_if< std::is_object::value, T>::type set_empty() { return T(); } template<> int set_empty() { return 0; } template<> string set_empty() { return ""; } template class maybe { private: const maybe_state state; const T x; template maybe fromValue(R&& result) const { return maybe(std::forward(result)); } template maybe> fromValue(std::shared_ptr&& result) const { if (result == nullptr) return maybe>(); else return maybe>(std::forward>(result)); } public: // monadic return maybe(T&& x) : x(std::forward(x)), state(maybe_state::normal) {} maybe() : x(set_empty()), state(maybe_state::empty) {} // monadic bind template auto operator()(F f) const -> maybe()))> { using ResultType = decltype(f(std::declval())); if (state == maybe_state::empty) return maybe(); return fromValue(f(x)); } // extract value T getOr(T&& anotherValue) const { return (state == maybe_state::empty) ? anotherValue : x; }; }; template inline auto operator|(maybe && monad, F&& f) -> decltype(monad(f)) { return monad(std::forward(f)); } template inline T operator||(maybe && monad, TDefault&& t) { return monad.getOr(std::forward(t)); } template maybe just(T&& t) { return maybe(std::forward(t)); }
This class can be extended to handle error messages and so on.

I use the same pipe | operator here because its ‘unix’ meaning is perfectly applicable here.

Examples:
maybe(2) | (ffind << users << isId) | [](User u){ return u->name; } | [](string s){ cout << s << endl; return s; }; // Bob (maybe(6) | (ffind << users << isId) | [](User u){ return u->name; }).getOr("Not found") | (uprint << "User: "); // User: Not found just(vector{1,2,6}) | countUsers | [&](int count){ count | (uprint << "Count: "); return count; }; // Count: 2
So here we execute processing chain on non existing user and nothing bad happens. Also we can pipe our saved chain countUsers into maybe as expected.

I could produce a lot of other examples here, but may be this is nice topic for another post.

CONCLUSION

Using compile-time recursion and type-detection you can create powerful tools for combining templated functional blocks. Using currying and piping together with functional chains gives very flexible instrument to build compact functional-style methods in C++11. Monadic execution could be added to this scheme without any problems.

Full working sample on github: gist

ps. About templating – i think usage of templates in production should be very minimalistic because template overuse can lead to very unmaintainable code. So when using proposed scheme keep in mind that all blocks should be very small and logically reasonable.

C++11: Implementation of list comprehension in SQL-like form

Victor Laskin — Mon, 16 Feb 2015 18:29:31 +0000

List comprehension in functional languages is the name of list constructor syntax, which is similar to set-builder notation from math.

What are the benefits of using list comprehension? One is readability, and the other one is the fact of decoupling iteration from actual construction. We could even hide parallel execution under the hood of list comprehension. Also by adding additional options to such declaration we could make the list construction a lot shorter.

If we look closer at list comprehension’s syntax, it will remind of one another very familiar thing – SQL select! Output expression, input set, predicates are equivalent to select, from, where sequence (of course not exactly, but they are very alike). Ok, let’s implement such syntax sugar using C++11 (without boost and LINQ-like libs).

You can skip implementation details and scroll to examples to see the result first.

SHORT IMPLEMENTATION

Let’s define whole operation as simple function which will produce vector<> of something.
template vector select(std::function f, ///< output expression const std::tuple& sources, ///< list of sources const std::function& filter, ///< composition of filters const Options&... options ///< other options and flags ) { ... }
Feel the power of variadic templates. First argument is simple function and its declaration defines output type and input variables. The next argument is tuple of source containers and their number should be equal to number of output expression’s arguments. Third argument is composition of ‘where’ filters. The rest parameters are optional flags.

You would expect a lot of code as implementation but the core will be quite compact.

Let’s write main processing cycle:
template inline typename std::enable_if::type for_each_in_sources(const std::tuple &, FuncT& f, Args&... args) { return f(args...); } template inline typename std::enable_if::type for_each_in_sources(const std::tuple& t, FuncT& f, Args&... args) { bool isFinished; for(auto& element : std::get(t)) { isFinished = for_each_in_sources(t, f, args..., element); if (isFinished) break; } return isFinished; } // .... inside ..... vector result; int count = 0; auto process = [&](const Args&... args){ if (filter(args...)) { result.push_back(f(args...)); count++; } return (count == limit); // isFinished }; for_each_in_sources(sources, process);
This is quite strait-forward approach – we will go through all combinations of input elements performing simple ranged for cycles. So when using list comprehension the one should keep in mind that providing too vast input arrays will result in too slow computation time. For example, if we have 3 sources, complexity will be O(n^3). So i decided to add some minor protection here – limit of requested data (it is like LIMIT instruction inside SQL-request syntax). When the limit is reached – cycles will be aborted (but this anyway will not solve all possible conditions).

As for main working part – here we use template recursion over tuple of sources, performing a cycle on each step. If you are not familiar with such approach here is working sample of simple iteration over tuple elements.

All other stuff below is optional!

First additional thing to add is set of optional flags to setup query limit, sorting options and so on. For implementation of simple bool flags we would just use class enum from C++11, but if we want to set complex options (like manual sorting with compare function, etc) we could go with the following approach:
class SelectOption { public: virtual bool imPolymorphic() { return true; } }; class SelectOptionLimit : public SelectOption { public: int limit; SelectOptionLimit(int limit) : limit(limit) {} }; class SelectOptionSort : public SelectOption {}; class SelectOptionDistict : public SelectOption {}; // ..... INSIDE select() ..... int limit = -1; bool isDistinct = false; bool isSorted = false; for_each_argument([&](const SelectOption& option){ if (auto opt = dynamic_cast(&option)) { limit = opt->limit; }; if (dynamic_cast(&option)) { isSorted = true; }; if (dynamic_cast(&option)) { isDistinct = true; }; }, options...); // .... HERE IS MAIN CYCLE .... // sort results if ((isDistinct) || (isSorted)) std::sort(result.begin(), result.end()); // remove duplicates if (isDistinct) { auto last = std::unique(result.begin(), result.end()); result.erase(last, result.end()); }
Now our select has 3 additional options – limit, district and sort. Of course, we assume our result items to be comparable if we expect sort to work.

We can add additional declaration to be able to call select function without filters when we do not need them:
template inline vector select(std::function f, const std::tuple& sources, const Options&... options) { return select(f, sources, to_fn([](Args... args){ return true; }), options...); }
And to have fun let’s do some evil thing…
#define SELECT(X) select(to_fn(X), #define FROM(...) std::make_tuple(__VA_ARGS__) #define WHERE(...) ,fn_logic_and(__VA_ARGS__) #define SORT ,SelectOptionSort() #define DISTINCT ,SelectOptionDistict() #define LIMIT(X) ,SelectOptionLimit(X)) #define NOLIMIT LIMIT(-1)
Aside from quite obvious shortcuts for function calls we are making two important conversions here. First – we are converting first function into std::function using to_fn(). This is equivalent to to_function() from my previous post about functional decomposition in C++ and AOP. In short, it’s equal to casting to std::function<..> with automatic detection of arguments types and return type.

Second thing is more interesting – fn_logic_and(…). This method is making one function from bunch of filter functions (to be provided as third argument into main list comprehension function). This is not classic composition of functions as we don’t have chained calls, but instead it’s like logical operation over list of functions. For implementation of fn_logic_and() look at post bottom. This function is optional – you can pass the filter functions as tuple and iterate over it or store them as vector.

Note that if we don’t want to make copies of sources (as std::make_tuple does it) we can switch to std::forward_as_tuple as the solution for passing tuple of references. But this might not be so safe. Anyway the approach itself does not require to copy the sources – you can use references or smart-pointers to pass sources (after minor modifications).

As final touch before getting something that works, we need source class for range of natural numbers (as the majority of list comprehension examples use integers). To use ranged for over this source, we need to implement Iterator inside.
class Naturals { int min; int max; public: Naturals() : min(1),max(1000) {} Naturals(int min, int max) : min(min),max(max) {} int at(int i) const { return i + min; } ; int size() const { return max - min + 1; } ; class Iterator { int position; public: Iterator(int _position):position(_position) {} int& operator*() { return position; } Iterator& operator++() { ++position; return *this; } bool operator!=(const Iterator& it) const { return position != it.position; } }; Iterator begin() const { return { min }; } Iterator end() const { return { max }; } };
Note that by default the range is only [1..1000].

Ok, let’s have some fun.

EXAMPLES OF LIST COMPREHENSION

Let’s start from something trivial:
SELECT([](int x){ return x; }) FROM(Naturals()) LIMIT(10); // List: 1 2 3 4 5 6 7 8 9 10
We can implement map operation as list comprehension:
auto ints = {11,2,1,5,6,7}; SELECT([](int x){ return x + 5; }) FROM(ints) NOLIMIT; // List: 16 7 6 10 11 12 SELECT([](int x)->string { return x % 2 == 0 ? "BOOM" : "BANG"; }) FROM(ints) NOLIMIT; // List: 'BANG' 'BOOM' 'BANG' 'BANG' 'BOOM' 'BANG'
Ok, time for filters. Let’s find matching elements inside two sets of integers (and sort them):
auto ints = {11,2,1,5,6,7}; auto ints2 = {3,4,5,7,8,11}; result = SELECT([](int x, int y){ return x; }) FROM(ints, ints2) WHERE([](int x, int y){ return (x == y); }) SORT NOLIMIT; // List: 5 7 11
Another example using DISTINCT:
SELECT([](int x, int y){ return x + y; }) FROM(Naturals(), Naturals()) WHERE([](int x, int y){ return (x*x + y*y < 25); }) DISTINCT LIMIT(10); // List: 2 3 4 5 6
Next example from Erlang guide on list comprehension – pythagorean triplets:
int N = 50; SELECT([&](int x, int y, int z){ return make_tuple(x,y,z); }) FROM(Naturals(1,N), Naturals(1,N), Naturals(1,N)) WHERE([&](int x, int y, int z){ return (x+y+z <= N); }, [&](int x, int y, int z){ return (x*x + y*y == z*z); }) LIMIT(N);
Result:
-> 3 4 5 -> 4 3 5 -> 5 12 13 -> 6 8 10 -> 8 6 10 -> 8 15 17 -> 9 12 15 -> 12 5 13 -> 12 9 15 -> 12 16 20 -> 15 8 17 -> 16 12 20
We could run select using custom data classes ( select id and name from users where id in [….] )
vector users {make(1, "John", 0), make(2, "Bob", 1), make(3, "Max", 1)}; auto ints = {11,2,1,5,6,7}; SELECT([](int x, User u){ return make_pair(u->id, u->name); }) FROM(ints, users) WHERE([](int x, User u){ return (u->id == x); }) LIMIT(10); // -> 2, Bob // -> 1, John
We also can perform nested selects:
SELECT([](int x, int y){ return x*y; }) FROM( SELECT([](int x){return x;}) FROM(Naturals()) WHERE([](int x){ return x % 2 == 0; }) NOLIMIT, SELECT([](int x){return x;}) FROM(Naturals()) WHERE([](int x){ return x % 2 == 1; }) NOLIMIT ) SORT LIMIT(10); // Result: 2 6 10 14 18 22 26 30 34 38
Sure in haskell this will be a lot more compact, but, as you can see, it’s not as scary as expected from C++.

LAZY EVALUATION

If we want to make whole select lazy this can be done pretty easily:
template std::function()> select_lazy( const std::function& f, ///< output expression const std::tuple& sources, ///< list of sources const std::function& filter, ///< composition of filters const Options&... options ///< other options and flags ) { return to_fn([=](){ return select(f, sources, filter, options...); }); } #define LAZYSELECT(X) select_lazy(to_fn(X), auto get = LAZYSELECT([](int& x){ return x; }) FROM(ints) WHERE([](int& x){ return x % 2 == 0; }) LIMIT(20); LOG << "Evaluate it later... " << NL; auto result = get(); // evaluation
Notice that function contains capture by value inside. This is optional choice. Anyway, if your ‘where‘ filters contain capture by reference of some additional parameters this could lead to undefined behaviour when is used out of scope. But this is general practice with lambdas.

There is nice post from Bartosz here about laziness, list comprehension and C++’s way to implement it from different angle. Here is described more strait-forward and thus more simple and clear way (it does not mean it’s better). We don’t wrap data and sources into something and so on, but implementation of laziness from Bartosz has one nice feature – it gives you ability to request results of query part by part as true lazy evaluation should do. Let’s add this feature into current scheme.

Let’s add new option, which will contain information about current request state. For more compact example it will just contain array of integer indexes (you can change it to Iterators if you want).
class SelectContinuation : public SelectOption { public: vector indexes; };
When this option is provided, iterations over tuple of sources should be slightly altered:
template inline typename std::enable_if::type for_each_in_sources_indexed(const std::tuple &, FuncT& f, vector& indexes, Args&&... args) { return f(args...); } template inline typename std::enable_if::type for_each_in_sources_indexed(const std::tuple& t, FuncT& f, vector& indexes, Args&&... args) { bool isFinished; int size = std::get(t).size(); int i = indexes[I]; if (I != 0) if (i >= size) i = 0; if (i >= size) return false; for (; i < size; i++) { isFinished = for_each_in_sources_indexed(t, f, indexes, args..., std::get(t).at(i)); if (isFinished) break; } indexes[I] = ++i; return isFinished; }
We just changed cycle to indexed version. Function stores last iteration positions inside indexes[] array and starts from it when iteration is called again. One of options would be encapsulating this state inside select functor, but here we store state it inside outer option object. This is just one of possible ways.

Let’s change Pythagorean Triplets example… (this is not final version)
// just for beauty #define LAZY(X) ,(X) SelectContinuation state; auto lazypyth = LAZYSELECT([&](int x, int y, int z){ return make_tuple(x,y,z); }) FROM(range, range, range) WHERE([&](int x, int y, int z){ return (x+y+z <= N) && (z > y) && (y > x); }, [&](int x, int y, int z){ return (x*x + y*y == z*z); }) LAZY(state) LIMIT(5); // Request 5 items... auto pyth = lazypyth(); // Output: -> 3 4 5 // -> 5 12 13 // -> 6 8 10 // -> 7 24 25 // -> 8 15 17 // Request 5 more items... pyth = lazypyth(); // Result: -> 9 40 41 // -> 10 24 26 // -> 11 60 61 // -> 12 16 20 // -> 12 35 37
Looks nice? But there is one major problem. Lazy evaluation could work with very large ranges of input data (even infinite ranges are fine in Haskell). If we provide Naturals(1,100000000) as input for our 3 variables in current example, the cycles will run forever… What to do to solve this? Many ways:

we could recreate input sources before each iteration depending on current indexes of upper cycles

we could wrap sources into classes which will provide method narrowing iteration range before each cycle

we could provide additional restricting functions as options which will break inner cycles conditionally

The most flexible way is providing functions which will produce sources instead of forwarding sources themselves (as expected in functional world…). There is no problem to change implementation to handle such input, but i want add one more feature – i want to support both object sources and functions which will produces sources simultaneously. And even to be able to use them mixed in single request. And again, to implement this we could add relatively tiny layer of code (for C++):

Inside main cycles we add additional call to getSource() function to provide the source on each iteration. So now it looks like:
template inline typename std::enable_if< I < sizeof...(Tp), bool>::type for_each_in_sources_indexed(const std::tuple& t, FuncT& f, vector& indexes, Args&&... args) { bool isFinished; auto&& src = getSource(std::get(t), args...); int size = src.size(); int i = indexes[I]; if (I != 0) if (i >= size) i = 0; if (i >= size) return false; for (; i < size; i++) { isFinished = for_each_in_sources_indexed(t, f, indexes, args..., src.at(i)); if (isFinished) break; } indexes[I] = ++i; return isFinished; }
And getSource() has conditional implementation depending on which source type it gets. The main problem here is how to identify that provided class is lambda function?

To do this we can use std::enable_if in couple with std::declval and std::is_object:
template inline typename std::enable_if< std::is_object().begin()) >::value, Src&&>::type getSource(Src && src, Args&&... args) { return std::forward(src); } template ()(std::declval()...))> inline typename std::enable_if()(std::declval()...)) >::value, FRes >::type getSource(F && f, Args&&... args) { return std::forward(f(std::forward(args)...)); }
First variant checks that object has begin() method. If it has it – it’s raw object. Second variant just attempts to get result type of possible function call. Luckily we can just pass the list of arguments as variadic template.

Let’s test it:
auto lazypyth = LAZYSELECT([&](int x, int y, int z){ return make_tuple(z,y,x); }) FROM(Naturals(1,100000000), [](int x){ return Naturals(1,x); }, [](int x,int y){ return Naturals(1,y); }) WHERE([&](int x, int y, int z){ return x*x == y*y + z*z; }) LAZY(state) LIMIT(5); lazypyth(); // Result: // -> 3 4 5 // -> 6 8 10 // -> 5 12 13 // -> 9 12 15 // -> 8 15 17 lazypyth(); // Result: // -> 12 16 20 // -> 15 20 25 // -> 7 24 25 // -> 10 24 26 // -> 20 21 29
We had to change the order of parameters and now it works fast enough (enough for not optimised example code).

Note: the same way you can extend select to support different types of sources. For example, wrapped inside shared_ptr<>, etc.

Also note that after all modifications this is still just simple function. All macros are optional and you could use select() as normal function.

HOW ABOUT CONCURRENCY?

Making acceptable assumptions that our sources are safe to iterate over at the same time, there will be no problem to extend our sample to support concurrent execution! We can slightly modify continuation option, which we used for laziness by adding special constructors. This will allow us to manually set iteration start indexes as job parameters.
class SelectContinuation : public SelectOption { public: vector indexes; SelectContinuation(){} template SelectContinuation(Args... args) : indexes{ args... } { } };
To be able to call select function in concurrent manner let’s add alias:
template std::function(const SelectContinuation& job)> select_concurrent( const std::function& f, ///< output expression const std::tuple& sources, ///< list of sources const std::function& filter, ///< composition of filters const Options&... options ///< other options and flags ) { return to_fn([=](const SelectContinuation& job){ return select(f, sources, filter, job, options...); }); } // optional #define CONCURRENTSELECT(X) select_concurrent(to_fn(X),
That’s all! Next is the example how run select concurrently:
// concurrent test SelectContinuation job1(0); SelectContinuation job2(100); auto get = CONCURRENTSELECT([](int& x){ return x; }) FROM(Naturals(1,200)) WHERE([](int& x){ return x % 2 == 0; }) LIMIT(10); // two async threads auto part1 = std::async(std::launch::async, [&](){ return get(job1); }); auto part2 = std::async(std::launch::async, [&](){ return get(job2); }); part1.wait(); part2.wait(); print(part1.get()); print(part2.get()); // Result: // 10 numbers: 2 4 6 8 10 12 14 16 18 20 // 10 numbers: 102 104 106 108 110 112 114 116 118 120
This approach can be extended to support additional border conditions and automatic split into parts.

You also could implement implicit concurrency inside main iterations of list comprehension by splitting upper cycle into several equal parts.

CONCLUSION

It’s possible to implement list comprehension in C++11 without using LINQ or similar libs. It will not be haskell-short but still you can gain some profits from it, as more readable and maintainable code in some cases and so on. Combined with custom self-defined additional options you can make such declaration even more profitable.

Lazy and concurrent evaluation could be added with no big effort.

But be aware of inner implementation – as there can be cases when naive list comprehension can lead to high cpu-consumption. Also in some cases list comprehension can have longer declaration than other ways of functional processing possible in modern C++ (like the one discussed here).

If you don’t like SQL syntax or macros you can still use list comprehension as function without any macro. If you invent more short names for options, add short function from() which will just make tuple from arguments, you can already use it like:
// without macro select(to_fn([](int x,int y){ return x + y; }), from(ints, ints), where([](int x, int y){ return x+y<10; }), limit(10));
and it’s also possible to get rid of to_fn() there.

Working example can be found on ideone / gist

APPENDIX: LOGIC OPERATIONS OVER FUNCTION LIST (OPTIONAL)

We could pass filters just as vector of functions slightly changing ‘checking’ part of cycle, so the next part is totally optional. But the following code might be useful somewhere else.

This part contains two functions: fn_compose(…) and fn_logic_add(…).

The first one is making normal mathematical composition: if you have input functions f1(),f2(),f3() it will return f(..) = f3(f2(f1(..))).

fn_logic_add will return f(..) = f1(..) && f2(..) && f3(..) as single std::function. Of cause, all input functions should have the same list of arguments and bool return type. This can be considered as logical and operation over the list of functions.
template struct function_composition_traits : public function_composition_traits {}; template struct function_composition_traits { typedef std::function composition; typedef std::function boolOperation; template inline static composition compose(const Func1& f1, const Func2& f2) { return [f1,f2](Args1... args) -> ReturnType2 { return f2(f1(std::forward(args)...)); }; } template inline static boolOperation logic_and(const Func1& f1, const Func2& f2) { return [f1,f2](Args1... args) -> bool { return f1(std::forward(args)...) && f2(std::forward(args)...); }; } }; // fn_compose template typename function_composition_traits::composition fn_compose(const F1& lambda1,const F2& lambda2) { return function_composition_traits::template compose(lambda1, lambda2); } template auto fn_compose(F f, Fs... fs) -> decltype(fn_compose(f, fn_compose(fs...))) { return fn_compose(f, fn_compose(std::forward(fs)...)); } // fn_logic_and template typename function_composition_traits::boolOperation fn_logic_and(const F1& lambda1,const F2& lambda2) { return function_composition_traits::template logic_and(lambda1, lambda2); } template auto fn_logic_and(F f, Fs... fs) -> decltype(fn_logic_and(f, fn_logic_and(fs...))) { return fn_logic_and(f, fn_logic_and(std::forward(fs)...)); } template auto fn_logic_and(F f) -> decltype(to_fn(f)) { return to_fn(f); }
This can be extended to support wide range of other operations over the list of functors. As soon as i expand it to size of some considerable value i will publish this part coupled with other additional handy conversion functions as tiny lib or something on github.

Any thoughts and corrections are welcome at comments.

C++11 functional decomposition – easy way to do AOP

Victor Laskin — Tue, 03 Feb 2015 11:41:31 +0000

This post is about making functional decomposition from perspective of Aspect Oriented Programming using C++11. If you are not familiar with ideas of AOP don’t be afraid – it’s rather simple concept, and by the end of this post you will understand the benefits of it.

You also can treat this post just as example how to use high-order functions in C++11.

In short – AOP tries to perform decomposition of every business function into orthogonal parts called aspects such as security, logging, error handling, etc. The separation of crosscutting concerns. It looks like:

Old picture about AOP

Since C++11 supports high-order functions now we can implement factorization without any additional tools and frameworks (like PostSharp for C#).

You can scroll down to ‘what for’ chapter to check out the result to get more motivated.

PART 1 – TRIVIAL SAMPLE

Let’s start from something simple – one aspect and one function.

Here is simple lambda with trivial computation inside:
auto plus = [](int a, int b) { LOG << a + b << NL; };
I want to add some logging before and after computation. Instead of just adding this boilerplate code into function body let’s go other way. In C++11 we just can write high-order function which will take function as argument and return new function as result:
template std::function wrapLog(std::function f) { return [f](Args... args){ LOG << "start" << NL; f(args...); LOG << "finish" << NL; }; }
Here we used std::function, variadic templates and lambda as result. (LOG, NL – my own logging stream and you can just change it with std::cout , std::endl or your another logging lib).

As i hoped to achieve the most simple and compact solution, i expected to use it like this:
auto loggedPlus = wrapLog(plus);
Unfortunately this will not compile. ‘no matching function to call ….’ The reason is that lambda is not std::function and automatic type conversion can’t be done. Of cause we can write something like this:
auto loggedPlus = wrapLog(static_cast>(plus));
This line will compile, but this is ugly… I hope cpp committee will fix this casting issue. Meanwhile, the best solution i found so far is the following:
template struct function_traits : public function_traits {}; template struct function_traits { typedef ReturnType (*pointer)(Args...); typedef std::function function; }; template typename function_traits::function to_function (Function& lambda) { return typename function_traits::function(lambda); }
This code is using type traits to convert anonymous lambda into std::function of same type. We can use it like this:
auto loggedPlus = wrapLog(to_function(plus));
Not perfect but much better. Finally we can call functional composition and get the result.
loggedPlus(2,3); // Result: // start // 5 // finish
Note: if we had declared aspect function without variadic template we could compose functions without to_function() conversion, but this would kill the benefit from writing universal aspects discussed further.

PART 2 – REALISTIC EXAMPLE

Introduction is over, let’s start some more real-life coding here. Let’s assume we want to find some user inside database by id. And while doing that we also want log the process duration, check that requesting party is authorised to perform such request (security check), check for database request fail, and, finally, check in local cache for instant results.

And one more thing – i don’t want to rewrite such additional aspects for every function type. So let’s write them using variadic templates and get as universal methods as possible.

Ok, let’s start. I will create some dummy implementation for additional classes like User, etc. Such classes are only for example and actual production classes might be completely different, like user id should not be int, etc.

Sample User class as immutable data:
// Simple immutable data class UserData { public: const int id; const string name; UserData(int id, string name) : id(id), name(name) {} }; // Shared pointer to immutable data using User = std::shared_ptr;
Let’s emulate database as simple vector of users and create one method to work with it (find user by id):
vector users {make(1, "John"), make(2, "Bob"), make(3, "Max")}; auto findUser = [&users](int id) -> Maybe { for (User user : users) { if (user->id == id) return user; } return nullptr; };
make<> here is just shortcut for make_shared<>, nothing special.

Maybe<> monad

You, probably, noticed that return type of request function contains something called Maybe. This class is inspired by Haskell maybe monad, with one major addition. Instead of just saving Nothing state and Content state, it also might contain Error state.

At first, here is sample type for error description:
/// Error type - int code + description class Error { public: Error(int code, string message) : code(code), message(message) {} Error(const Error& e) : code(e.code), message(e.message) {} const int code; const string message; };
Here is minimalistic implementation of Maybe:
template < typename T > class Maybe { private: const T data; const shared_ptr error; public: Maybe(T data) : data(std::forward(data)), error(nullptr) {} Maybe() : data(nullptr), error(nullptr) {} Maybe(decltype(nullptr) nothing) : data(nullptr), error(nullptr) {} Maybe(Error&& error) : data(nullptr), error(make_shared(error)) {} bool isEmpty() { return (data == nullptr); }; bool hasError() { return (error != nullptr); }; T operator()(){ return data; }; shared_ptr getError(){ return error; }; }; template Maybe just(T t) { return Maybe(t); }
Note, that you don’t have to use Maybe<> and here it’s used only for example.

Here we also use the fact that nullptr in C++11 has it’s own type. Maybe has defined constructor from that type producing nothing state. So when you return result from findUser function, there is no need for explicit conversion into Maybe<> – you can just return User or nullptr, and proper constructor will be called.

Operator () returns possible value without any checks, and getError() returns possible error.

Function just() is used for explicit Maybe construction (this is standard name).

Logging aspect

First, let’s rewrite log aspect so it will calculate execution time using std::chrono. Also let’s add new string parameter as name for called function which will be printed to log.
template std::function logged(string name, std::function f) { return [f,name](Args... args){ LOG << name << " start" << NL; auto start = std::chrono::high_resolution_clock::now(); R result = f(std::forward(args)...); auto end = std::chrono::high_resolution_clock::now(); auto total = std::chrono::duration_cast(end - start).count(); LOG << "Elapsed: " << total << "us" << NL; return result; }; }
Note std::forward here for passing arguments more clean way. We don’t need to specify return type as Maybe because we don’t need to perform any specific action like error checking here.

‘Try again’ aspect

What if we have failed to get data (for example, in case of disconnect). Let’s create aspect which will in case of error perform same query one more time to be sure.
// If there was error - try again template std::function(Args...)> triesTwice(std::function(Args...)> f) { return [f](Args... args){ Maybe result = f(std::forward(args)...); if (result.hasError()) return f(std::forward(args)...); return result; }; }
Maybe<> is used here to identify error state. This method can be extended – we could check error code and decide is there any sense to perform second request (was it network problem or database reported some format error).

Cache aspect

Next thing – let’s add client side cache and check inside it before performing actual server-side request (in functional world this is called memoization). To emulate cache here we can just use std::map:
map userCache; // Use local cache (memoize) template std::function(K,Args...)> cached(C & cache, std::function(K,Args...)> f) { return [f,&cache](K key, Args... args){ // get key as first argument if (cache.count(key) > 0) return just(cache[key]); else { Maybe result = f(std::forward(key), std::forward(args)...); if (!result.hasError()) cache.insert(std::pair(key, result())); //add to cache return result; } }; }
This function will insert element into cache if it was not there. Here we used that knowledge that cache is std::map, but it can be changed to any key-value container hidden behind some interface.

Second important part, we used only first function argument here as key. If you have complex request where all parameters should act as composite key – what to do? It’s still possible and there are a lot of ways to make it. First way is just to use std::tuple as key (see below). Second way is to create cache class which will allow several key parameters. Third way is to combine arguments into single string cache using variadic templates. Using tuple approach we can rewrite it like this:
map,User> userCache; // Use local cache (memoize) template std::function(Args...)> cached(C & cache, std::function(Args...)> f) { return [f,&cache](Args... args){ // get key as tuple of arguments auto key = make_tuple(args...); if (cache.count(key) > 0) return just(cache[key]); else { Maybe result = f(std::forward(args)...); if (!result.hasError()) cache.insert(std::pair(key, result())); //add to cache return result; } }; }
Now it’s much more universal.

Security aspect

Never forget about security. Let’s emulate user session with some dummy class –
class Session { public: bool isValid() { return true; } } session;
Security checking high-order function will have additional parameter – session. Checking will only verify that isValid() field is true:
// Security checking template std::function(Args...)> secured(S session, std::function(Args...)> f) { // if user is not valid - return nothing return [f, &session](Args... args) -> Maybe { if (session.isValid()) return f(std::forward(args)...); else return Error(403, "Forbidden"); }; }
‘Not empty’ aspect

Last thing in this example – let’s treat not found user as error.
// Treat empty state as error template std::function(Args...)> notEmpty(std::function(Args...)> f) { return [f](Args... args) -> Maybe { Maybe result = f(std::forward(args)...); if ((!result.hasError()) && (result.isEmpty())) return Error(404, "Not Found"); return result; }; }
Im not writing here about error handling aspect, but it’s also can be implemented via same approach. Note that using error propagation inside Maybe<> monad you can avoid using exceptions and define your error processing logic different way.

Multithread lock aspect
template std::function locked(std::mutex& m, std::function f) { return [f,&m](Args... args){ std::unique_lock lock(m); return f(std::forward(args)...); }; }
No comments.

FINALLY

Finally, what for was all this madness? FOR THIS LINE:
// Aspect factorization auto findUserFinal = secured(session, notEmpty( cached(userCache, triesTwice( logged("findUser", to_function(findUser))))));
Checking (let’s find user with id 2):
auto user = findUserFinal(2); LOG << (user.hasError() ? user.getError()->message : user()->name) << NL; // output: // 2015-02-02 18:11:52.025 [83151:10571630] findUser start // 2015-02-02 18:11:52.025 [83151:10571630] Elapsed: 0us // 2015-02-02 18:11:52.025 [83151:10571630] Bob
Ok, let’s perform tests for several users ( here we will request same user twice and one non-existing user ):
auto testUser = [&](int id) { auto user = findUserFinal(id); LOG << (user.hasError() ? "ERROR: " + user.getError()->message : "NAME:" + user()->name) << NL; }; for_each_argument(testUser, 2, 30, 2, 1); //2015-02-02 18:32:41.283 [83858:10583917] findUser start //2015-02-02 18:32:41.284 [83858:10583917] Elapsed: 0us //2015-02-02 18:32:41.284 [83858:10583917] NAME:Bob //2015-02-02 18:32:41.284 [83858:10583917] findUser start //2015-02-02 18:32:41.284 [83858:10583917] Elapsed: 0us // error: //2015-02-02 18:32:41.284 [83858:10583917] ERROR: Not Found // from cache: //2015-02-02 18:32:41.284 [83858:10583917] NAME:Bob //2015-02-02 18:32:41.284 [83858:10583917] findUser start //2015-02-02 18:32:41.284 [83858:10583917] Elapsed: 0us //2015-02-02 18:32:41.284 [83858:10583917] NAME:John
As you can see it’s working as intended. It’s obvious that we got a lot of benefits from such decomposition. Factorisation leads to decoupling of functionality, more modular structure and so on. You gain more focus on actual business logic as result.

We can change order of aspects as we like. And as we made aspect functions rather universal we can reuse them avoiding a lot of code duplication.

Instead of functions we can use more sophisticated functors (with inheritance), and instead of Maybe<> also could be more complex structure to hold some additional info. So whole scheme is extendable.

Note also, that you can pass lambdas as additional aspect parameters.

Working sample to play with: github gist or ideone

Ps. BONUS:
template void for_each_argument(F f, Args&&... args) { (void)(int[]){(f(forward(args)), 0)...}; }

Small presentation of my cross-platform engine for mobile and desktop applications

Victor Laskin — Wed, 21 Jan 2015 14:50:39 +0000

I made small presentation about my cross-platform engine for mobile and desktop applications. Codename Kobald. Click on image to play it in new window (use arrows and space to move through):

This is not-so-technical presentation and main info about engine will come later as separate post.

Functional pipeline in C++11

Victor Laskin — Mon, 19 Jan 2015 15:26:55 +0000

I stumbled upon this nice blog post – pipable functions in C++14. This is realy fun idea as its usage plain for anybody who is familiar with unix pipelines. So i tried to use it in C++11 (without boost) from slightly different angle to make it more real-life concept.

First sample (fun, but not so interesting):
auto add = piped([](int x, int y){ return x + y; }); auto mul = piped([](int x, int y){ return x * y; }); int y = 5 | add(2) | mul(5) | add(1); // Output: 36
Second sample (functional style array processing):
vector numbers{4,8,15,16,23,42}; auto result = numbers | where([](int x){ return (x > 10); }) | map([](int x){ return x + 5; }) | log(); // List: 20 21 28 47
Simple and short implementation is under the cut…

To implement the first sample you need only the following short number of lines (C++11 without boost):
template struct pipeable { private: F f; public: pipeable(F&& f) : f(std::forward(f)) {} template auto operator()(Xs&&... xs) -> decltype(std::bind(f, std::placeholders::_1, std::forward(xs)...)) const { return std::bind(f, std::placeholders::_1, std::forward(xs)...); } }; template pipeable piped(F&& f) { return pipeable{std::forward(f)}; } template auto operator|(T&& x, const F& f) -> decltype(f(std::forward(x))) { return f(std::forward(x)); }
You can see a lot of C++11 features here coupled together – perfect forwarding, &&, std::bind, auto and decltype. Using this you can even define functions inside pipeline:
int y2 = 5 | add(2) | piped([](int x, int y){ return x * y; })(5) | piped([](int x){ return x + 1; })(); // Output: 36
For second sample we need to define some extension functions. We don’t need linq library to make things work though. Let’s do it generic way:
template T whereInList(const T& list, std::function f) { T result; for (auto& item : list) if (f(item)) result.push_back(item); return result; } template T mapToList(const T& list, std::function f) { T result; for (auto& item : list) result.push_back(f(item)); return result; } template T logList(const T& list) { std::cout << "List: "; for (auto& item : list) std::cout << item << " "; std::cout << std::endl; return list; } template auto piped1Arg(F f) -> decltype(piped(std::bind(f, placeholders::_1))) { return piped(std::bind(f, placeholders::_1)); } template auto piped2Args(F f) -> decltype(piped(std::bind(f, placeholders::_1, placeholders::_2))) { return piped(std::bind(f, placeholders::_1, placeholders::_2)); }
Processing functions here are just example and not so suitable for large amount of data – but whole concept is still viable.

Last two functions are used to create pipable functors from static methods:
vector numbers{4,8,15,16,23,42}; using setType = decltype(numbers); auto where = piped2Args(whereInList); auto map = piped2Args(mapToList); auto log = piped1Arg(logList); auto result = numbers | where([](int x){ return (x > 10); }) | map([](int x){ return x + 5; }) | log();
As you can see the concept requires very compact implementation. If you somehow applied this in real project plz write in comments.