Tube Design Tutorial | PyF, flow-based python programming Tube Design Tutorial

Tube Design Tutorial

This tutorial assumes you’ve already installed and setup pyf.services. If you haven’t, please follow the steps on the getting started pages of the website.

This tutorial will show how to design a simple tube directly with your web browser on your PyF.Services instance.

The tube that we will create is similar to the ones we created in the other tutorials (one producer, two filters, two adapters and one csv writer, similar to a process managing user accounts for security).

Finding you way around and creating the tube

  1. Click on the Tube menu item.
    ../../../../_images/tube_menu.png
  2. Now, create a new tube in design view
    ../../../../_images/newtube.png
  3. Name your tube, declare it as active and standalone
    ../../../../_images/props.png

Designing the tube

  1. First, let’s drag and drop a code producer on our tube.
    Drag and drop from the left list:
    ../../../../_images/dragndrop-anim.png
  2. Our node is pretty empty right now.

  1. First, we will name it: enter “source1” in the “Name” field.

  2. Then, enter code in the code editor. We will add a generator that will yield simple python objects representing users:

    class User(object):
        def __init__(self, name, email, level):
            self.name = name
            self.email = email
            self.level = level
    def get_source():
        for index in range(0, 10):
            yield User(
                    "John%04d" % index,
                    "john%04d@some-where.com" % index,
                    ['high', 'low', 'high', 'high', 'low'][index%5])
    

    Hint

    To get more information about nodes from code (code items), see pyf.componentized.components.CodeComponent

  3. Here is what your node should look like:
    ../../../../_images/producer.png
  1. Now, let’s separate the flow in two branches, one with the “high” level and another branch with the “low” and “medium” items.
  1. For our first branch, we will use a plugin, named SimpleFilter that will only let the items validating the expression “item.level == ‘high’” pass through it.
    ../../../../_images/filter1.png
  2. Link the OUT port of the producer to the IN port of the filter adapter.
    ../../../../_images/link-dnd-small.png
  3. For our other branch, let’s try another method of filtering : a custom code component.

    Note

    A very important thing when you make adapters, is to remember the way PyF works : it uses generators.

    So, to let the other branches keep executing you have to yield items. For that matter, we use a special Python objects : Ellipsis. Those objects will be ignored by the rest of the components but let you say “I’ve received an item, but I don’t want to send anything in the flow” (it is very useful for filters or aggregators that will take a lot of items and yield Totals at the end for example).

    A good example code for custom filtering is this:
    def low_or_med(items):
        for item in items:
            if item.level == "low" or item.level == "med":
                yield item
            else:
                yield Ellipsis
    
  4. Now we have our two branches:
    ../../../../_images/two_branches.png
  1. Now that we have two branches, we want to adapt data in those branches. There is a few ways to do that, and like at the step before, we will try two ways:
  1. For our first branch, we will use the plugin way, with a plugin named ComputeAttributes that is specialised in adaptation of input objects by defining attribute getters on them, without modifying the input object (the objects that are outputed by this plugin are proxy objects, DynamicObjectAdapter with attribute getters).
    ../../../../_images/adapter1.png
    We added an adapter named numeric_level that translates the level in a numeric fashion.

    Note

    Each of the attribute getters can access item (the adapted item, with access to other attributes getter defined, and base_item, the item not adapted (useful to redefine attributes)).

  2. For our second branch, we will do a code version. Here is the code we use for this:

    def set_numeric_level(items):
        for item in items:
            item.numeric_level = 0
            yield item
    

    Please not that this example isn’t the best one as we modify the entering object (so you shouldn’t make it pass in another branch at the same time), using Packets removes this problem as packets ca be modified and won’t be the same on all branches (another solution is to use DynamicObjectAdapter).

  1. Last step is to add a CSVWriter. This is self explanatory, really. Just one thing, if you don’t specify the attribute, the column title will be used.
Here is what your tube should look like :
../../../../_images/tutorial_tube.png
  1. Oh, last but not least, you should “Save” your Tube (there is a button for that on the left).

Launching your tube

  1. To launch your tube, either click the launch button from the design view or visit the tube page (the green arrow on the actions for the tubes in the tube list) and click Launch.
  2. Once launched, you are on the event track page with progression and messages.
  • To add messages there, you can call message_callback in your nodes
  • To update progression, use progression_callback in the producer.