Reactive applications, Shaders and all that

posted by on 2018.04.06, under Processing
06:

We have already discussed the advantage of using shaders to create interesting visual effects. This time we will have to deal with fragment shaders *and* vertex shaders. In a nutshell, a vertex shader takes care of managing the vertices position, color, etc. which are then passed as “fragments” to the fragment shader for rasterization. “OMG, this is so abstract!!”. Yeah, it is less abstract than it seems, but nevertheless it requires some know how. As previously, I really suggest this : I find myself going back and forth to it regularly, always learning new things.
Good, so, what’s the plan? The main idea in the following code is to use a PShape object to encode all the vertices: we basically are making a star shaped thing out of rectangles, which in 3d graphics parlance are referred to as “quads”. Once we have created such a PShape object, we will not have to deal with the position of vertices anymore: all the change in the geometry will be dealt by the GPU! Why is this exciting? It’s because the GPU is much much faster at doing such things than the CPU. This allows in particular for real-time reactive fun. Indeed, the code gets input from the microphone and the webcam, separately. More precisely, each frame coming from the webcam is passed to the shader to be used as a texture for each quad. On the other hand, the microphone audio is monitored, and its amplitude controls the variable t, which in turns control the rotation (in Processing) and more importantly the jittering in the vertex shader. Notice that the fragment shader doesn’t do anything out of the ordinary here, just apply a texture.
Here’s how the code looks like

import processing.video.*;
import processing.sound.*;

Amplitude amp;
AudioIn in;



PImage  back;
PShape mesh;
PShader shad;

float t = 0;
float omega = 0;
float rot = 0;
int count = 0;

Capture cam;


void setup() {
  size(1000, 1000, P3D);
  background(0);
 
  //Set up audio

  amp = new Amplitude(this);
  in = new AudioIn(this, 0);
  in.start();
  amp.input(in);

  //Set up webcam

  String[] cameras = Capture.list();

  cam = new Capture(this, cameras[0]);

  cam.start();

  textureMode(NORMAL);  

  mesh = createShape();
  shad = loadShader("Frag.glsl", "Vert.glsl");

  back = loadImage("back.jpg");


  //Generates the mesh;

  mesh.beginShape(QUADS);
  mesh.noStroke();

  for (int i = 0; i < 100; i++) {
    float phi = random(0, 2 * PI);
    float theta = random(0, PI);
    float radius = random(200, 400);
    PVector pos = new PVector( radius * sin(theta) * cos(phi), radius * sin(theta) * sin(phi), radius * cos(theta));
    float u = random(0.5, 1);

    //Set up the vertices of the quad with texture coordinates;

    mesh.vertex(pos.x, pos.y, pos.z, 0, 0);
    mesh.vertex(pos.x + 10, pos.y + 10, pos.z, 0, u);
    mesh.vertex(-pos.x, -pos.y, -pos.z, u, u);
    mesh.vertex(-pos.x - 10, -pos.y - 10, -pos.z, 0, u);
  }

  mesh.endShape();
}

void draw() {

    background(0);
    //Checks camera availability;

    if (cam.available() == true) {
      cam.read();
    }
 

    image(back, 0, 0); //Set a gradient background;

    pushMatrix();
    translate(width/2, height/2, 0);
    rotateX( rot * 10 * PI/2);
    rotateY( rot * 11 * PI/2);

    shad.set("time", exp(t) - 1); //Calls the shader, and passes the variable t;

    shader(shad);
    mesh.setTexture(cam); //Use the camera frame as a texture;
    shape(mesh);

    popMatrix();

    t += (amp.analyze() - t) * 0.05; //Smoothens the variable t;

    omega +=  (t  - omega) * 0.01; //Makes the rotation acceleration depend on t;

    rot += omega * 0.01;

    resetShader(); //Reset shader to display the background image;
   
}

// Frag.glsl

varying vec4 vertColor;
varying vec4 vertTexCoord;


uniform float time;
uniform sampler2D texture;

void main(){

gl_FragColor = texture2D(texture, vertTexCoord.st ) * vertColor;

}

// Vert.glsl

uniform mat4 transform;
uniform mat4 modelview;
uniform mat4 texMatrix;


attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

varying vec4 vertColor;
varying vec4 vertTexCoord;
varying vec4 pos;


uniform float time;


void main() {
  gl_Position = transform * position;

  gl_Position.x += sin(time * 2 * 3.145 * gl_Position.x) * 10 ;
  gl_Position.y += cos(time * 2 * 3.145 * gl_Position.y) * 10 ;

  vertColor = color;

  vertTexCoord = texMatrix * vec4(texCoord, 1.0, 1.0);


}

Notice the call to reset the shader, which allows to show a gradient background, loaded as an image, without it being affected by the shader program.
Here’s a render of it, recorded while making some continuous noise, a.k.a. singing.

Try it while listening to some music, it’s really fun!

Worlds

posted by on 2018.03.18, under Processing
18:

Yesterday I have been to the beautiful exhibition by Pe Lang at the Museum of Digital Art here in Zurich. The exhibition consists of several kinetic systems producing complex behaviours. I was in particular fascinated by a piece called “polarization”, where different disks with polarized filters provide very interesting visual patterns. Those who read this blog know that I am really into systems, and their emergent features, so I was inspired to make the following piece, called “Worlds”. It is also an excuse to show how object oriented programming allows very quickly to replicate a little “cosmos” over and over.
The idea is the following. We have discussed more than once systems of particles which bounce on the canvas, but we never gave the canvas its own ontological properties, a fancy way to say that we never considered the canvas to be an object itself. That’s precisely what is going on in the code below. Namely, there is a class World whose scope is to be the box in which the particles are bound to reside. It comes with a position vector for its center, with a (half) length for the box itself, and with a particle system. The bounce check is done internally to the class World, in the update() function, so to make it behave like its own little universe. Once you have such a gadget, it’s immediate to replicate it over and over again! I disposed the box in a simple array, and I really like the visual effect that comes from it. I also did something else: inspired by statistical mechanics, each box has a “temperature”, which is influenced by how often the particles bounce on the walls of the box. The “hotter” the box, the more red it becomes. There is also a cooling factor: each box tends to cool down. So, after some time, the system goes to equilibrium, and each box stabilizes on a shade of red. This shows also something very nice, and at first counter-intuitive: there are boxes with a lot of particles, which are though very slow, making the box very “cold”.
Here is the code

// Worlds
// Kimri 2018

ArrayList<World> boxes;
int n = 10;



void setup(){
  size(1000, 1000);
 init();
 
 frameRate(30);
 

 
}


void draw(){
  background(255);

  for (int i = 0; i < boxes.size(); i++){
   World w = boxes.get(i);
  w.display();
  w.update();
  }
 
 
}

void init(){
 
  background(255);
 
 boxes = new ArrayList<World>();
 
 float step = width/n;
//Generate the array of boxes;

  for (float x = step; x < width; x+= step){
    for (float y = step; y < height; y+= step){
      boxes.add(new World(x, y, step * 0.4));
    }
  }

}

void keyPressed(){
  init();
}

// World class


class World {
  PVector pos;
  int num;
  float len;
  float temp = 255;
  float coeff = 1.7;

  ArrayList<Particle> particles;

  World(float _x, float _y, float _len) {
    pos = new PVector(_x, _y);
    len = _len;
    num = int (random(10, 60));
    //Add particles to the box
    particles = new ArrayList<Particle>();

    for (int i = 0; i < num; i++) {
      float part_x = pos.x + random(-len, len);
      float part_y = pos.y + random(-len, len);
      particles.add(new Particle(new PVector(part_x, part_y)));
    }
  }

  World(float _x, float _y, float _len, int _num) {
    pos = new PVector(_x, _y);
    len = _len;
    num = _num;
    //Add particles to the box
    particles = new ArrayList<Particle>();

    for (int i = 0; i < num; i++) {
      float part_x = pos.x + random(-len, len);
      float part_y = pos.y + random(-len, len);
      particles.add(new Particle(new PVector(part_x, part_y)));
    }
  }

  void display() {
    fill(255, temp, temp, 90);

    stroke(0, 100);
    strokeWeight(1.2);
    rectMode(CENTER);
    rect(pos.x, pos.y, 2 * len, 2 * len);
  }

  void update() {
    for (int i = 0; i < num; i++) {
      Particle p = particles.get(i);
      p.move();

      if ( (p.pos.x - pos.x) >= len - p.rad) {
        p.pos.x = pos.x + len - p.rad;
        p.vel.x  = -p.vel.x;
        temp -= 1;
      }
      if ( (p.pos.x - pos.x) <= -(len - p.rad)) {
        p.pos.x = pos.x - (len - p.rad);
        p.vel.x  = -p.vel.x;
        temp -= 1;
      }
      if ( (p.pos.y - pos.y) >= len - p.rad) {
        p.pos.y = pos.y + len - p.rad;
        p.vel.y  = -p.vel.y;
        temp -= 1;
      }
      if ( (p.pos.y - pos.y) <= -(len - p.rad)) {
        p.pos.y = pos.y - (len - p.rad);
        p.vel.y  = -p.vel.y;
        temp -= 1;
      }
      p.display();
    }
    if (temp < 0) temp = 0;
    temp += coeff;
  }
}

//Particle class



class Particle {
  PVector pos;
  PVector vel;
  float rad = 2;

  Particle(PVector _pos) {
    pos = new PVector(_pos.x, _pos.y);
    vel = new PVector(random(-3, 3), random(-3, 3));
  }

  void move() {
    pos.add(vel);
  }

  void display() {
    noStroke();
    fill(0, 100);
    ellipse(pos.x, pos.y, 2 * rad, 2 *rad);
  }

}

And here is how it looks like

Glitch Art and Shaders

posted by on 2017.02.11, under Processing
11:

It’s been a while since the last post. I have been busy with (finally!) starting to set up a website to collect some of my works, and I’ve been more or less finishing a couple of interactive installations. For this reason, interactivity and real-time processing have captured my attention recently. It turns out that when you want to interact with a piece of code which produces graphics, and as soon as what you are doing involves more than just a couple of pairs of colored circles, you run quickly into performance issues. So, unless you are one of those digital artists drawing a blinking white circle in the middle of the screen and call it art (it’s fine, don’t worry, go on with it), you need to find your way around these types of issues. In practice, this amounts to get comfortable with words like Vertex Buffer Object, C++, and shaders, to which this post is dedicated.
The story goes like this: modern graphic cards (GPU) have a language they use, called GLSL . For instance, when in Processing you draw a line or a circle, what is actually happening behind the curtains is a communication between the CPU and the graphic card: Processing informs the GPU about the vertices of the line, the fact that it has to be line, the color of the vertices, etc. There are several stages from when the vertices are comunicated to the final result that you see on your screen. Some of these stages are user programmable, and the little programs that take care of each of these stages are called “shaders”. Shaders are notoriously difficult to work with: you have to program them in C, basically, and they are quite unforgiving with respect to errors in the code. On the other hand, they are really really fast. If you want to know why it is so, and how a (fragment) shader operates, give a look here.
So, why the hell would you want to learn such a tool? Well, if you, like me, are fond of glitch art, you must have realized that interactive real-time glitch art is almost impossible if you try to work pixel by pixel: even at a resolution of 800×600, the amount of computations for the CPU to get a framerate of 30fps is impractical. Enter fragment shaders! If you delegate the work to the GPU, it becomes more than doable.
I can’t go into the detail of the code I present in the following, but there are very good tutorials on the web that slowly teach you how to tame shaders. In particular, give a look here. Rest assured: you really need to be programming friendly, and have a lot of patience to work with shaders!

PImage img;
PShader glitch;


void setup(){
  size(800, 600, P2D);
  background(0);
  img = loadImage(insert_link_to_image);
  img.resize(800, 600);
 
 
  glitch = loadShader("glitchFrag.glsl");
  glitch.set("iResolution", new PVector(800., 600., 0.0) );
 
}

 
}

void draw(){
 
  glitch.set("iGlobalTime", random(0, 60.0));
 
   if (random(0.0, 1.0) < 0.4){
  shader(glitch);
   }
 
  image(img, 0, 0);
 
  resetShader();
 
}

---------------

// glitchFrag.glsl

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif


#define PROCESSING_TEXTURE_SHADER

varying vec4 vertTexCoord;
uniform sampler2D texture;
uniform vec3      iResolution;          
uniform float     iGlobalTime;          



float rand(vec2 co){
    return fract(cos(dot(co.xy ,vec2(12.9898,78.233))) * 43758.5453);
}


void main(){
   vec3 uv = vec3(0.0);
   vec2 uv2 = vec2(0.0);
   vec2 nuv = gl_FragCoord.xy / iResolution.xy;
   vec3 texColor = vec3(0.0);

   if (rand(vec2(iGlobalTime)) < 0.7){
    texColor = texture2D(texture, vertTexCoord.st).rgb;
}
 else{
   texColor = texture2D(texture, nuv * vec2(rand(vec2(iGlobalTime)), rand(vec2(iGlobalTime * 0.99)))).rgb;
}
       
    float r = rand(vec2(iGlobalTime * 0.001));
    float r2 = rand(vec2(iGlobalTime * 0.1));
    if (nuv.y > rand(vec2(r2)) && nuv.y < r2 + rand(vec2(0.05 * iGlobalTime))){
    if (r < rand(vec2(iGlobalTime * 0.01))){
       
   if ((texColor.b + texColor.g + texColor.b)/3.0 < r * rand(vec2(0.4, 0.5)) * 2.0){
       
        uv.r -= sin(nuv.x * r * 0.1 * iGlobalTime ) * r * 7000.;
        uv.g += sin(vertTexCoord.y * vertTexCoord.x/2 * 0.006 * iGlobalTime) * r * 10 *rand(vec2(iGlobalTime * 0.1)) ;
        uv.b -= sin(nuv.y * nuv.x * 0.5 * iGlobalTime) * sin(nuv.y * nuv.x * 0.1) * r *  20. ;
        uv2 += vec2(sin(nuv.x * r * 0.1 * iGlobalTime ) * r );
   
}
       
    }
}

  texColor = texture2D(texture, vertTexCoord.st + uv2).rgb;
  texColor += uv;
   gl_FragColor = vec4(texColor, 1.0);  
   
}

In the following, you can see the result applied to a famous painting by Caravaggio (yes, I love Caravaggio): it matches real time framerate.
If you want to apply the shader to the webcam, you just need to set up a Capture object, called, say, cam, and substitute img with cam in the Processing code. Enjoy some glitching! :)

Glitch Shader from kimri on Vimeo.

n-grams, Markov chains and text generators

posted by on 2016.11.03, under Processing, Supercollider
03:

An n-gram is a contiguous sequence of n elements of a text, where each element can be a phoneme, a syllable, a character, a word, etc. Given a text, the collection of its n-grams allows to infer some statistical correlation, and moreover to assemble the n-grams collected into a Markov chain . Right, what’s a Markov chain, then? A Markov chain describes a process in which the next step depends probabilistically only on the current step. For instance, a random walk is an example of a Markovian process. One way to assign a Markov chain to a text is to collect all its n-grams, and for each n-gram we keep track of the next n-gram. We then go through the collection of all the n-grams, and for each of them we choose randomly among the list of the subsequent n-grams. We form then the new n-gram, and proceed. Confusing? Let’s see an example. Suppose we have the text “this is a book and this is my pen”, and suppose we are interested in 2-grams, where a single gram is a word. We have then then the pair (This, is), the pair (is, a), etc. Then, we keep track of all the 1-grams which follow a single 2-gram: for instance, after (this, is) we can have (a) or (my), and we assign to each of them an equal probability. Suppose we start from (This, is): if we happen to choose (a), we form the pair (is, a), to which it must follow (a, book), and so on, until we reach the end of the text. In this way, we can generate a text which has similar statistical distribution of n-grams, in this case pairs of words. The greater n, the closer your generated text will be to the original one.
Inspired by this, I have written a code in p5.js, a set of libraries for Javascript, that generates text starting from n-grams in words. Here “word” only means “groups of characters separated by a whitespace”. Punctuation is not considered from the grammatical point of view, nor the role of articles, nouns, etc. are analysed; neverthless, the results are still quite interesting. The choice of Javascript is dictated by the fact that Processing/Java is not very friendly with dictionaries, and in this case they are really useful.
Here’s the code

var words;
var ngrams = {}
var order = 2;
var txt = "";
var a = 0.1;
var target = 255;

function setup() {
  createCanvas(600, 600);

  words = source.split(" ");

  for (var i = 0; i < words.length - order; i++){
    gram_temp = [];
    for (var j = 0; j < order; j++){
    gram_temp.push(words[i + j]);
      }
      gram = join(gram_temp," ");
      if (!ngrams[gram]){
     ngrams[gram] = [];
    }
      if (i < words.length - order){
     ngrams[gram].push(words[i + order])   
  }
}
  markovIt(ngrams);
  txt = spText(txt);
}

function draw() {
    background(255);
    a += (target - a) * 0.1;
    textSize(12);
        fill(0, a);
    textDisplay(txt);
    if (a  < 0.099 ){
     restart();
     }
}

function restart(){
    markovIt(ngrams);
    txt = spText(txt);
    a = 0;
    target = 255;
}


function textDisplay(ttext){
    textAlign(CENTER);
    text(ttext, 100, 60, width - 100, height - 60);
}

function spText(txt){
    return txt.split(".");
}

function mousePressed(){
   target = 0;
}

function markovIt(ngrams) {
    var index = int(random(0, words.length - order + 1));
    curr_temp = [];
    for (var j = 0; j < order; j++){
     curr_temp.push(words[index + j]);
      }
      current = join(curr_temp, " ");
      result = current;
      if (!ngrams[current]){
        return null;
    }
      var range = int(random(30, 500));
      for (var i = 0; i < range; i++){
        if (!ngrams[current]){
        break;
          }
        possibilities = ngrams[current];
        if (possibilities.length == 0){
          break;
        }
        next = possibilities[int(random(0, possibilities.length))];
        result = result + " " + next;
        tokens = result.split(" ");
        curr_temp = [];
        for (var j = order; j > 0; j--){
        curr_temp.push(tokens[tokens.length - j]);
    }
         current = join(curr_temp, " ");
         }
    txt = result;
   
}

Notice that you need to declare a variable source, which should contain the input text.
As a little homage to Noam Chomsky, a pioneer of grammar studies (and much more), here you can find a working version of the code above using 2-grams in words, and based on this. Click on the canvas to generate some new text.

Non-Real Time Analysis and Rendering

posted by on 2016.08.01, under Processing
01:

This is a “utility” post dealing with Non-Real Time analysis, which I hope can help someone who has struggled with this issues before.
Here’s the setting. You have a nice sketch in Processing which reacts to an external input, say music or mouse position, and want to render it to show the world how fun and joyful the time spent while coding might be. You then realize that using the function saveFrame() creates problems: each single frame takes too long to save to disk, and everything goes horribly out of sync. In this case it is convenient to have a sketch that retrieves the data needed frame by frame, say the frequency spectrum of a piece of audio. One can later load this data, and render it via saveFrame(), knowing that when the frames are reassembled at the prescribed framerate, everything will be in sync.
The following code does exactly that. It uses a class called Saver, which in this case keeps track of the frequency spectrum. Under the more than reasonable assumption that the Fast Fourier Transform done by the Minim library is computed in less than 1/30 of seconds, the data you’ll retrieve for each frame will be in sync with the audio. Then you can go to your sketch which visualizes this data, load the value saved in the .txt file, and use it anywhere you would use the array of frequencies, say. It should be immediate to adapt the piece of code to your need. To save to disk you have to press any key

import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioPlayer song;
FFT fft;

String songPath = "InsertPathToSongHere";

Saver mySaver;

boolean saved = false;
boolean pressed = false;

void setup() {
  size(200, 200);
  background(0);

  minim = new Minim(this);
  song = minim.loadFile(songPath, 1024);

  frameRate(30);

  song.play();

  fft = new FFT(song.bufferSize(), song.sampleRate());
  mySaver = new Saver(fft.specSize(), "data/analysis.txt");
 
}

void draw() {
  background(0);

  fft.forward(song.mix);

  for (int i = 0; i < fft.specSize(); i++) {
   float a = fft.getBand(i);
   mySaver.setElement(a);
  }

  mySaver.update();

  if (pressed & !saved) {
    mySaver.write();
    saved = true;
  }
}

void keyPressed() {
  pressed = true;
}

//Define the Saver class

class Saver {

  int buffSize;
  String pathTosave;
  ArrayList data;
  int arraylength;

  Saver(int _buffSize, String _pathTosave) {

    buffSize = _buffSize;
    pathTosave = _pathTosave;
    arraylength = 0;  
    data = new ArrayList();
 
  }

  void setElement(float a) {
    data.add(a);
  }

  void update() {
    arraylength++;
  }

  void write() {
    String[] dataString = new String[arraylength];
    int index;
   
    for (int i = 0; i < dataString.length; i++) {
      String temp = "";
      for (int j = 0; j < buffSize; j++) {
        index = i * buffSize + j;
        if ( j == buffSize - 1){
          temp += (float) data.get(index);
        } else {
          temp += (float) data.get(index) + " ";
        }
      }

      dataString[i] = temp;
   
    }

    saveStrings(pathTosave, dataString);
    println("Data saved!");
  }
}

This technique was inspired by something I’ve read somewhere, I will try to retrieve it and point to it here. 😉

Abstract expressionism: a generative approach

posted by on 2016.06.28, under Processing
28:

I have always been fascinated by abstract expressionism, and in particular the work of Jackson Pollock. The way paint, gravity and artistic vision play together was always for me very representative of that tension between chaos and structural patterns one often finds in art.
So, here it is a little homage to the drip-painting style of Pollock. The Processing code is not clean enough to be useful, and I don’t think I understand what it exactly does yet (yes, it happens more than often that what I code is a surprise to me!). Let me say that it incorporates many of the topics discussed in this blog: object oriented programming, noise fields, etc. I’ll update the post when I’ll get it (hopefully) cleaned up.
Meanwhile, enjoy. 😉

Textural Terrain Generation

posted by on 2016.06.22, under Processing
22:

This post is inspired by some of the stuff Daniel Shiffman has on his YouTube channel.
The idea is based on the use of meshes and textures. So, what’s a mesh and what’s a texture?
A mesh is a collection of vertices, edges and faces that are used in 3D computer graphics to model surfaces or solid objects. If you are familiar with the mathematical notion of a triangulation, you are more or less in business. Even though the faces are not necessarily triangles in general, the analogy works quite well. In Processing there is a nice and quick way to generate a triangular mesh, and it’s via beginShape()/endShape(), which is what I have used in the code below. One starts with a grid (check out some earlier posts for rants about grids), and from the collection of points of the grid Processing will then build a triangular mesh*. This is achieved via the TRIANGLE_STRIP mode: we only need to specify the vertices (though in a precise order), and they will be connected via triangular shapes. Very cool. Ok, we have a tons of little triangles which assemble in a huge square: what do we do with this? Here comes the notion of a texture map. The idea is very simple: we have an image, and we want to “glue” it to a face of the mesh. Once it is glued to such a face, the image will follow the face: for instance, if we rotate such a face, the image which is stuck to that face rotates as well! Now, you should know that mapping textures on a complicated surface is kind of an art, but in our case it is pretty easy, since the surface is just a flat square. To achieve this gluing, we have to define some anchor points in the image. In other words, we have to give information about how points in the original image are associated to vertices in the mesh. The double loop in the code below does exactly this: the last two parameters in the vertex() function specify indeed the gluing.
If we had halted our imagination here, we would end up with something very static: an image attached to a square. Meh. Here comes the simple, but interesting idea. Since we are in 3D geometry, we can modify the z-coordinate of the vertex at point (x,y) with a function of the plane. In this case, the function used is Perlin noise. If we rotate our system of coordinates with respect to the x-axis, you start seeing little mountains appear. Nice! Still, though, there is no movement in place. To achieve such a movement effect, we can increment the y coordinate slightly at each frame, so that the new z-coordinate at point (x, y) will be the value of the z-coordinate at a previous point in the grid, achieving the moving effect. In the code, I’ve furthermore decided to control the offset of the z-coordinate with a variable r whose minimum value is 0, and gets triggered randomly. Notice that I’ve also allowed some “release” time for r, so to achieve some smoothness. In this way you obtain a nice beating feeling. Instead of doing it randomly, what happens if you trigger r via a beat detector listening to some music (using the Sound or Minim library, say)? Yep, you get a nice music visualizer. :)
Last couple of things I added is to “move along the image”, and using tint() instead of clearing the screen. The first one is achieved via the variable speed: basically, at each new frame, we don’t glue the image to the mesh in the exact same way, but we translate it a bit in the y-direction.
Oh, I’ve also used more than one image, to get a change in the color palette.
Here’s the code

int w = 1900;
int h = 800;
int cols;
int rows;
int scl = 30;

PImage[] img = new PImage[3];
PImage buff;

float speed = 0.01;
int speed2 = 0;
float r = 0;


void setup() {
  size(1200, 720, P3D);
  background(0);

  //Load the images we are using as textures;
  img[0] = loadImage("path_to_image1");
  img[1] = loadImage("path_to_image2");
  img[2] = loadImage("path_to_image3");

  for (int i = 0; i < img.length; i++) {
    img[i].resize(1900, 2000);
    img[i].filter(BLUR, 0.6);
  }
  buff =img[0];

  noStroke();

  cols = w / scl;
  rows = h / scl;
}


void draw() {

//Triggers a "beat";
  if ( (random(0.0, 1.0) < 0.05) & (r < 0.1) ) {
    r = 8.0;
  }

  //This allows for some release time;

  if (r > 0.01) {
    r *= 0.95;
  }


//From time to time, choose another texture image;
    if (random(0.0, 1.0) < 0.008) {
    int i = int(random(0, 3));
    buff = img[i];
  }


  float yoff = speed;
  speed -= 0.03;
  if (frameCount%2 == 0) {
    speed2 += 1;
  }
  speed2 = speed2 % 60;

  translate(width/2, height/8);
  rotateX(PI/3);
  translate(-w/2 + sin(frameCount * 0.003)* 20, -h/2, -450); //The sin function allows for some left/right shift

  //Building the mesh

  for (int y = 0; y < rows; y++) {
    beginShape(TRIANGLE_STRIP);
    tint(255, 14);
    texture(buff); //Using the chosen image as a texture;
    float xoff = 0;
    for (int x = 0; x < cols + 1; x++) {
      vertex(x * scl, y * scl, map(noise(xoff, yoff), 0, 1.0, -60, 60) * (r + 2.9), x * scl, (y + speed2) % 2000 * scl);
      vertex(x * scl, (y + 1) * scl, map(noise(xoff, yoff + 0.1), 0, 1.0, -60, 60) * (r + 2.9), x * scl, ((y + 1 + speed2) % 2000) * scl);
      xoff += 0.1;
    }
    endShape();
    yoff += 0.1;
  }
}

Here’s it how it looks like

Very floaty and cloud-like, no? 😉

Exercise 1: instead of loading images, use the frames of a video as textures.
Exercise 2: instead of controlling the z-coordinate of a point (x,y) via a Perlin noise function, use the brightness at (x,y) of the texture image obtained in Exercise 1.
Exercise 3: Enjoy. :)

*One should really think of a mesh as an object which stores information about vertices, edges, etc., while we are here concerned only with displaying a deformed grid.

Mondrian and recursion

posted by on 2016.05.03, under Processing
03:

As I mentioned more than once in this blog, one shouldn’t abuse object oriented programming. There is still a lot of beautiful art generated with a few lines of code. I hope the following Processing sketch, inspired by the geometric minimalism of Piet Mondrian, falls in this category

color[] cols;
int num = 5;

void setup() {
  size(600, 800);
  background(0);
  cols = new color[num];

  for (int i = 0; i < num; i++) {
    cols[i] = color(int(random(0, 255)), int(random(0, 255)), int(random(0, 255)));
  }

  strokeWeight(1.5);
  for (int i = 0; i < 10; i++) {
   // rotate(i * 0.1 * 2 * PI);
    Mondrian(0, width, height);
  }
}

void draw() {

}

void mousePressed() {
  fill(255, 255);
  rect(0, 0, width, height);

  for (int i = 0; i < 10; i++) {
    for (int j = 0; j < num; j++) {
      cols[j] = color(int(random(0, 255)), int(random(0, 255)), int(random(0, 255)));
    }
   
//    rotate(i * 0.1 * 2 * PI);
    Mondrian(0, width, height);
   
  }
}


void Mondrian(float x, float len1, float len2) {
  if (len1 > 20 & len2 > 20) {
    pushMatrix();
    translate(x, 0);
    color c1 = cols[int(random(0, cols.length))];

    if (random(0.0, 1.0) < 0.5) {
      float x_new = random(0, len1 * 0.5);
      fill(c1, 255);
      rect(0, 0, x_new, len2);
      Mondrian(x_new, len1 - x_new, len2);
    } else {
      float len_new = random(0, len2);
      fill(c1, 255);
      rect(0, 0, len1, len_new);
      Mondrian(0, len1, len_new);
    }
    popMatrix();
  }
}

It is essentially based on recursion, and exploits again the idea of a grid, even if in a conceptually different way from the latest post.
If you run the code, you should get something like this

1

I suggest you uncomment the line which contains the rotate call. :)
Also, here’s an exercise: modify the function Mondrian() in such a way that the same color is never chosen in successive iteration.
Remark: In his neoplastic paintings, Mondrian used mostly primary colors and white, with white being usually predominant. Moreover, he used thicker lines. Try to modify the code above to achieve the same effect.

Grids: an object oriented approach

posted by on 2016.04.07, under Processing
07:

I have been working recently on a small audio-visual installation based essentially on grid manipulations, inspired by the art of the amazing Casey Reas.
Grids are some of the first interesting repetitive structures one learns to build. For instance, a one parameter function which displays a regular grid of rectangles maximising the surface of the screen used might look like this

void grid(int n){
float stepx = width/n;
float stepy = height/n;

for (int x = 0; x < width; x+=stepx){
   for (int y = 0; y < height; y+=stepy){
    rectMode(CENTER);
    noStroke();
    fill(255);
    rect(x + stepx/2.0, y + stepy/2.0, 20, 20);
    }
  }
}

It will produce a grid of equally spaced white rectangles (so, better set the background to black before in your code). This is all fun and cosy for a couple of milliseconds, but if you are like me, you will look immediately for possibilities of explorations, and honestly the bit of code above doesn’t offer much. Let’s be precise: it *can* offer a lot of ways of tweaking, but they become cumbersome very very quickly. Also, I like to write codes which are conceptually structural and modular, and which allow to separate more clearly the “data holding” part from its representation. I find that this helps a lot the sense of surprise I get from playing with my newly created tool/toy. For instance, all that white is boring: how about we assign a different color to each rectangle? Not randomly though, which would be very cheap. Let’s use an image as a palette, and have each rectangle carry the color of the image pixel at the center of the rectangle. This can be attained by introducing the following line

fill(img.pixels[x + y * width]);

where img is the PImage variable holding your image (which I assume you have resized to the screen width and height, unless you have a fetish for ArrayOutOfBound messages :) ). Nice. But I guess you still can’t see my point, of course. Okay, let’s do the same but with a video. Now the image is changing at each frame, so the function grid() must be called in the animation loop, i.e. inside draw(). Here you can see my point: by proceding in this way, you are doing a certain amount of redundant computations. Indeed, the only thing that you would like to modify in this case is how each node of the grid is represented, and *not* the grid structure itself. So, it helps then to think of a grid as a data holder for the positions of its nodes, which, once we require some extra properties, are directly determined by a single integer n, the number of nodes per row and column. What we draw on the screen is then a representation of this bunch of information. Anytime you have something that behaves like a collection of data, object oriented programming (OOP) is not far from sight. Building objects (or rather classes) is kind of an art: extracting the relevant properties we want in our objects so that further manipulation becomes reactive and enjoyable is not an easy task. This to say that the are some basic rules in object oriented programming, but for artistic reason we can (and should) ignore many standard design patterns, and look at each case separately. Since I like thinking in terms of objects when programming, I feel the following warning is due: do *not* overuse OOP! There are fantastic pieces of algorithmic/generative art which do not use objects at all. The point is that you might run the risk to decompose the problem in its atomic parts, which has a certain intellectual appeal, no doubt, but then get lost when it comes to exploration and tweaking. So, case by case, see what works best, and, most importantly, what puts you in that spot where you can still get surprised. This said, let’s move on.
So, we can model a grid with the following Grid class

class Grid {
  int n;
  float[] posx;
  float[] posy;
  float offsetx;
  float offsety;
  Shape[] shapes = {};
 
  Grid(int _n){
    n = _n;
    int stepx = width/n;
    int stepy = height/n;
    offsetx = stepx/2.0;
    offsety = stepy/2.0;
    posx = new float[n];
    posy = new float[n];
   
    for (int i = 0; i < n; i++){
      posx[i] = i * stepx;
      posy[i] = i * stepy;
    }

    for (int i = 0; i < n; i++){
        for (int j = 0; j < n; j++){
        shapes = (Shape[]) append(shapes,new Shape(posx[i], posy[j]));
        }
      }

    }
   
    Grid(int _n, float _offsetx, float _offsety){
    n = _n;
    int stepx = width/n;
    int stepy = height/n;
    offsetx = _offsetx;
    offsety = _offsety;
    posx = new float[n];
    posy = new float[n];
   
    for (int i = 0; i < n; i++){
      posx[i] = i * stepx;
      posy[i] = i * stepy;
    }

    for (int i = 0; i < n; i++){
        for (int j = 0; j < n; j++){
        shapes = (Shape[]) append(shapes,new Shape(posx[i], posy[j]));
        }
      }

    }
   
    Grid(int _n, float len, float _offsetx, float _offsety){
    n = _n;
    int stepx = int(len/n);
    int stepy = int(len/n);
    offsetx = _offsetx;
    offsety = stepy;
    posx = new float[n];
    posy = new float[n];
   
    for (int i = 0; i < n; i++){
      posx[i] = i * stepx;
      posy[i] = i * stepy;
    }

    for (int i = 0; i < n; i++){
        for (int j = 0; j < n; j++){
        shapes = (Shape[]) append(shapes,new Shape(posx[i], posy[j]));
        }
      }

    }  
   
    void display(){
      pushMatrix();
      translate(offsetx, offsety);

      for (int i = 0; i < shapes.length; i++){
        shapes[i].display(20);
      }

      popMatrix();
    }
   
    void display(PImage _img){
      _img.loadPixels();
      pushMatrix();
      translate(offsetx, offsety);

      for (int i = 0; i < shapes.length; i++){
        shapes[i].setColor(_img.pixels[int(shapes[i].x) + int(shapes[i].y) * _img.width]);
        shapes[i].display(20);
      }

      popMatrix();
    }
   
  }

So, you can see that I have heavily used that you can overload constructors and methods, so to give some default behaviours when we don’t want to bother passing a lot of parameters. An instance of Grid will also carry an array of Shape objects: this is the “representational” informations of our grid. Here’s how the Shape class looks like:

class Shape {
  float x, y;
  color c;
 
  Shape(float _x, float _y){
    x = _x;
    y = _y;
    c = color(255, 255, 255);
  }
 
  Shape(float _x, float _y, color _c){
    x = _x;
    y = _y;
    c = _c;
  }
 
  void display(float w){
    rectMode(CENTER);
    noStroke();
    fill(c, 255);
    rect(x, y, w , w);
    rectMode(CORNER);
  }
 
  void setColor(color _c){
    c = _c;
  }
 
}

If you don’t specify any color, the rectangle will be white.
So, now the previous function grid() is subsumed in the following lines of code

Grid grid = new Grid(20);
grid.display(img);

Apart from the elegance of it, that as I mentioned before should not be the only criterion of judgement, the code now is amenable to vast explorations. For instance, by modifying the display() method of the Shape class we can completely change the appearance of our grid, even so much that it doesn’t look like a grid anymore! Moreover, we don’t have all those redundant computations to be made.
One thing that comes to mind when you have a class is that you can produce many objects from that class with slightly different properties. In this case I have decided to use this expedient in order to explore “fractalization”, which is another (made up?) word for “apply recursion carefully”. Add the following function to your code

void fractalize(int n){
  if (n > 1){
  grid = new Grid(n, x, y);
  grid.display();
  x+= grid.offsetx;
  y+= grid.offsety;
  fractalize(n - 1);
  }
}

To make the most of it, you might want to lower the alpha of the filling for the rectangles. Actually, as I mentioned before, we can start exploring the code and the questions it naturally suggests, like: “why restrict ourselves to rectangles?” Or “why even filling them?” “Can I add some stochastic noise here and there to make everything not so hearthless and rigid?” “What is the meaning of life?”
When you start raising these questions, a vast playground (or/and a deep existential pit) opens up.
Believe it or not, after very few adjustments to the Shape class here’s what I got

1

1

which I find having an interesting balance between structure and randomness. A nice surprise! :)

Noise, flows and generative art

posted by on 2016.03.26, under Processing
26:

What is and what isn’t generative art is a long standing debate, in which I do not want to enter here. Just to put things in context, though, I’ll share some words about it. For some people, a piece of art is generative if it is the product of some sort of system which, once set in motion, is left to itself, with no further interaction. These systems are also called autonomous. Though generative art is usually associated to algorithmic art, i.e. art generated by some computer algorithm, autonomous systems can be found in biology, chemistry, mechanics, etc. Personally, I find the constraint on the autonomy of the system a bit too tight. While I do think that autonomous systems produce fascinating pieces of art, hence showing the beauty of complexity and emergency, I’m also very much interested in the dichotomy creator/tool, which in this case manifests itself as a shadow of the interaction between human and machine. I’m thinking about art which is computer assisted, more specifically which arises from the interaction of some sort of (algorithmic) system with the creator/spectator. This interaction poses interesting questions. We can indeed consider the total generative system as the combination autonomous system + spectator. This would be an autonomous system itself, if not for one little detail: the spectator is aware (whatever that means) that he’s a tool of a machine which is producing a piece of art. A more concrete example would be the following. Consider a live art installation in which the movements of spectators are used to control some given parameters of a system, which is then used to draw on a big screen, or produce sounds. There is going to be a huge conceptual difference if the spectators are aware of the tracking of their movements or not. In the second case, we are in the presence of something which looks like an autonomous system, while in the first case the spectators could use their awareness to drive the artistic outcome. The topic is incredibly fascinating and worth thinking about: these few words were only meant as a support to the fact that I would consider as generative art* the piece that you are going to see in the following.
Since discussions surrounding art are not notoriously controversial enough, I’ll move to noise and randomness (yeeeih!). Let’s start with saying: you can’t generate random numbers with a computer. Behind any random number produced with a programming language, there is an algorithm: in general it is a very complicated one and takes as parameter physical parameters of your machine (say, the speed of the CPU at the moment you request a random number), but it is still an algorithm. It is just so complex that we (as humans) can’t see any logical pattern behind it. I already feel the objection coming: “Well, what is randomness anyway? Is there anything truly random?”. I’m going to skip this objection quickly, pointing here instead. Enjoy! 😉
Processing offers two functions to treat randomness and noise: one is random(), and the other is noise(). Actually, noise() is a function that reproduces Perlin noise, a type of gradient noise which is extremely useful to reproduce organic textures.
Though Perlin noise is great, no doubt about that, and since randomness for a machine is just a function which looks unpredictable, why not make one’s own noise? After all, one of the point of generative art is that it allows to build tools which can be played with and explored. The function customNoise() in the code below does exactly that: it is a function from -1 to 1 which behaves in an erratic enough way to be a good substitute for noise(). You have now got your very own noise function, well done! The question is: what are we going to do with that? That’s where the second noun in the title of this post enters the stage. Every time you have a nice function in two variables, you can build out of it a vector field. “What’s that?”, you might say. You can think of it as an assignment of a little arrow to each point of the screen, with the angle (in this case) respect to the x-axis determined by our noise function. Once we have such a little arrow, we can use it to tell a particle which is at a given position on the screen where to go next. You can imagine the vector field as being associated to a fluid which at each point moves exactly with velocity given by the value of the vector field. If you then drop tiny particles in the fluid, they will start moving along curves, which are called the flow curves of the vector field. Moreover, they will start accumulate along specific flow curves: I leave you to investigate why it is that. 😉
So, the following Processing code brings home all these ideas, plus a last one, which has to do with the beginning of this post. You will notice that the function customNoise() has a mouseX inside, and there’s a mouseY controlling the variable depth. This means that the function interacts with the mouse movement, and hence the output of the code can be driven by the user. In particular, the piece you get stays comfortably in that gray area between generative and nongenerative art, one of those interesting arguments you can entertain your friends with at the next vernissage or pub quiz you go. 😉
Here’s the code:

float[] x;
float[] y;
color[] col;
float s = 0.001;
float depth = 0.5;
PImage img;

void setup() {
  size(1000, 1000);
  background(0);
  int n = 1000;
  x = new float[n];
  y = new float[n];
  col = new color[n];
  img = loadImage(pathtoimage/image);
  img.resize(width, height);
  img.loadPixels();
  for (int i = 0; i < x.length; i++) {
    x[i]= random(0, width);
    y[i]= random(0, height);
    int loc = int(x[i]) + int(y[i])*width;
    col[i] = img.pixels[loc];
  }
}

void draw() {
  noStroke();
  depth = map(mouseY, 0, height, 0.5, 1.5);
  //fill(255, 4); //Uncomment if you don't want to use an image;
  for (int i = 0; i < x.length; i++) {
    float alpha = customNoise(x[i] * s, y[i] * s)*2*PI;
    x[i]+= depth * cos(alpha); // + random(-0.4, 0.4);
    y[i]+= depth * sin(alpha); // + random(-0.4, 0.4);
    if (y[i] > height) {
      y[i] = 0;
      x[i] = random(0, width);
    }
    x[i]= x[i]%width;
    fill(col[i], 4); //Comment if you don't want to use an image;
    ellipse(x[i], y[i], 2, 2);
  }
}


float customNoise(float x, float y) {
  return pow(sin(0.9*x + noise(x, y)*map(mouseX, 0, width, 0, 5)*y), 3);
}

You will get something like this

noise2

noise1

Notice that the first piece is obtained by commenting/uncommenting some few lines.
Finally, there is one last question you might ask, and it is the following:”How did you come up with those peculiar numbers for the parameters in the code?”. Well, the answer is: by a certain amount of trial and error. As I mentioned more than once, making art with code allows for a great degree of exploration: tweaking a parameter here and changing a line there can give you very different and unexpected results. That what you get at the end is artistically appealing or not, well, nobody else can tell but you. These highly subjective decisions are what transforms a bunch of programming lines into something meaningful and beautiful. So, go on tweaking and looking for something you find interesting and worth sharing, then! :)

*If you are post-modernly thinking “Who cares if it’s called generative or not?”, you definitely have all my sympathy.

pagetop