rss feedReactive applications, Shaders and all that
We have already discussed the advantage of using shaders to create interesting visual effects. This time we will have to deal with fragment shaders *and* vertex shaders. In a nutshell, a vertex shader takes care of managing the vertices position, color, etc. which are then passed as “fragments” to the fragment shader for rasterization. “OMG, this is so abstract!!”. Yeah, it is less abstract than it seems, but nevertheless it requires some know how. As previously, I really suggest this : I find myself going back and forth to it regularly, always learning new things.
Good, so, what’s the plan? The main idea in the following code is to use a PShape object to encode all the vertices: we basically are making a star shaped thing out of rectangles, which in 3d graphics parlance are referred to as “quads”. Once we have created such a PShape object, we will not have to deal with the position of vertices anymore: all the change in the geometry will be dealt by the GPU! Why is this exciting? It’s because the GPU is much much faster at doing such things than the CPU. This allows in particular for real-time reactive fun. Indeed, the code gets input from the microphone and the webcam, separately. More precisely, each frame coming from the webcam is passed to the shader to be used as a texture for each quad. On the other hand, the microphone audio is monitored, and its amplitude controls the variable t, which in turns control the rotation (in Processing) and more importantly the jittering in the vertex shader. Notice that the fragment shader doesn’t do anything out of the ordinary here, just apply a texture.
Here’s how the code looks like
import processing.sound.*;
Amplitude amp;
AudioIn in;
PImage back;
PShape mesh;
PShader shad;
float t = 0;
float omega = 0;
float rot = 0;
int count = 0;
Capture cam;
void setup() {
size(1000, 1000, P3D);
background(0);
//Set up audio
amp = new Amplitude(this);
in = new AudioIn(this, 0);
in.start();
amp.input(in);
//Set up webcam
String[] cameras = Capture.list();
cam = new Capture(this, cameras[0]);
cam.start();
textureMode(NORMAL);
mesh = createShape();
shad = loadShader("Frag.glsl", "Vert.glsl");
back = loadImage("back.jpg");
//Generates the mesh;
mesh.beginShape(QUADS);
mesh.noStroke();
for (int i = 0; i < 100; i++) {
float phi = random(0, 2 * PI);
float theta = random(0, PI);
float radius = random(200, 400);
PVector pos = new PVector( radius * sin(theta) * cos(phi), radius * sin(theta) * sin(phi), radius * cos(theta));
float u = random(0.5, 1);
//Set up the vertices of the quad with texture coordinates;
mesh.vertex(pos.x, pos.y, pos.z, 0, 0);
mesh.vertex(pos.x + 10, pos.y + 10, pos.z, 0, u);
mesh.vertex(-pos.x, -pos.y, -pos.z, u, u);
mesh.vertex(-pos.x - 10, -pos.y - 10, -pos.z, 0, u);
}
mesh.endShape();
}
void draw() {
background(0);
//Checks camera availability;
if (cam.available() == true) {
cam.read();
}
image(back, 0, 0); //Set a gradient background;
pushMatrix();
translate(width/2, height/2, 0);
rotateX( rot * 10 * PI/2);
rotateY( rot * 11 * PI/2);
shad.set("time", exp(t) - 1); //Calls the shader, and passes the variable t;
shader(shad);
mesh.setTexture(cam); //Use the camera frame as a texture;
shape(mesh);
popMatrix();
t += (amp.analyze() - t) * 0.05; //Smoothens the variable t;
omega += (t - omega) * 0.01; //Makes the rotation acceleration depend on t;
rot += omega * 0.01;
resetShader(); //Reset shader to display the background image;
}
// Frag.glsl
varying vec4 vertColor;
varying vec4 vertTexCoord;
uniform float time;
uniform sampler2D texture;
void main(){
gl_FragColor = texture2D(texture, vertTexCoord.st ) * vertColor;
}
// Vert.glsl
uniform mat4 transform;
uniform mat4 modelview;
uniform mat4 texMatrix;
attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;
varying vec4 vertColor;
varying vec4 vertTexCoord;
varying vec4 pos;
uniform float time;
void main() {
gl_Position = transform * position;
gl_Position.x += sin(time * 2 * 3.145 * gl_Position.x) * 10 ;
gl_Position.y += cos(time * 2 * 3.145 * gl_Position.y) * 10 ;
vertColor = color;
vertTexCoord = texMatrix * vec4(texCoord, 1.0, 1.0);
}
Notice the call to reset the shader, which allows to show a gradient background, loaded as an image, without it being affected by the shader program.
Here’s a render of it, recorded while making some continuous noise, a.k.a. singing.
Try it while listening to some music, it’s really fun!
comment
I have checked the code on my system (Processing 3.3.6 + NVIDIA GPU) and it works fine.
You might want to check your OpenGL compatibility, maybe via shadertoy.
Just to be on the safe side: the code for Vert.glsl and Frag.glsl must be in separate files (with th respective name) in a folder called “data” inside your sketch folder.
I appreciate your taking the time to check.
This range of application is beyond my current understanding of Processing — I was curious about how it would work and look — so it will take a bit of swatting up to sort myself out.
On my MacBook Pro, I’m being told there are 2 GPUs: AMD Radeon R9 M370X and, onboard, an Intel Iris Pro, presumably for the Retina display. I’m unclear as to the role, if any, OpenGL is configured to take. My only explicit encounter with GPU matters is with DaVinci Resolve which basically takes care of itself in deciding what to do. It will be an enlightening investigation.
Do you write/demo sound and music matters in Processing? I was initially drawn to your site by the Supercollider posts. But I’m on a bit of a sabbatical from SC, looking for a non-Max/MSP environment that supports good audio and visual integration.
Thanks again. …edN
I generally do not use Processing in sound related matters. I do not have any suggestion regarding an A/V environment, sorry.
Not to worry. Thanks nonetheless. I’ll continue to watch and be stimulated by your posts. ATB …edN
Please Leave a Reply
TrackBack URL :
Enjoying your posts. However, as a neophyte I’m puzzled by the “unexpected token: uniform” parsing error I’m getting at line 112. The more obvious issue to me would be a missing semi-colon on a previous line, but that’s not it. Maybe ‘uniform’ is a token in a library I don’t have loaded; but there is no other diagnositc along those lines. Help? Thanks.