Get color at pixel

I'm looking to create a color utility plugin, to provide a variety of color related expressions and conditions.

The first expressions I want to add would form a color picker that will get a color at a specified pixel coord on C2's game canvas.

[edit] (By "canvas", I mean C2's game window area, not the 3rd party "Canvas" plugin.)

It would look something like this:

pxR( x , y )

pxG( x , y )

pxB( x , y )

Ideally the picker expressions would get the color as it appears on the screen. (i.e. After all webGL effects have been applied.)

Does anyone know how best to do this from within the SDK?

Thanks in advance for any advice or suggestions.

Possibly related R0J0hound example code

I saw in another post that R0J0hound posted some javascript that could be executed from within a C2 event sheet to retrieve color at a given pixel, and this looks like what I want to do.

Granted, I'm not sure if this is the best way to do it from within the SDK, since I might be able to get access to the canvas directly, rather than looking it up in the DOM by element id.

// This is a paraphrase of R0J0hound's original code found at the link above.
var canvas = document.getElementById( 'c2canvas' );
var ctx = canvas.getContext( '2d' );
var pixel = ctx.getImageData( x , y , 1 , 1 );
pixel.data[ 0 ];") [/code:2hd9dwhd](That said, I wasn't able to get this code to work from within C2. It seemed to work up until assigning the canvas context into "ctx", which as near as I can tell always assigned null/undefined, as I could never get any data out of it, or .toString() the object. So I'm probably doing something wrong.)

Reading color of pixel in canvas plugin will return the original color in this canvas, not the final result after shader of weblg, imo.

fisholith

That should work with webgl off i'm pretty sure. Even with webgl on I think it should work. For reference you could look at the snapshot canvas action, which basically uses the same approach i think. i can''t test webgl currently.

From a plugin you should be able to access the canvas from the runtime variable i think. There is a way to grab a pixel with just webgl which I tried to use with the paster plugin when webgl was working for me but it proved unreliable.

Thanks for the reply rexrainbow,

I'm familiar with that aspect of the third party Canvas plugin. I meant the HTML5 canvas that C2 renders the game into. Apologies, I should have been more specific in my first post. I think (though I may be wrong) R0J0hound's code I linked to in the first post samples C2's main HTML5 canvas directly.

I was also thinking that since C2 can store a screencap of the canvas that includes applied shaders, would it be better to try to use an approach related to that screencap mechanism to get the post shader colors?

I've been trying to get a simple method for picking screen colors for a while and all the event-and-object approaches I've tried end up being kind of fiddly, especially when it comes to capturing post shader colors.

[edit] Oh, hey R0J0hound, I think you just posted right before I did.

I have to head out briefly, but I'll be able to reply shortly. Thanks for the info.

[edit 2] Thanks for the info, and the suggestion R0J0hound, I'll take a look at the screencap action code.

Okay, I've done some digging through the C2 js files, but I'm having trouble finding the point at which any image data is actually encountered.

C2 snapshot code

The Action in system.js calls "doCanvasSnapshot()"

[system.js , line 1913]

SysActs.prototype.SnapshotCanvas = function (format_, quality_)
{
	this.runtime.doCanvasSnapshot(format_ === 0 ? "image/png" : "image/jpeg", quality_ / 100);
}; [/code:355ixjpx]

"doCanvasSnapshot()" is called in system.js, and seems to be declared in preview.js ... maybe?
I'm not especially familiar with the setup of the HTML5 canvas in C2, or in general. So I suspect there are probably some conventions I'm not recognizing.
[preview.js , line 5043]
[code:355ixjpx]Runtime.prototype.doCanvasSnapshot = function (format_, quality_)
{
	this.snapshotCanvas = [format_, quality_];
	this.redraw = true;		// force redraw so snapshot is always taken
}; [/code:355ixjpx]

I found a spot in the tick code that seems to be involved with the snapshot process, and uses "this.snapshotCanvas" , but it seems like it's arranging for the snapshot data to be available to some other system somewhere, rather than handling image data itself.
[preview.js , line 2346]
[code:355ixjpx]// Snapshot the canvas if enabled
if (this.snapshotCanvas)
{
	if (this.canvas && this.canvas.toDataURL)
	{
		this.snapshotData = this.canvas.toDataURL(this.snapshotCanvas[0], this.snapshotCanvas[1]);
		
		if (window["cr_onSnapshot"])
			window["cr_onSnapshot"](this.snapshotData);
		
		this.trigger(cr.system_object.prototype.cnds.OnCanvasSnapshot, null);
	}
		
	this.snapshotCanvas = null;
} [/code:355ixjpx]

The only other place I'm even seeing the word snapshot come up is in an assignment to an array named "window".
I have no idea what's going on here. Is this some kind of variable function callback thing?
[preview.js , line 6061]
[code:355ixjpx]window["cr_getSnapshot"] = function (format_, quality_)
{
	var runtime = window["cr_getC2Runtime"]();
	
	if (runtime)
		runtime.doCanvasSnapshot(format_, quality_);
} [/code:355ixjpx]

[b]My thoughts so far[/b]
So I'm starting to suspect that the image data saved by the snapshot action is setup inside code much closer to the actual drawing of the main canvas, and the snapshot action just flags it to be saved to a variable for later use. I could be completely wrong though, as I'm not quite sure what I'm looking at.

All that said, I'm not sure if I really need to get at the snapshot code in the first place. I was going to take a look at it just to get an idea of how to pick a color from the final rendered scene, though I don't know if that's the direction I should be going, or if there's a better way to get at the color data.

Any suggestions are still welcome. I'm still not quite clear on how to access the canvas from within the SDK.

I found a pc to test webgl on, and it seems that you can't get both a webgl and 2d context on the same canvas. That explains why my code doesn't work.

You can still save the entire canvas. Ex:

var canvas = document.getElementById( 'c2canvas' );

canvas.toDataURL("image/png")

or

canvas.toDataURL("image/jpeg", 0.75)

That is all the runtime is doing, except it waits till after everything is drawn for that frame.

From a plugin you can get the canvas with:

this.runtime.canvas

instead of

document.getElementById( 'c2canvas' );

So the process to get the color of a pixel that also works with webgl would be

var img = new Image();
var self = this;

img.onload = function ()
{
	var canvas = document.createElement('canvas');
	canvas.width = img.width;
	canvas.height = img.height;
	var ctx = canvas.getContext('2d');
	ctx.drawImage(img,0,0);
	var pixel = ctx.getImageData( x , y , 1 , 1 );
	
	// save the value in the plugin and call a trigger to use the color in your program
	self.redvalue=pixel.data[ 0 ];
	self.runtime.trigger(cr.plugins_.Sprite.prototype.cnds.OnURLLoaded, self);
}

img["crossOrigin"] = "anonymous";
img.src = this.runtime.canvas.toDataURL("image/png")[/code:15nrlz3e]
It's referenced from the loadUrl sprite action.

Basically load toDataUrl to a image.  This is asynchronous so we need a callback when it finishes.  When it's done loading we can then create a canvas, draw the image to it and retrieve a pixel with a 2d context.  Then we save the pixel to a variable we can access from an expression and call a event sheet trigger.

Whew, that's not exactly straightforward.  When webgl is off a pixel can be retrieved in place with the code in your op.

Now to go the webgl route instead there's a nifty readpixels function so supposedly it's as simple as this:
[url=https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/readPixels]https://developer.mozilla.org/en-US/doc ... readPixels[/url]
[code:15nrlz3e]var canvas=document.getElementById('c2canvas');
var gl=canvas.c2runtime.gl;
var pixel = new Uint8Array(4);
gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
pixel[0];[/code:15nrlz3e]
which works only on frame 1 it seems. I don't know why, and my google fu hasn't been helped me yet.  But that could be a start.

Edit:
Here's the proper solution:
[url=https://asalga.wordpress.com/2011/08/01/using-webgl-readpixels-turn-on-preservedrawingbuffer/]https://asalga.wordpress.com/2011/08/01 ... ingbuffer/[/url]
But that requires modifying the runtime in preview.js.

Wow, thanks R0J0hound, that's super helpful.

Sprite loadURL code

I just looked up the Sprite's loadURL code.

I see what look like 4 parts to the code.

1. Declare some variables, including the image "img". [3 lines]

2. Assign an onload() function. [a bunch of lines]

3. An if-statement that checks stuff about the "url_". [2 lines]

4. Finally "img.src = url_;". [1 line]

So does that final assignment in part 4 "img.src = url_;" trigger the onload() function assigned in part 2? Or rather, does the completion of loading that newly assigned URL trigger it?

Because otherwise I'm not sure where the loading of the image is happening.

If that's the case, I may understand what's going on in your code. I think the assignment of that onload() function just threw me a bit, because it looked like nothing explicitly called it. So I'm assuming that again the assignment to img.src sets some behind-the-scenes stuff in motion that ultimately triggers the onload() function.

Possibly realized my in-event color picking troubles

As an added bonus, when looking through the Sprite's loadURL code, I may have figured out why my previous in-event attempts to pick colors acted so finicky. I think I was waiting till the screenshot was taken, via the triggered event, but I wasn't waiting until after I was done loading it into a sprite, via the sprite's triggered event.

I've been trying to pick the post-webgl colors out of the 3rd party Canvas plugin by saving a screenshot with the System action, loading the screenshot into a Sprite, and then pasting that screenshot Sprite into a Canvas plugin, so I can pick it's colors. I keep forgetting that every image loading step is going to be asynchronous, not just the screenshot.

Some questions about your code example

Will the "canvas.getContext('2d');" in your first code example work if webgl is enabled?

Or does that not matter because, once inside the onload() function, we're dealing with our own custom canvas for the screencap, rather than C2's game canvas?

Thoughts on readpixels

The "working only on frame one" thing might be because of how buffer clearing works when using webgl with preserveDrawingBuffer set to false, which I assume it is for C2 for performance reasons. Not sure though.

At the second link you posted one of the comments seems to indicate that it might be possible to get data from the buffer after it's drawn but before it's handed to the compositor and cleared. Granted that would require inserting code into C2's runtime I gather, which is probably not the direction I should be going. Of course I could be totally wrong about all of the above, I'm just trying to follow along.

[update]

So I set up the plugin and implemented the example code, but I'm always getting [ 0 , 0 , 0 , 255 ] as the pixel values regardless of the coordinates I use. I'm using an alert() to print the pixel directly out of the onload() function, right after the call to ctx.getImageData().

Still working on figuring out what's going on, but I'm open to any suggestions.

Do I maybe need to wait for some aspect of the C2 canvas to load before passing it to "img.src"?

In js as soon as you give a URL to img.src the image will be loaded asycronously and when it finishes it calls the function at img.onload. It's an html thing.

Yeah you'd probably want to want to wait till the image load trigger to draw it to the canvas.

It should work fine getting a 2d context because it's using the new canvas created right above it.

The preserveDrawingBuffer setting is slower when it's on which is why it typically is off. The fact readpixels worked once with that flag off probably has to do with undefined behavior since it's not supposed to work when the flag is false. As far as I can tell, you can only set that flag when first getting the context, aka when the runtime creates it. But ya it's not really acceptable for a user to have to make that change when using such a plugin. :p

If the image URL idea isn't working paste the code for that function and the trigger it calls. The events should look like this:

On click

--- get pixel at X,y

On pixel got

--- set redvar to pixel.red

At the moment I have an action that captures a pixel to a plugin instance variable.

It sets the onload() callback, and sets the img.src.

When the onload() callback runs it grabs a pixel out of the image data at the coords passed to the action function.

Speaking of which, now that I think of it, I should try printing the coords from within the onload() function to see if they are the correct values or undefined, as the action function scope is gone when onload() runs. Maybe that's the issue, and so I need to store the coords in a plugin instance variable.

That's the point at which everything I print out is (0,0,0,255).

So within the onload() I save the pixel data to a plugin instance variable, and then trigger a C2 plugin condition to indicate that the load is complete, and is ready for use.

From there I can use an expression to get the R, G, or B values. Or at least I could if they weren't all 0.

I'll post my code for the action condition and expression in a little bit, in an update.

Thanks again for all the extra info and clarification.

One thing I didn't cover was in the action right before the callback you see the line:

var self=this;

Then inside the callback self is used instead of this. The reason for that is "this" is the object instance inside the action, but when the onload function is called "this" is instead the global variable space because the function is called without a object. You can find more information by googling "JavaScript this".

Anyways that just means to just use self instead of this everywhere inside the onload function. If you used "this" instead it would explain why the instance variables aren't being changed.

Fortunately I've run into the disembodied "this" issue before, so I think I set that up correctly, but it's good to know I wasn't being paranoid. That said, I did set it up wrong the very first time. <img src="{SMILIES_PATH}/icon_e_smile.gif" alt=":)" title="Smile">

img.src always blank

Within the onload() function, I set up an alert() to print out the image URL that was being assigned to img.src. The src was always a base64 image string of a blank canvas with a size matching the game's window dimensions. For some reason, I'm just never getting anything but a blank image from:

this.runtime.canvas.toDataURL("image/png")

I was able to use this base64 image decoder to see what exactly was stored in the base64 image text.

http://codebeautify.org/base64-to-image-converter

Workaround

So with some experimenting I may have found a workaround that allows me to get legitimate pixel color values.

Rather than trying to capture the canvas inside my Action function, I'm instead passing one extra parameter to the action, an Image URL, so I can simply pass C2's System CanvasSnapshot URL variable to my Action.

This guarantees that the URL contains an actual image. And it seems to work, so far. I'm actually getting correct color data. Ultimately I would like to be able to capture the game canvas entirely from within my plugin, but I don't know if there's a way to ensure that the capture is timed and sequenced correctly so that I actually get an image.

Capturing from within my plugin?

Do you have any ideas on what might be going wrong with capturing the canvas from within my plugin? You mentioned waiting for the image load trigger. Are you talking about waiting for a trigger related to the game canvas itself? It looks like when this.runtime.canvas.toDataURL("image/png") is used from within my plugin, the URL returned is a text representation of a large blank image matching the game window size. If there is a way to wait for it to contain image data that would be cool.

My current code

Instance vars.

instanceProto.onCreate = function()
{
	// Captured pixel.
	this.pixelCap = []; // Will be filled by a getImageData() call.
	this.pixelCapX = 0; // Coord of captured pixel.
	this.pixelCapY = 0; // Coord of captured pixel.
};[/code:2lbr1k2g]

Action.
[code:2lbr1k2g]Acts.prototype.pixelCap = function ( posX , posY , dataURL )
{
	var img = new Image();
	var self = this;

	img.onload = function ()
	{
		// Make temp canvas, and draw image data into it.
		var canvas = document.createElement('canvas');
		canvas.width = img.width;
		canvas.height = img.height;
		var ctx = canvas.getContext('2d');
		ctx.drawImage(img,0,0); // Paste image starting at coord (0,0).

		// Save pixel data.
		self.pixelCap = ctx.getImageData( posX , posY , 1 , 1 ); // Save to instance var.
		self.pixelCapX = posX;
		self.pixelCapY = posY;

		// Trigger notice of completion.
		self.runtime.trigger(cr.plugins_.fi_color.prototype.cnds.onCapturedPixel, self);
	}

	img["crossOrigin"] = "anonymous";
	img.src = dataURL;
};[/code:2lbr1k2g]

Condition triggered from the onload in the Action code above.
[code:2lbr1k2g]Cnds.prototype.onCapturedPixel = function () // REMEMBER: These params are C2 event sheet params.
{
	return   true;
};[/code:2lbr1k2g]

The Expression that retrieves the Red value of the pixel.
[code:2lbr1k2g]Exps.prototype.cpR = function ( ret )
{
	ret.set_float(   this.pixelCap.data[ 0 ]   );
};[/code:2lbr1k2g]

I think the answer is in c2's runtime with it's capture action. It doesn't grab the canvas right there, instead it grabs it later (I think it's right after the next redraw).

I was fiddling with that with the paster plugin before and you have some places in the plugin that's run at different times.

Two that come to mind are the tick() and tick2() functions. I forget if there are others. One is run before the event sheet and one is done after. You do have to tell the runtime to call those functions with I think a tickme() call? Searching for tick2 probably would give a post with more info as I recall, as well as any I'm fogetting here. Anyways my guess is in those places might be better times to grab the canvas, but it's kind of test and see.

Basically the process would be to set a Boolean with your action and in the tick function it would check if the Boolean is set and then do what your action does currently.

The idea can probably be more definite, you just need to break down what the runtime is doing. Off hand I think it looks something like this? I'd have to verify where the capture is done.

Tick()

Events

Tick2()

Capture canvas

Draw

Repeat

The tick2() suggestion sounds like it could be pretty cool, but I actually just started looking through the "preview.js" file, and I'm not sure if tick2() will work. Though I could be misinterpreting what I found.

In preview.js I regex searched for

tick\w*\(

to find all the different tick types.

I found the various tick calls inside the logic() function declaration:

// Process application logic
Runtime.prototype.logic = function (cur_time) [/code:34pbx0dk]
pretick() "// Tick objects-to-pre-tick"
tick() "// Tick behaviors"
posttick() "// Call posttick on behaviors"
tick() "// Tick objects-to-tick"
run() "// If the running layout has an event sheet, run it"
tick2() "// Post-event ticking (tick2)" [Looks to tick behaviors]
tick2() "// Tick objects-to-tick2"
[i](The comments are taken from preview.js and precede the respective tick calls.)[/i]

The logic() function is called from within a [i]different[/i] runtime tick() function, not to be confused with the object tick() function above.
[code:34pbx0dk]Runtime.prototype.tick = function (background_wake, timestamp, debug_step)[/code:34pbx0dk][i](Note this is a function also named "tick" but it belongs to the runtime, where all the tick() functions called inside the logic() function belong to objects or behaviors.)[/i]

Within this runtime tick() function, logic() is called, and afterwards draw() and drawGL() are called, and after that the snapshot canvas code runs.

So I think the canvas may not get redrawn until after all of the various object tick calls are completed, and the logic() function exits. Would this matter? I'm not sure if there's a reason I wouldn't just get the canvas result from the previous drawGL call, unless it's been erased by the time the tick code runs.

[b][update][/b]
I had a weird and possibly terrible idea. Could I force the canvas to redraw from within the plugin by calling draw() or drawGL()? That seems like it could work, but also be really bad if put inside a loop in the event sheet.

Looks like the closest you can get to the runtime snapshot is to use pretick.

Basically run this when the plugin is created:

this.runtime.pretickMe(this);

Then when you run the action to capture the pixel, you set a boolean in your plugin and tell the runtime to redraw.

this.runtime.redraw=true;

Then in pretick you'd grab it.

I wouldn't call drawgl directly. I mean you could but it would be very slow. Getting pixels from the canvas is slow anyway. What we're doing is grabbing the entire canvas and loading it into an image, which can take several frames. If you called drawgl directly you'd just be making the runtime draw something it would draw anyway.

Yeah, I figured the drawGL() thing would be hilariously awful.

I'm not quite sure if I follow how pretick is the closest to the snapshot, unless you mean, it's the closest to a snapshot taken on the prior tick, in which case I think I understand.

At the moment I'm triggering a condition (belonging to my plugin) when the img.onload() function runs. This means that I get to act as soon as the color info is available, but outside of that triggered condition, I can't guarantee that the image has been loaded yet.

So, is the boolean you're talking about part of a system that would make the color info accessible from regular non-triggered events? That is, as soon as the boolean is true, then the data would be stored during the very next pretick and, by the time the event sheet code runs, you know you can trust the image data?

Finally I'm also not sure if I understand the reason for setting the runtime redraw to true. Would that indicate to C2 that the scene need to be redrawn during the next render phase?

All that said, with your help so far, I've gotten a system that does allow me to pick any color on the screen as it appears after shaders are applied, which is really handy, so thanks.

My main plan was to use it as part of a color picking UI. Fortunately a color picking UI doesn't require thousands of picks per frame-draw or anything, so it's probably okay that it's not super efficient at the moment.

I also just noticed that because my current version of the picking function takes a dataURL as an image parameter, it also works with Pode's base64 image extractor and injector. Though again, that extracts and injects underneath the shading pass, but it's a cool discovery none the less.