SikuliX API 1.1.0 is on Maven Central

It is available for dependency download in projects and this is the index entry.

If you used the snapshot version until now, just remove the -SNAPSHOT from the version in the dependency specification.

The OSSRH repository entry is not needed anymore, until you decide to again use snapshots of version 1.1.1 towards end of this year.

SikuliX API 1.1.0 is on Maven Central

New in 1.1.0 — observe revised with new features

The observe feature (see docs especially for the usage with Java) allows to wait for an image to appear or vanish or to wait for changes of the pixel content in a region on the screen. This observation can be done inline (the script waits until the observe ends) or in parallel to the continuing workflow, meaning that the observation is delegated to a thread (hence runs in background).

Before running the observation, you register one or more events (appear, vanish or change) with the region, where you expect the events to happen. Since the beginning of the observe feature in Sikuli, you have to specify a handler function (callback), that is called, when the event happens and where you could handle the situation with the option to stop the observation at this point. With one region object (you can have different objects for the same area on the screen though) only one observation can be run at any time, but you might specify as many events as needed (… or what makes sense).

Here I only want to talk about background observes, which is the more interesting situation (the inline observe is only some bundled waits just a bit more sophisticated).

So the usual pattern in a Python script is:

def handler(event):
    # do something
reg = <some region object>
reg.onAppear("someImage.png", handler)
reg.observeInBackground(FOREVER) # observe in background
# script immediately continues here

In the moment someImage.png appears in the given region, we want to do something (which is done in the handler function). Since the observe runs in background, our script continues and the handler does its job in parallel to our script in the moment the event happens.

When the script ends all running observations are stopped automatically.

With the revision in 1.1.0 the observe feature now is very reliable and produces valuable debug information if needed.

These are the major changes and new features:

  • observation of appear and vanish events are paused when they happen – they have to be explicitely told to continue their observation
  • there is a count how often an event happened until now
  • the events get a unique name at time of registering, that can be used, to later access the event information (region, image, match, count, …)
  • the event attributes are accessed using getters
  • you might inactivate (pause) the observation of an event and activate it again later
  • at any time you can get relevant information about the state of an observation

Another new helpful feature, is the fact, that mouse actions are now handled like transactions: even in parallel workflows (like with a main script and background handlers) only one compound mouse action can be processed at any one time. This means a mouse click in the handler is completed before a parallel click in the main workflow will be processed and vice versa. This rather complex feature is worth its own article.

The same example as above, now continuing the observation after the event happened:

def handler(event):
    # do something
    if event.getCount() < 5:
reg = <some region object>
reg.observeInBackground(FOREVER) # observe in background
# script immediately continues here

Besides showing the getters it is principally the same as above. But now we limit the appearences to 5 (after that the event is deactivated) and say that the observation for this event should continue only after a pause of 1 second after returning from the handler.

With this last example I want to show the usage of named events.

The challenge is to scroll a webpage until an image gets visible. I know, that this can be rather easily done with exists() too, it is here done with observe for demonstration only.

For this snippet to work, the mouse must be positioned at a place, where the focused window accepts mouse wheeling.

reg = <some region object>
imgStop = "imgStop.png" # the stop image to wait for 

stopAppeared = reg.onAppear(imgStop) # no handler, saving the name
reg.observeInBackground(FOREVER) # start observation 
while not reg.isObserving(): wait(0.3) # hack currently needed

while reg.isObserving(): # until observing is stopped
   wheel(WHEEL_UP, 10) # wheel (might be WHEEL_DOWN on Windows)

m = r.getEvent(stopAppeared).getMatch() # get the event's match

We register an appear event without specifying a handler, but store the event’s name in stopAppeared. We use Region.isObserving(), to wait for the observation to stop because our one event has happened. While the observe is running, we use the mouse wheel to scroll the page. After the observation has ended, we get the event’s match to move the mouse there and add a 2-seconds-highlight.

The mentioned hack after starting the observation is currently needed, to bridge the startup time for the observation (loading the image and preparing the threaded search loop). This will be revised with version 1.1.1.

Another possibility would have been to use the features Region.hasEvents() or Region.hasEvent(stopAppeared) instead of Region.isObserving() like this:

while len(reg.hasEvents()) == 0: # returns a maybe empty list
# or
while not reg.hasEvent(stopAppeared): # returns the event or None

Hope this gives some ideas about new possibilities with version 1.1.0

New in 1.1.0 — observe revised with new features