/benchmark - with a twist

By jaytea on Oct 18, 2010



/benchmark [-cs[N]ev] {M} {text}

Executes or evaluates {text} exactly {M} times and displays
the results.

-c: interprets {text} as a full command, otherwise it is simply
evaluated as code in a /noop (do nothing) command.

-s: enables 'sensitive' mode wherein the main loop will iterate
fewer than {M} times by forcing more than one evaluation or
execution of {text} per iteration. This is especially useful
for benchmarking basic operations. The alias uses the following
loop to perform the main benchmark:

while (%main) { {command} | dec %main }

-s is used to force {command} to be performed multiple times
within the loop body:

while (%main) { {command} | {command} | ... | dec %main }

If the operations in {command} are of comparable intensity to
the overhead involved in using a while loop, ie. the /dec %main
coupled with the testing of the (%main) condition, then you may
want to use the -s switch to force {text} to be evaluated more
than one time, or {text} performed more than one time if -c is
used, per iteration of the while loop.

If N is specified with -s then the alias will attempt to evaluate
or perform {text} N times per iteration of the while loop. If no
N is given, the alias will try to maximize this value by looking
at the length of {text}. It is recommended that you do not specify
N unless you are certain that it can be handled without exceeding
any mIRC length limit - the alias will not check this for you.

-e: used only with /benchmark -s. When trying to maximize the
number of times to evaluate {text} per iteration, the alias will
typically evaluate {text} once before commencing the benchmark
and consider the length of the return value in its calculations.
Including the -e switch prevents this from occurring, which you
may want to use if the return value of {text} varies and its
length is not estimated properly by the alias. For even greater
control, use the N option with the -s switch instead.

v: verbose; displays results of the 'speed factor' calculation.

The alias conducts a slightly more sophisticated test than the
common existing methods.

The 'speed factor' is an attempt to standardize the results of these
benchmarks across different machines by calculating (before and
after the main benchmark) the time it takes for your PC to perform a
simple while loop which it then uses in the overall calculation.
If done correctly, a 'speed factor' should be a property of {text}
and should be independent of the number of iterations and, more
importantly, the PC on which the benchmark is performed.

It makes the assumption that the ratio between the execution times
of two pieces of code on one machine will be equivalent to that of
a different machine. While this assumption is far from realistic,
it allows for the simple calculation of a measurement that would
otherwise be practically unattainable.


/benchmark 10000 $myidentifier
; Measures 10,000 iterations of /!noop $myidentifier

/benchmark -c 10000 /mycommand
; Measures 10m000 iterations of /mycommand

/benchmark -s 1000000 $null
; Sensitively measures 1,000,000 evaluations of $null

/benchmark -cs100 100000 /noop
; Sensitively measures 100,000 executions of /noop
; Using 100 executions per iteration of a loop.



alias benchmark {

  var %i = 50000 | ; number of simple iterations to use to calculate the 'speed factor'

  if (1 \\ + $+ %i) {
    echo -egac info * /benchmark: invalid value for % $+ i. Must be a non-negative integer.

  var %m, %switches

  if (-?* iswm $1) {
    %switches = $1
    tokenize 32 $2-

  if (1 \\ + $+ $1) {
    echo -egac info * /benchmark: invalid <M> parameter. See documentation for more info.

  var %command, %command2, %full, %reps, %remain, %pre = $iif(c !isincs %switches, !noop) $2-

  %full = scid $cid !set $(%t $ticks |, 0) while (%m) $({, 0) ~%c $(| !dec %m }, 0) $(| !set %m $ticks - %t, 0) 

  if (s isincs %switches) {
    %reps = $int($deltok(%switches, 1, $asc(s)))

    if (%reps > $1) {
      echo -egac info * /benchmark: invalid N supplied. See documentation for more info.

    if (c isincs %switches) {      
      if (!%reps) %reps = $int($calc((4156 - $len(%full)) / $len($2- ..)))

      if (%reps <= $1) %command = $mid($str($chr(32) $+ $2- |, %reps), 2, -2)

      var %remain = $1 % %reps
      if (%remain) %command2 = $mid($str($chr(32) $+ $2- |, %remain), 2, -2)
    else {
      if (!%reps) {
        %max = $len($2-.)
        if (e !isincs %switches) && ($len($eval($2-, 2)) >= %max) %max = $v1 + 1
        %reps = $int($calc((4148 - $len(%full)) / %max))

      if ($regex($1-, /(?=((?:^| )\[(?= )(*SKIP)(?>(?1)|.)*? ](?=$| )))./g)) {
        if (%reps > $calc(200 / $v1)) %reps = $v2      

      if (%reps <= $1) %command = !noop $mid($str($chr(32) $+ $2-, %reps), 2)

      var %remain = $1 % %reps
      if (%remain) %command2 = !noop $mid($str($chr(32) $+ $2-, %remain), 2)

    %reps = $int($calc($1 / %reps))
  else {
    %reps = $1
    %command = %pre

  var %t, %before, %main, %after

  %m = %i
  !set %t $ticks | while (%m) !dec %m | !set %m $ticks - %t
  %before = %m

  if (%reps) {
    %m = %reps
    $replace(%full, ~%c, %command)
    %main = %m
  if (%remain) {
    %m = %remain
    scid $cid $(!set %t $ticks |, 0) %command2 $(| !set %m $ticks - %t, 0) 
    inc %main %m

  %m = %i
  !set %t $ticks | while (%m) !dec %m | !set %m $ticks - %t
  %after = %m

  var %total = %before + %after

  if (v isincs %switches) {
    echo -qgace info First loop: %before $+ ms - Second loop: %after $+ ms - A total of $&
      %total $+ ms for $calc(2 * %i) trivial cycles ( $+ $calc(%total / 2 / %i) $+ ms per cycle)
  if (%remain) inc %reps
  var %inf = $iif(c isincs %switches, execution, evaluation), %s = $iif($1 != 1, s)
  echo -gac info2 $1 %inf $+ %s  ( $+ $iif(%remain, ~) $+ $calc($1 / %reps) time $+ $iif($v2 != 1, s)  $&
    over %reps iterations) of: %pre
  echo -gac info2 %main $+ ms for $1 %inf $+ %s ( $+ $calc(%main / $1) $+ ms per %inf $+ ) - Speed factor: $&
    $calc((%main / $1) / (%total / %i))


Sign in to comment.
jaytea   -  Feb 23, 2011

huge update. i made the syntax more sensible and added a few new switches.

/benchmark -s is a new switch that is indispensable for mSL aficionados who require extreme precision in their measurements. if you thought $() and $null were equally fast based on older and less sophisticated benchmark methods, think again! now you can perform up to 1,000,000 evaluations of a basic piece of code in a matter of seconds, with minimal overhead.

sorry about all the documentation, but there is quite a bit to explain. if you can be bothered to go through it all, you will get the most out of this alias :P

FordLawnmower   -  Oct 19, 2010

Nice jaytea :)
Can't wait to test this out. This will probably make some coding decisions easier.

napa182   -  Oct 18, 2010

nice useful lil snippet jaytea 9/10 +Like

Are you sure you want to unfollow this person?
Are you sure you want to delete this?
Click "Unsubscribe" to stop receiving notices pertaining to this post.
Click "Subscribe" to resume notices pertaining to this post.