The VM should detect this earlier, and throw an ArgumentError.
I recently changed the argument validation code on bleeding edge, so maybe the problem is already fixed, or there may be some cases that slip through the detection. The error message looks like the old one.
Can you describe the function you use?

Added Area-VM, NeedsInfo labels.

This comment was originally written by alexe.kaigorodov@gmail.com

see co19 test LibTest/isolate/Isolate/spawn_A02_t01 from https://code.google.com/p/co19/source/detail?r=669 . Remove "return" statement from the middle of the text to enable the test.

This comment was originally written by alexe.kaigorodov@gmail.com

An Error is thrown, but not at the place of invokation:

main() {
  void f(SendPort replyPort) {
    replyPort.send("justas");
  }

  var receivePort = new ReceivePort();

  try {
    Isolate.spawn(f, receivePort.sendPort);
    print("no exception");
  } catch(e) {
    print("exception: $e");
  }
}

gives:

no exception
Uncaught Error: Illegal argument(s): Isolate.spawn expects to be passed a static or top-level function
Stack Trace:
#­0 Isolate._spawnFunction (dart:isolate-patch/isolate_patch.dart:272)
#­1 Isolate.spawn (dart:isolate-patch/isolate_patch.dart:233)
#­2 main (file:///home/kaigorodov/dash/dash/co19/tests/co19/src/LibTest/isolate/Isolate/spawn_A02_t01.dart:29:18)
#­3 _startIsolate.isolateStartHandler (dart:isolate-patch/isolate_patch.dart:216)
#­4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:115)

Unhandled exception:
Illegal argument(s): Isolate.spawn expects to be passed a static or top-level function
#­0 _rootHandleUncaughtError.<anonymous closure>.<anonymous closure> (dart:async/zone.dart:677)
#­1 _asyncRunCallback (dart:async/schedule_microtask.dart:18)
#­2 _asyncRunCallback (dart:async/schedule_microtask.dart:21)
#­3 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:119)

Validation should be performed as soon as possible, in the Isolate.spawn() function.

The VM now does throw an argument error, at the line of Isolate.spawn (4th frame of the stack).

$ dart --version
Dart VM version: 2.0.0-dev.63.0 (Fri Jun 15 00:42:43 2018 +0200) on "macos_x64"
$ cat 15234.dart 
import 'dart:isolate';

main() {
  void f(SendPort replyPort) {
    replyPort.send("justas");
  }

  var receivePort = new ReceivePort();

  try {
    Isolate.spawn(f, receivePort.sendPort);
    print("no exception");
  } catch(e) {
    print("exception: $e");
  }
}
$ dart --preview-dart-2 15234.dart 
no exception
Unhandled exception:
Invalid argument(s): Isolate.spawn expects to be passed a static or top-level function
#0      Isolate._spawnFunction (dart:isolate/runtime/libisolate_patch.dart:506:59)
#1      Isolate.spawn (dart:isolate/runtime/libisolate_patch.dart:359:7)
<asynchronous suspension>
#2      main (file:///Users/srawlins/code/15234.dart:11:13)
#3      _startIsolate.<anonymous closure> (dart:isolate/runtime/libisolate_patch.dart:279:19)
#4      _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:165:12)