Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control how an isolate resolves imports #16368

Closed
munificent opened this issue Jan 28, 2014 · 20 comments
Closed

Control how an isolate resolves imports #16368

munificent opened this issue Jan 28, 2014 · 20 comments
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. library-isolate P2 A bug or feature request we're likely to work on type-enhancement A request for a change that isn't a bug

Comments

@munificent
Copy link
Member

In pub's build system, we use isolates to implement a plug-in API. Those transformer plug-ins import other libraries. Currently, the VM gives us no control over how those imports are handled. This is broken for us for two reasons:

  1. The package root for those imports is wrong. It uses pub's own package root, which is obviously meaningless for some random application.
  2. The libraries being imported may be themselves the result of a transformer. In this case, we need to be in control of the actual contents of the library being imported.

These are both necessary for the semantics of the build system. To work around the lack of control here, we internally start an HTTP server. Then we use the analyzer to parse every loaded Dart file and rewrite its import directives to hit our server. Every byte of loaded code then has to go over HTTP before it gets into the isolate. Since HTTP imports are resolved synchronously, we also have to put that onto another isolate.

As you can imagine, this adds significant complexity to pub, as well as adding several seconds of pointless startup time when running pub build.

@lrhn
Copy link
Member

lrhn commented Jan 29, 2014

We definitely want to allow changing the package root for an isolate created with Isolate.spawnUri. That is on our list of isolate changes that we want to do.

I'm not sure I would want to allow intercepting arbitrary library URLs from the the isolate. It is a very powerful and intrusive measure. On the other hand, you can get exactly the same effect by running a web server and serving modified libraries through that, or copying the libraries to local files and modifying them to use file: links.
So, as long as it doesn't allow you to get around the browser's same-origin policy somehow, I don't see it as impossible.
It is not in the current top-ten of isolate changes to do, though.

@munificent
Copy link
Member Author

We definitely want to allow changing the package root for an isolate created with Isolate.spawnUri.

I can't speak for other users, but we will get no value from that feature.

I'm not sure I would want to allow intercepting arbitrary library URLs from the the isolate. It is a very powerful and intrusive measure. On the other hand, you can get exactly the same effect by running a web server and serving modified libraries through that, or copying the libraries to local files and modifying them to use file: links.

Both of those are complex to implement correctly, error-prone, and have a real negative performance impact. The majority of the time that pub build spends not actually running transformers is now taken up by exactly the workaround you describe.

It is not in the current top-ten of isolate changes to do, though.

What is in the top ten list, and how was the ranking determined?

@nex3
Copy link
Member

nex3 commented Feb 13, 2014

Added Library-Isolate label.

@floitschG
Copy link
Contributor

I already replied a long time ago, but by mail, and apparently that went into the void...

What is in the top ten list, and how was the ranking determined?
I'm only guessing, but they are probably things that work in dart2js too.

  • capabilities
  • control-ports
  • fatal/non-fatal errors
  • error-reporting
  • pause, resume, ping, kill
  • package-root
  • synchronous error-reporting
  • ...

@munificent
Copy link
Member Author

I already replied a long time ago, but by mail, and apparently that went into the void...

Sorry, I don't recall seeing a reply.

I'm only guessing, but they are probably things that work in dart2js too.

Why is that a priority? I still don't see any details here about how you prioritize things.

@lrhn
Copy link
Member

lrhn commented Feb 14, 2014

The things Florian mentioned are all things that have been scheduled for this quarter (or as many of them as possible).

We want to be able to create, control and inspect new isolates. For that we need pause and error handling.

Also, if possible, I would like to get to the point where you never need to run a new instance of the Dart runtime in order to run code in a different setting, but can just start a new properly configured isolate. That means being able to configure the things that you can set on the command line: package root and -D-defines (for spawnUri only).

These things are not formally prioritized between themselves, but they do have priority over things not on the list of Q1 TODOs.

@munificent
Copy link
Member Author

The things Florian mentioned are all things that have been scheduled for this quarter (or as many of them as possible).

OK, I'm still not seeing an answer here. Why are those the things you've scheduled for this quarter?

@andersjohnsen
Copy link

This discussion is not very constructive. It would be a lot easier to approach if an actual API modification was suggestion.

Should it be as simple as an extra named argument?:

  spawnUri(..., UriRewriteCallback rewriteUri);

  typedef Future<Uri> UriRewriteCallback(Uri);

And what are the requirements? I expect that the uri's can be of both 'http' and 'file' schemes. And what about stack-traces, should they include the original or rewritten Uri? What if it fails to load a resource - should we have a callback for that as well, that can supply an alternative?

Until we have these details sorted out, discussing priorities makes little sense.

I also believe this answers your questions about why: The other APIs are documented and (largely) decided upon.


cc @munificent.

@dgrove
Copy link
Contributor

dgrove commented May 27, 2014

My impression is that this issue will be solved by the implemented of issue #18813 .

@andersjohnsen
Copy link

The issues should be merged then.

@munificent
Copy link
Member Author

Like Dan says, I think resolvers will handle this, but just to clarify:

Should it be as simple as an extra named argument?:
spawnUri(..., UriRewriteCallback rewriteUri);
typedef Future<Uri> UriRewriteCallback(Uri);

No, see point 2 in the original discussion. We need to be able to provide the contents of the library, not just its URL. It could be as simple as this:

spawnUrl(..., Future<String> loadUri(Uri uri));

At least, as far as I know, that would work. There may be some corner cases with canonicalization that would be an issue.

And what are the requirements?

I know they aren't listed here, but we have discussed the requirements in detail with Florian and others over the past six months.

And what about stack-traces, should they include the original or rewritten Uri?

See above, this isn't about re-writing URLs, it's about controlling the loading process, so the original URL would be used.

Until we have these details sorted out, discussing priorities makes little sense.

The thing is, this has to be a priority in order for anyone (outside of Nathan and I, for whom this is a priority) to be willing to spend the time to sort out the details.

I also believe this answers your questions about why: The other APIs are documented and (largely) decided upon.

That just begs the question of how those other APIs were a high enough priority to spend the time to document and decide upon the APIs. It's not like those APIs just came down from the heavens. :)

@kasperl
Copy link

kasperl commented Jul 10, 2014

Added this to the 1.6 milestone.

@sethladd
Copy link
Contributor

Marked as High, assigning owner. If I get this wrong, please help me by:

* assigning a new owner

  • changing priority
  • changing milestone

Set owner to @lrhn.

@DartBot
Copy link

DartBot commented Jul 28, 2014

This comment was originally written by @sethladd


This is apparently causing pub run to take 2+ seconds. So +1 for addressing. :)

@sethladd
Copy link
Contributor

Marked this as blocking #20212.

@munificent
Copy link
Member Author

My understanding is that Ivan is working on (or plans to work on) either this directly, or some underpinning of it, but I don't see him on this bug.

CC'ing.


cc @iposva-google.

@dgrove
Copy link
Contributor

dgrove commented Aug 6, 2014

Based on talking with Ivan, this is not going to land in 1.6 .


Removed this from the 1.6 milestone.
Removed Priority-High label.
Added Priority-Medium label.

@munificent
Copy link
Member Author

It's still high priority, though, isn't it?

@dgrove
Copy link
Contributor

dgrove commented Aug 6, 2014

Without a milestone defined, we can't have it be a high-priority issue.

@lrhn lrhn added area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. and removed area-core-library SDK core library issues (core, async, ...); use area-vm or area-web for platform specific libraries. labels Dec 8, 2015
@lrhn lrhn assigned iposva-google and unassigned lrhn Dec 8, 2015
@kevmoo kevmoo added P2 A bug or feature request we're likely to work on type-enhancement A request for a change that isn't a bug and removed triaged labels Mar 1, 2016
@iposva-google iposva-google removed their assignment May 31, 2016
@rmacnak-google
Copy link
Contributor

Package maps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. library-isolate P2 A bug or feature request we're likely to work on type-enhancement A request for a change that isn't a bug
Projects
None yet
Development

No branches or pull requests