Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archive sources for integration builds #6929

Closed
ricowind opened this issue Nov 26, 2012 · 14 comments
Closed

Archive sources for integration builds #6929

ricowind opened this issue Nov 26, 2012 · 14 comments
Assignees
Labels
area-infrastructure type-enhancement A request for a change that isn't a bug

Comments

@ricowind
Copy link
Contributor

For easy access to the source code for a specific revision we should store a tarball of the sources for integration builds.

@ricowind
Copy link
Contributor Author

cc @mkustermann.

@DartBot
Copy link

DartBot commented Oct 29, 2013

This comment was originally written by rod...@gmail.com


Without source releases, it's increadibly akward to package dart as an official Arch Linux package. Please provide a source tarball per release.

@anders-sandholm
Copy link
Contributor

Removed Area-Build label.
Added Area-Infrastructure label.

@sgjesse
Copy link
Contributor

sgjesse commented Dec 10, 2013

Issue #15459 has been merged into this issue.

@sgjesse
Copy link
Contributor

sgjesse commented Dec 10, 2013

Supplying a tarball of the sources for building will be a lot of additional work. Besides the sources from the dart SVN repro there are the third_party directories, and on top of that additional platform specific dependencies are fetched form Google Cloud Storage. Also the third_party directory will contain stuff (like the d8 binaries) which should be filtered out when preparing a tarball.

The next issueare the makefiles. These are generated from the gyp-files also using gclient. The makefiles generated are suitable for distribution - they may contain absolute paths and other specific details. There are no guarantees of portability.

Last but not least we will need to test this additional distribution. This means additional build-bots for tracking this.

We would very much like it to be easy for Linux distributions to include Dart, by making the build process as smooth as possible using the current build-tools. The depot_tools (containing gclient and additional tools) is available both as an SVN checkout and a tarball, and the following commands should build the 64-bit runtime target:

  svn co http://src.chromium.org/svn/trunk/tools/depot_tools
  export PATH=$PATH:pwd/depot_tools
  gclient config http://dart.googlecode.com/svn/branches/bleeding_edge/deps/all.deps
  gclient sync
  gclient runhooks
  cd dart
  tools/build.py -mrelease -ax64 runtime

Building the SDK and editor require installing Java (openjdk-6-jdk) as well.

  sudo install openjdk-6-jdk
  tools/build.py -mrelease -ax64 create_sdk

The above uses bleeding_edge, change accordingly for the other channels:

  gclient config http://dart.googlecode.com/svn/trunk/deps/all.deps
  gclient config http://dart.googlecode.com/svn/branches/1.0/deps/all.deps


Removed Type-Defect label.
Added Type-Enhancement label.

@sgjesse
Copy link
Contributor

sgjesse commented Dec 10, 2013

In comment#­5

sudo install openjdk-6-jdk

should be

  sudo apt-get install openjdk-6-jdk

or

  sudo yum install java-1.6.0-openjdk-devel

depending on the distribution.

@ricowind
Copy link
Contributor Author

It is possible to get specific revisions by changing:
gclient sync
to
gclient sync --revision REVISION --transitive

@DartBot
Copy link

DartBot commented Nov 25, 2014

This comment was originally written by beatg...@gmail.com


Has there been any update on this?

Dart has been in 1.x for over a year now, so perhaps external dependencies have settled down a bit, especially for just the Dart VM?

Source releases with minimal (hopefully zero) dependencies on forked projects would make porting a lot easier. As I've mentioned in issue #10260, I'm interested in porting to FreeBSD, but working with gclient (and to some extent gyp) is painful, especially since the build process is so bizarre and complicated.

@rmacnak-google
Copy link
Contributor

Do the tarballs at, e.g., http://gsdview.appspot.com/dart-archive/channels/stable/raw/44672/src/ resolve this issue? The third party dependencies are included.

@DartBot
Copy link

DartBot commented Apr 17, 2015

This comment was originally written by @mulander


#­9 looks like it could.

I've been waiting a long time for this. I'm attempting to do an OpenBSD port, starting
as we speak.

The main question I have with your URL:

The chromium project hosts a https://gsdview.appspot.com/chromium-browser-official/ which nicely lists all release tarballs in a single place. This makes it easy for port maintainers to find the proper download links and allows the port infrastructure to do most of the work.

The URL you provided http://gsdview.appspot.com/dart-archive/channels/stable/raw/44672/src/ has an unpredictable path. How do I locate the proper release tarball for a dart version as you seem to include the revision in the URL and some of the other folders I looked at don't even contain the src folder?

@DartBot
Copy link

DartBot commented Apr 17, 2015

This comment was originally written by @mulander


The biggest difference I see between the Chrome build & dart is the handling of makefiles.

Doing the standard:
$ gclient config http://dart.googlecode.com/svn/branches/1.1/deps/all.deps
$ gclient sync
$ gclient runhooks

Results in a source tree that's ready to build with:
$ cd dart
$ tools/build.py --mode=release --arch=x64 create_sdk

Using the provided download URL both on Linux & OpenBSD
we do have a proper checkout with all third parties (major props)
but there's no way to continue with the build without significant
intervention:

Linux:
root@build:~/dart/dart-1.9.1/dart# python2.7 tools/build.py -m release --arch=x64
make -j 4 BUILDTYPE=ReleaseX64 all
make: *** No rule to make target `all'. Stop.
BUILD FAILED

OpenBSD:
$ make build
===> Building for dart-1.9.1
gmake -j 2 BUILDTYPE=ReleaseX64 all
gmake: *** No rule to make target 'all'. Stop.
BUILD FAILED
*** Error 1 in . (Makefile:70 'do-build': @­cd /usr/ports/pobj/dart-1.9.1/dart-1.9.1/dart && /usr/local/bin/python2.7 ./tools/build.py -m rel...)
*** Error 1 in . (/usr/ports/infrastructure/mk/bsd.port.mk:2747 '/usr/ports/pobj/dart-1.9.1/.build_done')
*** Error 1 in /usr/ports/mystuff/lang/dart (/usr/ports/infrastructure/mk/bsd.port.mk:2476 'build')
$

I cross checked with the www/chromium build on OpenBSD and it seems that Chromium provides a great build/linux/unbundle/replace_gyp_files.py which
allows the port to generate the required makefiles without going through the gclient stuff, from the chromium build:

gyp_chromium generates all the Makefiles from gyp

do-configure:
        cd ${WRKSRC} &&
                build/linux/unbundle/replace_gyp_files.py ${GYP_DEFS} &&
                env -i ${CONFIGURE_ENV} ${MODPY_BIN} build/gyp_chromium
                ${GYP_DEFS} "--depth=${WRKSRC}"

Is there a way to achieve the same with Dart?

@DartBot
Copy link

DartBot commented Apr 18, 2015

This comment was originally written by @mulander


OK I managed to generate the Makefiles without using gclient.

do-configure:
    @cd ${WRKSRC} && /usr/local/bin/python2.7 ./dart/tools/compiler_scripts/generate_my_projects.py &&
    /usr/local/bin/python2.7 ./dart/editor/build/generate_sources.py &&
    /usr/local/bin/python2.7 ./dart/tools/gyp_dart.py

$ make configure
===> Configuring for dart-1.9.1
$

So the tarball indeed looks workable. I guess this issue can be closed though a more predictable path for downloads would be appreciated).

@DartBot
Copy link

DartBot commented Apr 18, 2015

This comment was originally written by @mulander


OK one more issue. Why is third_party/clang not included? Are we forced to build with GCC?

$ ls third_party/clang
ls: third_party/clang: No such file or directory

From a manual gclient checkout:
$ ls third_party/clang/linux/lib/clang/3.4/include/
Intrin.h cpuid.h mm3dnow.h shaintrin.h tmmintrin.h
__wmmintrin_aes.h emmintrin.h mm_malloc.h smmintrin.h unwind.h
__wmmintrin_pclmul.h f16cintrin.h mmintrin.h stdalign.h varargs.h
altivec.h float.h module.map stdarg.h wmmintrin.h
ammintrin.h fma4intrin.h nmmintrin.h stdbool.h x86intrin.h
arm_neon.h fmaintrin.h pmmintrin.h stddef.h xmmintrin.h
avx2intrin.h immintrin.h popcntintrin.h stdint.h xopintrin.h
avxintrin.h iso646.h prfchwintrin.h stdnoreturn.h
bmi2intrin.h limits.h rdseedintrin.h tbmintrin.h
bmiintrin.h lzcntintrin.h rtmintrin.h tgmath.h

@ricowind
Copy link
Contributor Author

First off, very nice that somebody is doing work the get us going on bsd.

Regarding clang, we explicitly exclude some binary dependencies due to size, see tools/create_tarball.py in a regular dart checkout. If this is a major blocker we can add it to the tarball, but you should be able to easily pull it of svn (will be git soon) if needed:
svn co http://dart.googlecode.com/svn/third_party/clang

Regarding predictable paths for download:
gsutil cat gs://dart-archive/channels/stable/release/*/VERSION
will get you the revision to version mappings
Or if you want urls to version mappings, something like:
gsutil ls gs://dart-archive/channels/stable/release/ | awk '{print "echo " $1 " && gsutil cat " $1 "VERSION | grep version" }' | sh


Added Fixed label.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-infrastructure type-enhancement A request for a change that isn't a bug
Projects
None yet
Development

No branches or pull requests

6 participants