please add a line (for noobs like me) to say what python version is required (2.5?) to convert logs ;)

You can also use ffmpeg for generating the video (from your frames directory): ffmpeg -f image2 -i swarm-%05d.png /tmp/foobar.mpg

maybe plan a option to convert apache/httpd access log into xml event would be a good idea ?

"To execute code_swarm, you need the Processing application (which uses Java)." I'm newbie, so can you please explain me what exactly I have to type in my command line to execute code_swarm (I use Ubuntu 8.04)?

"Processing" is an application you down load from

http://processing.org/

String

mencoder mf://.png -mf fps=24:type=png -ovc lavc -oac copy -o movie.avi 

must be changed to

mencoder mf://*.png -mf fps=24:type=png -ovc lavc -oac copy -o movie.avi  

Regards, Vyacheslav

Wher do I store activity.xml, I am using version 0.139, and it seems to execute but nothing happens. I have put activity.xml in the data folder, should it be somwhere else?

Just an FYI, if you are trying to generate an activity.xml for CVS (it's on this project's trunk as of this morning), you will need to do the following: 1) Line 142 should be "...strip()", not "...strip('\n')". I had some issues where there was still a carriage-return at the end of the line which caused it not to equate properly. 2) After your output, you WILL need to sort it. In order to do this, you can change line 182 (where the "xml_handle" writes out the event) to write the date out first. Once you've done this, run the converter. Edit the XML output, and remove the first two lines and the very last line. Run sort on it, then add the three lines back in. Now you're ready to go. If anyone has alternatives to make this easier, let me know!

Actually, you will need to write out the date, then author, then filename for 'sort' to work correctly.

What command necessary use with CVS Reposotory for generate correct log file?

I can only speak for Ubuntu 8.04 (although I wouldn't be surprised if this applies to all nix systems). After you've got the activity.xml for CVS and you've removed the appropriate lines do something like the following:

sort activity.xml > sorted-activity.xml

Here's my java n00b question: Is this geared to a specific JDK? I'm using JDK 1.6.0_10-beta-b25 and am getting stack traces using code_swarm.pde with Processing. Also, was this written with a certain version of Processing? (I'm going to see if I can answer my own question and user a previous version of Processing.) Thanks for any help. If I wasn't such a Java dork, I'd help out with this program.

There appears to be a limit, at least on my rinky system (only 1G RAM on Windows Vista), to the size allowed in the generically named activity.xml file. Had a 20 MB SVN dump from a very busy /trunk. Broke this down to ~5MB and everything works fine. I did (finally) read the FAQ. My processing version was too old (I grabbed the 0135 beta, not the newer 0142 beta) and figured out my jdk was just fine. Thanks guys/gals for the cool program.

In order to process a lot of data it does require a lot of memory. Each file node that ever enters CVS/SVN is kept internally for book keeping, so it does add up.

I generally run at about 512m with large files, which seems good enough, but if you don't have that free the only way is to split up the file indeed.

I created a converter to generate the activity xml from Perforce. Let me know if there is interest in this and I can try to upload it somewhere.

ajbanck - email it to me and i'll commit: acgourley (gmail)

For users of ffmpeg, here's the command I used: ffmpeg -f image2 -r 24 -i ./frames/swarm-%05d.png -sameq ./out.mov -pass 2

That means input as images, 24 frames per second, input filename format, high quality, output file, 2 passes.

ajbanck: could you mail the converter to "oregu at yandex dot ru"?

convert_logs.py script now supports perforce: python convert_logs.py "--perforce-path=//depot/..." This command will generate depot.xml including all file activities in specified perforce folder. Command "p4" should be installed and configured.

there is no sound, use the "-nosound" switch for mencoder and remove the -oac copy ...

If you want to create movies from Mercurial repositories, you can use the code from Baptiste Lepilleur:

http://www.nabble.com/forum/ViewPost.jtp?post=18430641&framed=y

With it he created this beautiful Mercurial codeswarm video:

http://www.vimeo.com/1334189

To answer some of the above questions:

To actually run this follow the instr. above and:

1. on Winblows XP for example, move convert_logs\activity.xml to data\activity.xml

2) edit data\sample.config and change the following:

a) InputFile?=data\activity.xml b) TakeSnapshots?=true c) DrawNamesHalos?=true (looks a little nicer)

3) create directory "frames" in 4) type run.bat 5) accept the default location for data\sample.config 6) follow the mencoder directions (mencoder is better than ffmpeg in my opinion - more efficient and I trust mplayer more on the front-end)

-Sebastian

I just created a script which can pull in and combine commits from many different projects and creates codeswarms from only the changes since the last created codeswarm.

It isn't yet polished, but I think it might be quite useful.

http://rakjar.de/shared_codeswarm/readme.html

I use it to gather commit data from many different repositories, but I plan to use it to track code evolution of bigger projects in near-realtime (means: checking once a day or so :) ).

It automatically creates a video, and names it shared_movie_DATE-TIME.avi

For using it, you need to write a "simple.config" file like the following: - http://freehg.org/u/ArneBab/shared_codeswarm/raw-file/tip/simple.config

Then put it into a directory and start the community_codeswarm.py script in that directory.

- python community_codeswarm.py --codeswarm-path "absolute path to the codeswarm dir"

For it to work, you need pyyaml, Mercurial and naturally codeswarm installed. - http://pyyaml.org/ - http://selenic.com/mercurial

Please tell me how the script works for you! - http://draketo.de/contact

License: GPLv3 - like codeswarm. - Arne Babenhauserheide

For everyone's info:

I was able to get code_swarm running beautifully on an Intel MacBook? Pro running 10.5+. I had to use MacPorts? to install a number of applications, but it works just like the instructions for a GNU/Linux setup.

I would also like to request that the steps in Sebastian's comment (Sept 04, 2008) be put in the general instructions. I was lost without the tip to create the frames directory and update the data/sample.config. One could also edit the run.sh file and point the "default_config" variable to a different named file.

-Ammon Shepherd

ajbanck, i see you've worked with perforce. anyone else? if you have the way to create the perforce activity xml - can you tell us where it's posted or how to get it?

cliff

@latestlinux thanks, that got me started quickly

Did the Perforce script get committed?

recent mencoders need the following for h264 output:

mencoder mf://code_swarm-0.png -mf fps=25:type=png -ovc x264 -x264encopts bitrate=200 -o output.avi

Hello @all, Generating den PNGs from SVN Activity was pretty easy.

Now i try to use mencoder for Windows to generate an avi. But all i get is: MEncoder Sherpya-SVN-r28311-4.2.5 (C) 2000-2009 MPlayer Team CPU: Intel(R) Core(TM)2 CPU T7600 @ 2.33GHz (Family: 6, Model: 15, Stepping: 6) CPUflags: Type: 6 MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 1 Compiled for x86 CPU with extensions: MMX MMX2 SSE SSE2

success: format: 16 data: 0x0 - 0x0 MF file format detected. mf? search expr: D:\Projekte\CodeSwarm?\frames\code_swarm-00001.png mf? number of files: 1 (4) VIDEO: MPNG? 800x600 24bpp 25.000 fps 0.0 kbps ( 0.0 kbyte/s) V? filefmt:16 fourcc:0x474E504D size:800x600 fps:25.000 ftime:=0.0400 Opening video filter: osd=1? Expand: -1 x -1, -1 ; -1, osd: 1, aspect: 0.000000, round: 1

Opening video decoder: ffmpeg? FFmpeg's libavcodec codec family Selected video codec: ffpng? vfm: ffmpeg (FFmpeg PNG decoder)

Flushing video frames. Filters have not been configured! Empty file? Writing index... Writing header... ODML: Aspect information not (yet?) available or unspecified, not writing vprp header.

Video stream: nan kbit/s (-2147483648 B/s) size: 0 bytes 0.000 secs 0 frames

Does anyone can help me, or tell me wich tool is easyer to use under Windows?

Regards

Klaus Buchwald

I had some real problems with mencoder

Trying to get an alreayd compiled version for Windows took for ages - for anyone who is interested, if you download "Media Coder" mencoder and ffmpeg are bundled with it in the codecs folder.

For some reason it didn;t like PNGs, came out with what looked muvh like static on a TV. So I converted all the files to bitmaps, that solved that issue.

Then I had trouble getting mencoder to create a file that windows media player could read without any extra codecs. Here's the command line I used in the end that allowed me to do this:

mencoder "mf://code_swarm-.bmp" -mf fps=25 -o swarm.mpeg -ovc lavc -lavcopts vcodec=msmpeg4v2

Hey, everyone I am using Windows XP Mencoder was throwing exepctions on my machine, I didn't manage to get it up. So I just used a tool called VideoMach?... and I got the pretty decent results.

Cheers.

Mercurial support

Assuming codeswarm is located in $CODESWARM_HOME, we have a python script in convert_logs that may be used for Mercurial repositories:

$ cd $CODESWARM_HOME/convert_logs/
$ python hg_log.py /Path/To/Your/Mercurial/Repository

this produces the file $CODESWARM_HOME/data/hglog.xml

Afterwards, just edit your configuration file ($CODESWARM_HOME/sample.config is fine) to reflect that InputFile=data/hglog.xml .

The final step is just to go into the main folder ($CODESWARM_HOME) and type ./run.sh (or run.bat).

Note: Do not use ant run, it will fail for mercurial repositories. But you might need to run ant to compile the code.

Would be nice to get some kind of feedback from the app on what's the status of png generation (like do a sys out when next percent of the work is done)

Instructions for running Codeswarm 0.1 on OS X 10.6 with a git repo

Make sure you have JDK 1.6 installed and setup, including having JAVAHOME set. These steps do not use the OpenGL option.

Build and Run

1. Change directories to your codeswarm directory. cd <PATH TO YOUR CODESWARM DIR>
2. Type ant command to start build
3. Change directories to your git repo directory. cd <PATH TO YOUR GIT REPO>
4. Type in this command:

git-log --name -status --pretty=format:'%n------------------------------------------------------------------------%nr%h | %ae | %ai (%aD) | x lines%nChanged paths:' > activity.log

1. Change directories to your codeswarm directory. cd <PATH TO YOUR CODESWARM DIR>
2. Change directories to the convert_logs directory. cd convert_logs
3. Convert the log files to XML format. The -g option means log format is git. (In git's case a modified git output log by step 2 above) (This command converts the git log to a XML version of itself). To do the conversion, type this command:

1. Change directories to the root codeswarm directory. cd ..
2. Change directories to the data directory. cd data
3. Edit the sample.config file to change the TakeSnapshots variable to true like this: TakeSnapshots=true
4. In the sample.config file also change the InputFile variable to be the path to your XML version of the activity logs. We just had step 5 output it into the same directory the command ran in, so in this case the setting is like this: InputFile=./convert_logs/activity.xml
5. Change directories back up one directory to the codeswarm root directory. cd ..
6. Create a 'frames' directory at top level of codeswarm directory if there isn't one already. This is where all the snapshots of the progress go. (You'll see exceptions about files not being able to be written if you don't do this.) mkdir frames
7. Type run.sh on command line in this directory.

Make a movie

1. Look in ./frames directory and you'll see all your snapshots. Depending on which tools you have available, you can then take these snapshots and make a movie. I pasted them all into iMovie, selected them all and gave them all a frame duration of 00:02 seconds (2 seconds) (In TimeCode? format the first set of zero's before colon is minutes and the second set after colon is second) Anyway this produced a nice little movie.

Cleanup

1. Change directories to your git repo directory. cd <PATH TO YOUR GIT REPO>
2. Cleanup the activity.log file in your git repo directory. rm activity.log
3. Change directories to your codeswarm directory. cd <PATH TO YOUR CODESWARM DIR>
4. Change directories to the convert_logs directory. cd convert_logs
5. Cleanup the activity.xml file in your convert_log directory in the codeswarm directory. rm activity.xml
6. Change directories to your codeswarm directory. cd <PATH TO YOUR CODESWARM DIR>
7. Change directories to your codeswarm directory. cd ./frames
8. Clean up all the snapshots. rm *.png

NOTE: Where absolute paths are such as in step 5 above, change the path to be the correct one for where you put your file.

LIttle better write up here:

http://sams-rants.blogspot.com/2010/02/codeswarm-01-on-os-x-106-with-git-repo.html

ffmpeg -f image2 -r 24 -i ./frames/swarm-%05d.png -sameq ./out.mov -pass 2

should be

ffmpeg -f image2 -r 24 -i ./frames/code_swarm-%05d.png -sameq ./out.mov -pass 2

It would be nice if I could just edit the wiki page.

Hi How do I do this with StarTeam?. Thanks.

@robrohan: Changed the wiki, let me know if anything else needs tweaking.

Please add link to java, ant and python2.6 installers

Can someone add proper steps up top for Perforce? Much appreciated! :)

Is there a verbose mode? I got it to the point where I gave it our activity log (which is huge) and the java app is stuck at white screen. I can't tell if it's still parsing the XML or has froze.

Also, I had a bit of a problem with my activity log needing to convert it to UTF8 otherwise the java parser threw some exceptions.

Hi,

Well, my experience with code_swarm wasn't so easy, but finally I got it to work for me and the results are great!

I'll try to summarize everything. First off, I'm using Rictic's fork. But many things here still apply to the original. Let me explain.

The best result I get is on OSX 10.5.8. What I do is that I git clone the repository, then add its /bin path to my path:

export PATH=$PATH:path/to/code_swarm/bin/

Then I go the the directory of the project I want to visualize and simply do: code_swarm

As soon as it starts drawing, I go to the code_swarm menu and I do quit. Then I look inside the project dir and there's a new dir: .code_swarm. It contains log.xml (the result of the output of git log processed by the python convert script - which is all automatic with Rictic's) and project.config. On my rig, it's THIS .config file that I have to modify to make things work the way I want i.e. enable snapshot frames etc. The resulting frames will go to the /frames folder in the code_swarm folder (NOT the .code_swarm dir in the project dir I was just talking about). I kept trying to modify the sample.config file and what not in the code_swarm folder and was getting no frames and no setting would change anything before I realized all of this.

Here's the config I use (some of the comments are mine):

# Input file - in the non rictic original code, this will be the output file (activity.xml) generated by the python script on the activity.log that was first output from the "git log" command whereas here this is all done automatically and the end result is found in the .code_swarm dir in the project repository as log.xml

InputFile?=log.xml

#Is the input xml sorted by date? It's faster and uses much less memory if it is

IsInputSorted?=true

##Output # Save each frame to an image? Use false when testing.

TakeSnapshots?=true

# Where to save each frame - That will be a folder in the code_swarm directory - create it beforehand. Later on if you use ffmpeg, there's reference to the number of digits here i.e. %05d.png - here I use only digits.

SnapshotLocation?=code_swarm_frames/#####.png

# Color assignment rules # Keep in order, do not skip numbers. Numbers start # at 1. # # Pattern: "Label", "regex", R,G,B, R,G,B # Label is optional. If it is omitted, the regex # will be used. # #In some examples, I see rather paths to different folders and there's no reference to file extensions?? And where does the "misc" type come from since it appears #during execution but is not referenced here??? #

ColorAssign1?="Tests",".test.", 153,255,255, 153,255,255 ColorAssign2?="Localizations","(.(\.mo|\.po))|(.\.lproj.)", 110,200,90, 110,200,90 ColorAssign3?="C",".(\.c|\.cpp|\.h)", 237,0,0, 237,0,0 ColorAssign4?="Python",".(\.py|\.pyx)", 65,105,225, 65,105,225 ColorAssign5?="CSharp",".(\.cs|\.csproj)", 255,255,0, 255,255,0 ColorAssign6?="Other Source Code",".(\.rb|\.erb|\.hs|\.sql|\.m|\.d| \.js|\.pl|\.sh|\.lhs|\.hi|\.hpp|\.cat|\.inf|\.sys|\.dll|\.as|\.cmake \.java)", 255,102,51, 255,102,51 ColorAssign7?="Documents/Images",".(\.txt|\.html|\.tex|\.tmpl|\.css| \.xml|\.yml|\.json|\.png|\.jpg|\.gif|\.jpeg|README|COPYING|LICENSE| AUTHORS|\.asciidoc|HACKING)", 238,130,238, 238,130,238

#This is terribly painfully slow on my old mac, in fact it stalls. Does it carry eye candy??? DrawNamesHalos?=false

##Core settings MaxThreads?=4 #I think this will translate to about 1 sec of video for 6 hours of real time project. FramesPerDay?=144

#Older methods of establishing frames #MillisecondsPerFrame?=21600000 #MillisecondsPerFrame?=43200000

#Yeah UseOpenGL=true

Width=800 Height=600

Font=SansSerif? FontSize?=10 BoldFontSize?=14

#What the hell are these? Not sure.

DrawFilesSharp?=true

# Draw fuzzy files

DrawFilesFuzzy?=false

# Draw jelly files

DrawFilesJelly?=true

# Show the Legend at start

ShowLegend?=true

# Show the History at start - Are there other options to histogram height/width for instance?

ShowHistory?=true

# Show the Date at start

ShowDate?=true

# Amount of life to decrement - default -2-2-1 - Not sure about this either.

EdgeDecrement?=-2 FileDecrement?=-2 PersonDecrement?=-1

#Speeds. #Optional: NodeSpeed?=7.0, If used, FileSpeed? and PersonSpeed? need not be #set. #Left to default

FileSpeed?=7.0 PersonSpeed?=2.0

#Masses

FileMass?=1.0 PersonMass?=10.0

# Life of an Edge - An "edge" you say?

EdgeLife?=300 EdgeLength?=50

##With lots of frames these should be long like this

# Life of a File FileLife?=1000 # Life of a Person PersonLife?=750

HighlightPct?=8

#These refer to a text file that contains some other params, any clue on possible arguments?

PhysicsEngineConfigDir?=physics_engine

# Force calculation algorithms ("PhysicsEngineLegacy?", "PhysicsEngineSimple?"...) :

PhysicsEngineSelection?=PhysicsEngineOrderly?

# then place png files named after the usernames of committers in # the data/local_avatars directory # data/local_avatars/default.png is used by default # Uncomment to not use avatars #AvatarFetcher?=NoAvatar? # To use local avatars, uncomment this line: #AvatarFetcher?=LocalAvatar?

So this just works great. But my old Mac is slow so I'd like to run this under my other rig i.e. Windows XP/Ubuntu. OMG that was a pain. It's confusing here because I tried both the original code and Rictic's and I use Rictic's in the same fashion as the original code because of issues I have with all of this.

First off git: if I ever have to do it manually. The wiki section about OSX 10.6 and Git here is wrong for me: http://code.google.com/p/codeswarm/wiki/GeneratingAVideo

There is no git-log command and --name-status is in one block so it should read like this on one single line and on my XP, this requires double quotation marks instead of single by the way.: git log --name-status -- pretty=format:'%n------------------------------------------------------------------------ %nr%h | %ae | %ai (%aD) | x lines%nChanged paths:' > activity.log

Then I'm not even talking of the need for the jdk for the build to work. I had created as per instructions the JAVA_HOME and also added the proper python, ant, and jdk to my PATH, but still the building of code_swarm wouldn't work. It was always referring to JRE6 even thought all my paths were properly set!!! I found on google I needed to copy tools.jar from someplace to some other place, most likely put it in the lib folder, whatever.

Then, the python convert_logs script doesn't work for me under XP. If I install Python 3.1, it gives a syntax error, if I install 2.7, it gives and indentation error, both around l 433 or 463 (Ubuntu 10.10 behaves like 2.7 - indentation error - I haven't tested Ubuntu more than that). It doesn't work. But of course, I can use the log.xml file from my OSX setup here, whether I'm using the original code or Rictic's fork. In both cases I can't do the python thing so Rictic's taking care of this doesn't work and it's complaining about python. Anyways. But if I edit the .config to input the log.xml file, it's different...

In this case, what I do under XP is I go to the code_swarm folder, edit the sample.config file to input my log.xml from my OSX, and run the run.bat from command inside the code_swarm folder. Yeah, in this case if I try calling the run.bat inside a project repository, it complains it's not finding the .class files that are in the code_swarm folder, and it never creates a .code_swarm folder like it does on my OSX - go figure. So it'll work if I use the original codeswarms_0.1or Rictic's, in both cases using almost the same .config file (except for physics engine Orderly vs. Legacy). But it's visually nicer and different on Rictic's so I'm using that. Visually it's equal to my OSX version, the only thing is I can't enable OpenGL, contrary to OSX. Whether I use the original or rictic's, I get this error when I enable OpenGL:

java.lang.RuntimeException?: You need to use "Import Library" to add processing.o pengl.PGraphicsOpenGL to your sketch.

at processing.core.PApplet.createGraphics(PApplet.java:1219) at processing.core.PApplet.size(PApplet.java:990) at processing.core.PApplet.size(PApplet.java:934) at code_swarm.setup(code_swarm.java:190) at processing.core.PApplet.handleDisplay(PApplet.java:1368) at processing.core.PGraphics.requestDisplay(PGraphics.java:

at processing.core.PApplet.run(PApplet.java:1542) at java.lang.Thread.run(Unknown Source)

The only difference is that under the original code, a little 1 inch by inch java box appears with nothing in it whereas Rictic'c just writes the error and stays there.

So what is this? I researched, I added this to code_swarm.java in / src :

import processing.core.PApplet; import processing.core.PFont; import processing.core.PImage; import processing.opengl.PGraphicsOpenGL;

I tried modifying the build.xml to include more of the stuff I found in the export.txt inside the /lib folder. I admit to not fully understanding what I'm doing here. If I add these import, ant will compile fine with no errors, but I get the same error about OpenGL when I try to do run.bat. I tried modifying run.bat at the java line to add more jar there like the gluegen-rt etc. Same thing.

And no I don't have the voltron lib. What the hell is that anyway. What is the fullscreen library that I could remove to fix this? Those are posts I read on the web. I don't have all the knowledge to understand this java compiling OpenGL related things. Am I missing some file from "Processing" or what?? Or maybe the opposite, I should remove a file somewhere? I would really like any info on this.

Also, I'll dump my env variables under XP:

JAVA_HOME:C:\Program Files\Java\jdk1.6.0_22 PATH:%SystemRoot?%\system32;%SystemRoot?%;%SystemRoot?%\System32\Wbem;C: \Program Files\QuickTime?\QTSystem\;C:\apache-ant-1.8.1\bin;C:\Program Files\Git\cmd;D:\Git_CS\code_swarm\bin;D:\Git_CS\code_swarm\build;D: \Git_CS\code_swarm;C:\Python27;C:\apache-ant-1.8.1\bin CLASSPATH:.;C:\Program Files\QuickTime?\QTSystem\QTJava.zip;D:\Git_CS \code_swarm\build

So for XP, in summary, I run from the application folder using a log.xml file generated on my OSX - because python runs there and does the conversion. And I can't use OpenGL on XP, whether it's with the original code swarm or Rictic's, whereas, I can under OSX.

Finally, I use this to encode the .png files under OSX and it's fast and clean :

ffmpeg -f image2 -r 24 -i %05d.png -sameq filename.mov -pass 2

I took ffmpeg from the package contents of ffmpegX and put it under / usr/bin or /usr/local/bin I can't remember.

So hope this helps someone maybe...

Any help on these issues would be appreciated!!

Thank you :)

BankP

edit : git log --name-status --pretty=format:'%n------------------------------------------------------------------------ %nr%h | %ae | %ai (%aD) | x lines%nChanged paths:' > activity.log

My XP tests were done on a Netbook N270 atom with Intel GMA.

Maybe I'll stick with OSX to run this after all... but if anyone has a "dream" .config file, I'd like to see it...

Thanks!

Read the account of my experiences with code_swarm here:

http://groups.google.com/group/codeswarm/browse_thread/thread/e95b37c372d271a5

Thanks!