Visualization Sprint

Global Water Experiment

Visualization Sprint:
an experiment in collaborative
data visualization
Visualization Sprint Process Outline

Global Water Experiment

Coders, designers, and data experts: contribute to a collaborative effort in data visualization. Show off your skills, discuss technique, or dive into a new HTML5 toolkit by participating in our first "Visualization Sprint." One lucky contributor will be selected at random to win a pass to this year's sold out Eyeo Festival and all contributors will be credited on the final project.

This first Visualization Sprint was conceived in response to a massive citizen science project involving 75,000 students from 80 countries around the world who have been collecting samples of water purity, pH, and salinity for the past year. The project is called the Global Water Experiment and it's organized by UNESCO and IUPAC as a central activity of the International Year of Chemistry. In keeping with the spirit of how the data was collected, we've devised this Sprint as an experiment in collaborative data visualization. Our goal is to have a finished visualization of the students' data that we can share with the world on World Water Day (March 22).

Jan Willem Tulp, the winner of our Eyeo 2011 Challenge, has kicked things off with a sketch using d3.

Data: we've posted an XLS version of the data set for exploration. URLs for JSON versions of the data are available in the Resources box when you fork a version.

How to participate:
  • Everyone should check out the edits made so far, vote for which modifications are successful, and comment on the direction of the visualization
  • Coders are encouraged to fork an existing version and improve it, or submit a new "alpha" version to start a branch from scratch
  • Data scientists can analyze the data set and send us your revised version for others to use

Explore Versions

Comments

vlandham's picture

Soooo...
who got the ticket to Eyeo ? :)

De Datagraaf's picture

@Charlene. :) Great reading that! Thanks for quoting it.
I really enjoyed contributing to this project. It's good to see that spontaneous cooperations like these can work so well.

Charlene Manuel's picture

@De Datagraaf - The organizers were thrilled! I'm not sure if you saw the comment from Fabienne from the IYC so I wanted to post it here as well:

Fabienne Meyers (19 hours ago)
I have been watching the sprint from the side line, and all I can think is that chemists have much to learn about collaborative work, and seeing what you guys (and girls!) have done is quite amazing. This sprint is an amazing way to wrap-up of the Global Experiment today, just a year after it all started on World Water Day 2011. Thank You!
Fabienne, for the IYC team @ chemistry2011.org

De Datagraaf's picture

@artzub Thanks for replacing my brute country name replacers by that shortened version. I had seen it in the map initialisation routine, but I only got how it works when I saw it on it's own in your last version.
wim

De Datagraaf's picture

@Thanks Edward, thanks for liking our work! I hope the experiments organizers like what we came up with.

Regards,

wim

De Datagraaf's picture

@artzub Thanks for those corrections. I tried to fix as many problems and kinks as I could, but it has become quite a huge file, so I am sure there are still a lot of smaller and bigger glitches in there still (like the graph axes texts disapearing every now and then).
Still, I think we can be pretty pleased with the final products.
I even put in country selections at the end. If we had a bit more time still, we could have included your country level graphs aswell. (By having the dots for one country move to the graph view, while adapting the y-axis length to the number of measurements.)
Who knows maybe we can still do that. I don't know what Edward has in mind for this project in the future.

Regards,

wim

artzub's picture

@De Datagraaf last version really cool! If look on the powerful computer then animation is mesmerizing =) Nice job!!!

Edward Lee's picture

@DeDatagraaf - Nice work! I'll update the final project with this latest code.

And excellent effort to you and everyone in both exploring many designs and then focusing on a final, polished piece. We're really thrilled by your dedication!

De Datagraaf's picture

Hello Edward, I havemade a version that is much more consistent when it comes to zooming and panning. Hope you get a chance to merge this into the presentation version.

Regards,
wim

De Datagraaf's picture

Edward, I am guessing you are talking about any new updates I come up with in the near future. I will post them if any. Meanwhile I posted a message on the d3 google groups, asking for help in restoring the zoom function. Hopefully I will have an answer by tomorrow morning (Amsterdam time).
Regards,
wim

De Datagraaf's picture

Hello Edward, great you like it! It has been a great project to participate in.
I will be happy to make a new version, but I am not exactly sure what you mean by that. Do you want to me to make a new alpha version of the version I posted last? Or are you referring to any additional changes I wont to make as of now?
Regards,
wim

Edward Lee's picture

Thanks for all your fantastic work, everyone. The project looks great! I've made the "final" version into a normal project page here: http://www.visualizing.org/visualizations/sprint-final-project-global-wa...

@DeDatagraaf: If you want to make the last few updates as a new version, I will port it over to the visualization. Getting the zoom feature back would be great, but I can't see where the bug is. And I know the messiness of the data has been a difficulty: we did a bunch of initial cleanup, but I think this is just the nature of a big citizen science project.

Thanks again to everyone who participated!

De Datagraaf's picture

Sorry 'bout that last minuite version. Just had to add New Zealand to the frame. Couldn't leave it like it was.

De Datagraaf's picture

Oh, in both artzub and my versions Fiji and Kuwait are still located in the Atlantic. I guess it is possible to write a stop-gap routine to sift these errors out. But it would have been a lot easier if someone had cleaned up the data set at the beginning of this sprint.
Anyway, this would have to be dealt with before anyone presents a graph to UNESCO.

De Datagraaf's picture

Shoot, somewhere on the line the zoom in my version got broken. It's not very important since in my graph the focus is on the global situation, not on what happens on the country level. But it is never the less somewhat annoying that I can't find the fault. Does anyone see what is going on?
The other thing I am trying to solve is the data not properly getting read, because of the errors made while people entered some of the country names. I guess I use the correction routines the wrong way, but I can't figure out what it is I am doing wrong.
Those are the last major corrections I would like to make to my version. Hope someone can help me out...

Regards,

wim

De Datagraaf's picture

@artzub Yw! See you around then, man!

De Datagraaf's picture

@editor I see there are 0 days left. When will the actual deadline be? Is there still some time left for a final version? If so, I would like to try and try and iron out one or two last issues. If not, my current version will have to do.
Regards,
wim

artzub's picture

@De Datagraaf Thanks for the nice review. I'm going to read what write in d3.js group on google.

All simply brilliant =)

De Datagraaf's picture

@artzub :) I have been staring at this stuff for too long! While looking for an elegant solution to the data update problem, I had completely forgotten about the one from earlier versions: just remove all the circles and refill them. It took me quite a while before I realized you never stopped using this method :) I'm done for today!
Regards,
wim

De Datagraaf's picture

@artzub Good to see your new version! You did a great job ironing out all kinds of problems. It's a pity we didn't manage to merge both our work into one version - I think our respective versions are really complementary. Together they are a pretty decent presentation of these datasets.
But, as you say, it was time consuming enough producing the stuff we managed to produce, let alone keeping things synchronized.

Still, what I would like to transfer from your version to mine, is the data refresh routines. At the moment I am trying to figure out how you managed to make the circles update properly when switching datasets.
If I am done doing that, I can call it a day too.

I enjoyed working together aswell. This project has been a lot of fun, and I think it marks the end of my days as D3 beginner. (Still a lot to learn of course.)
Hope to see you around (in the d3 google group maybe?),

Regards,

wim

artzub's picture

@De Datagraaf Your findings really interesting. Your penultimate version, where the delay is reduced — mesmerizing. But there are many problems.
I'm sorry, what i don't connect to your branch. Since i have already far behind your change.
Completed the mine. In general, I like it.

It was a pleasure working with you, learned a lot, and generally learned d3.js =)

artzub's picture

@Fangoria this is not the first error in the data I have already written about Georgia, a country which is not the state in the U.S..

De Datagraaf's picture

@Fangoria It would be great if someone did a thorough, systematic check of the data... ;)

Fangoria's picture

some locations are clearly wrong. for example the two point in the ocean by the West African coast can not be Fiji and Kuwait.

De Datagraaf's picture

@jerome & artzub
There´s a couple of things that still are not the way they should be. The timings of the transitions is one thing - but I have to say that I leave stuff like that for last most of the times: first I want to have things working, and only after I take care of the cosmetics and the exact timings.
And I haven't arrived at that just yet.
At the moment I am trying to have the y axis pop up when I want it to, and I am trying to scale all data properly. The last bit is quite tricky, but I think I can make it work. Especially sin

But styling the y axis is quite difficult, or at least not as straightforward as the styling of other elements. (When that is finished I would also like to add labels on the x axis.)

You are probably right about the mouseovers versus the mouseup or mouseclicks. (I wasn't sure just yet whether buttons were the way to go (but I guess they are).

What I haven't managed is to make selections by country. I don't think it is all that hard, but I just didn't have the time for it.

The most important thing that is still wrong about the graph, is the updating of the circles. As far as I can tell, everything works alright as long as the graph morphs between different sortings of one variable; but it goes awry as soon as I switch between variables (ph, salinity and dirt). I have tried to bind the data by means of just about every field (school, teacher, coordinates, etcetera), but none of them work consistently. Using d.coordinates works best (when staying with the same variable), but when I use it for switching between the research variables it results in the switching between the sortings no longer working.
I really could use your input in this.

Maybe I will leave the scales for now, and do something about those transitions.

Talk to you later...

Regards,
wim

artzub's picture

@jerome cukier I do not like it, too, and for this I'm working on optimization. Mad delays irritate users, which does not contribute to their re-return.
@De Datagraaf The idea certainly is not bad but a lot do not understand! It has become better with the addition of the y-axis.

jerome cukier's picture

you should try with a much lower duration, like 500~1000. 4 seconds is really a lot.also experiment with different easings. finally it would make more sense to trigger this on click (or, say, on mouseup) than on mouseover bc the transitions fire up somewhat unexpectedly

De Datagraaf's picture

okay, corrected that scaling thingy, the data points now stay within bounds.

De Datagraaf's picture

@all It is now possible to do all kind of sorts, and then return to the mapped data (I make a copy of the coordinates to achieve this).
Still trying to figure out how to add axes to the sorted data.

One thing that is a bit weird, is that the data scales differently on my home computer and on this site. Here at home the sorted pH data has the right margins left and right; on the website the data is too far to the right. Don't know what happens there - will look into it.

De Datagraaf's picture

Just to make sure: I partially corrected the variable switching routine, but there are still some loose ends there.

De Datagraaf's picture

@all Nope. Still something goin wrong there...
The dots move properly when staying with one variable - they don't when switching between variables...
I'll be working on that. Any help appreciated of course.
Regards,
wim

De Datagraaf's picture

@all Last night I struggled getting the circles to update properly after being sorted. This morning I think I managed to do so. My best version (51) to date, I think.
Please, have a (critical) look, and let me know what you think.
(And I would love some pointers towards adding the proper axes and labels.)

Regards,
wim

De Datagraaf's picture

@artzub I will be working on the graph off and on for the coming days, but I will have some more time next week. Hopefully we can produce something slick before the deadline comes up.
I have to admit I am quite happy with the new stuff in my current version (#50). But it will still take some doing before it can be called a finished product.
The next thing I have on my to-do list, is to add your selections by country to the moving circles.
But I'm guessing adding scales and axes is going to be the trickiest task that lies ahead.

(Funny you mention the bikers competition - one of the things I am working on is a program for presenting scores from a bouldering competition.)

Regards,
wim

artzub's picture

@De Datagraaf I'll say easy: So cool!!!!
Is finite, is now the main focus on integrate of all the features on together and put everything in order.

I can only in saturday start working on it, because i now writing programm for competition on the biker-cross. The software for judging.

Your made more numbers of change... It will be interesting to study them.

P.S. I always have positive thinks — easier to live.

Regards,
Artem.

De Datagraaf's picture

@artzub Thanks for the positive response, artzub. Since we have under a week to go, I thiknk the best thing would be to focus on getting all the good stuff integrated (instead of working on 'competing' versions).
We can look into layout and other details in the final days (i.e. color choice, exact placements of all elements, wording of the header, etcetera).
I am working on a solution for the details window that is rather different from what is in version 49, so don't pay to much attention to what's in there (I think you will like the new suggestion).
Another thing I would suggest, is to integrate the interaction elements for the main window with those in the details window - there is some redundancy in having multiple drop down menu's with more or less the same choices.
One solution I am working on, is to have the pies do something similar as the histogram: mouseovers on the slices hide and show different subsets of the data. But this is quite intricate. I can make the basics work, but optimizing it will take some doing, and I might need some help with that. I'll post an example later on today.

Regards,
wim

artzub's picture
artzub's picture

@De Datagraaf Well, i'll look into your code. Make it load all the data in one go is a good solution for performance. I integrate your solutions in my code.

Make the name of the city at the same distance is a good idea, but it must still finalize the details of the window to display more correctly, I'm working on it.

I did the story change your version. To make it more clear what and where. https://github.com/artzub/Global-Water-Experiment/commits/master/wim.html

De Datagraaf's picture

@artzub I like your sorting routines very much, but I haven't managed to incorporate them into my versions yet. Meanwhile I have reworked the code a bit (see version 49) so that all data from the three experiments is loaded in one go, and the circles are now proper D3 in the sense that they enter, update and exit in a truely D3 fashion. I don't think it is very hard to copy this to your version.
I am also working on some nifty ways to switch between the map and your details window. I want to experiment some more with these before I post them. Hopefully you don't mind waiting a bit, so that we can fit them in one of these days.

Regards,
wim

Roxana Torre's picture

Version 48 is based on version 40 + some elements of 47 (country details) a bit modified.
(and no dino's)

artzub's picture

@De Datagraaf Okey, idea, make window of details of countries smaller and save useful functions, is good idea. I will try merged our branches.

About color need think. Your color very sharp, mine — very soft, need to find compromise.

De Datagraaf's picture

Well, I don't have much time today, but I do want to post a new version.
(So, maybe things are work out as far as they should have been.)

Most important change: I have added pie charts to indicate the watertypes and -sources. These pies could replace the drop downs (i.e. a mousover on a pie could filter the appropriate types and sources).
The pies are still somwhat sketchy, but it's a start.

I changed the pH colors back to 'my' red, green and blue - they are a bit clearer and they are consistent with the use of colors for the other experiments (generally green indicates more or less save levels).

The addition of the city info is a good idea. But I found the window for this too big and not very consistent with rest of the layout. So I might a couple of changes to that.
One thing that should be done about the city info panel, is that it should be made to update properly when the user switches between experiments (I think the 'pH' label, for instance, is hard coded at the moment).

I corrected the positioning of the newly added tooltips (they dropped off the screen on the left).
I wonder: couldn't we make the tooltip code a bit more compact? First there was only one location in which the tooltips were controlled, now there are three of them.

There's probalby more. But I have to rush (family visit).

Regards,

wim

De Datagraaf's picture

And another query for artzub: what did you change in your last version (align circles in the center graphic)?

De Datagraaf's picture

Ah, okay, so if I fork version 37, I will be alright then(?)

artzub's picture

@De Datagraaf The error was in my filters for the data of countries. I looked on your code, how you filtering data salinity, but i've checked wrong parameter. I'm checked d[firstVar], but need check d[secondVar].

De Datagraaf's picture

@artzub What was the error in the salinity filter you corrected?

artzub's picture

I found mistake in data on pH! Coordinates the country of Georgia are specified in the USA. You can look it, if click by on USA and see first city.

Of course, this is partially to blame myself. Since copying the names of countries, depending on the coordinates. But the first source of a gross error.

artzub's picture

And what kind of bug with the zoom when double clicked?

Add comment

Login or register to post comments

How It Works

A "visualization sprint" is a new collaborative design and coding experiment where contributors work together to find the best solution, or solutions, for visualizing a data set. Taking cues from citizen science, open source culture and code sprints, Visualizing.org has conceived of this format in order to promote community dialog, learning, and feedback about visualization techniques.

Starting from an initial sketch written in a popular scripting language, anyone is invited to "fork" the code — adding to or modifying an existing version and then posting it back to the sprint. Any version can be forked, and new "alpha" sketches can also be submitted to start a new branch from scratch.

Check out the project prompt and data set at the top of the page, then look at the initial sketch. What are we trying to visualize here?
Explore other contributions: what are the branches the design has taken? where does the visualization still need work?
Add your voice to the discussion by voting versions up or down and commenting as you explore.
Choose a version to modify and click the "Fork This Version" button.
Work on the code, testing as you go along. When you're ready, commit your contribution along with a short description.
Repeat from step 2!

More Info

Is this project only for programmers?
Sort of. The purpose of this sprint is to explore and discuss techniques in interactive visualization, and it uses simple scripting languages so that we can easily see and modify the code. If you're not a coder we encourage you to participate by voting on the versions, adding your voice to the comments, analyzing the data set, or even sketching visual design directions that others could help realize.

What resources are available for my version?
When you fork a version, the top of the edit form will have a list of libraries and data sets you can use. If you need help figuring this out, just send us an email.

Can I transform the data set?
Yes! You are encouraged to transform and analyze the data set. If you modify the data in a useful way, send us the updated file so we can share it with everyone working on the sprint.

How does the sprint end?
The focus here is on process, but we also want to have a nice visualization (or several) to present on World Water Day. Hopefully, the consensus of all participants will bring the most promising branches to a state polished enough to be called "final."

Is there a prize?
Because this is a collaborative effort, no "winner" will be chosen. As a bonus for participating, however, we will give away a prize to one participant chosen at random: a pass to this year's sold out Eyeo Festival.