FERDY CHRISTANT - APR 10, 2015 (10:49:28 AM)
JungleDragon V4 - update 20 is now live. This update introduces a long-awaited feature called direct messaging.
What is direct messaging?
Until today, JungleDragon had no way for members to contact each other directly. Instead, you communicate via JungleDragon content, for example via comments on photos or via posts in the forum. It is an indirect way of communication.
This is intentional, as I believe JungleDragon discussions should be public, and be related to the goal of the website. JungleDragon is not a private chatroom. However, members have expressed the need to contact other members directly, and there definitely are situations in which this is useful. A prime example would be to connect to another member in your area to go on a photo trip together. This would require the exchange of private information that not everybody should see.
For situations like that, JungleDragon now has a simple and secure way to connect to each other.
Direct messaging demo
When you now visit any members profile, there is a new button in the header called "send message":
This button will only be visible if all of the following is true:
- You are signed in
- You have a minimum class of 2 in the karma system
- This is not your own profile
- The target user (Panda in this case) has direct messaging enabled, which is true by default
The minimum class level is to avoid spammers or bots from hijacking this functionality to send out emails on a large scale. The direct messaging setting per user is so that people can decide whether or not they want to receive such direct messages. Here is that new setting on the user profile:
If all the above security checks are passed, the "send message" button appears as shown. If you click the button, a simple form appears:
Here you just enter your direct message and click "Send message". At this point there is another security check, there is a maximum amount of messages one can send per hour and per day. Normal users will never reach that treshold, this is again a measure against automated bots.
The person you sent a direct message to will receive an email, like this:
The direct message is sent to the user, your email address is not shared in any way with the user. From the email, you can click the reply link to sent back a message immediately:
And this way you can continue to have your private conversation with the other member. If you trust each other enough, you may share contact info this way and take the conversation outside of JungleDragon. It's your choice.
Finally, an important thing regarding privacy: JungleDragon does not store the contents of your direct messages. I have no record of them and therefore cannot see them, JungleDragon only forwards messages. This also means that you cannot see a history of messages sent to you in JungleDragon, that history is in your email only.
With the direct messaging feature, a long-standing request is finally delivered. I have tried to strike a balance in keeping it simple and secure and to make possible the scenario of contacting each other. You can now initiate your private contact with another member from JungleDragon and continue to have the conversation in JungleDragon or outside of it. I hope you like it.
FERDY CHRISTANT - APR 6, 2015 (12:10:22 PM)
JungleDragon update 19 (of V4) is now live. This is a simple update that introduces an overview of members at the country level:
Above is an example for India. We are at the country level (India) and there is now a new "members" link. This overview simply shows members from that country. This is based on said members indicating their country on their user profile, which is optional. The list is sorted by karma, the most contributing users are on top of the list.
As you use this, you will notice that almost all members come from a top 10 list of countries, other countries will have few or no members.
Anyway, the idea of this feature is that it allows you to possibly connect with other members in your country. For that to happen, JungleDragon needs a way for members to directly communicate with each other. That is not yet possible, also not after this update. However, I am now considering to build that next.
The reason I have been delaying that feature, which has been requested before, is that it is a feature that is much harder to realize than one might think. Sending messages between each other is simple, yet it has to happen securely, and it should respect people's email address remaining hidden. Furthermore, it should be resistant to spammers. And then people want notifications, the ability to mute individuals, before you know it, a simple feature turns very complex.
I am going to build some way to communicate with each other directly though. I'm hoping to keep it simple. That will be the next update.
FERDY CHRISTANT - APR 3, 2015 (11:17:45 AM)
JungleDragon update 18 (of V4) is now live. This one is a breakthrough update that introduces the concept of Parks (protected areas) to JungleDragon. This idea was first explained and imagined in my Big Idea post. Although that post focuses on countries, parks are indicated as a secondary goal.
Today, that secondary goal is implemented, partially that is. There is both good and bad news to share, so let's see what we got.
"Parks" are a short term for conservation areas within a country. Those conservation areas can have various designations, they can be a national park, a habitat protection area, a bird conservation zone, many things. They exclude zoos.
The idea was for JungleDragon to have these parks available at the country level. With that, not only can you explore a countries' photos and species, but also interesting areas and habitats. Whether in your home country or a planned travel country, it can help in actually planning trips.
For this to work, we need a data source, the actual collection of parks. After many tryouts and much debate, we eventually discovered protectedplanet, which has this data. As outlined in my previous post, this data is not perfect and not without problems, but it is as good as it gets. This data is the base for the feature I'm about to explain next.
Parks are linked to a country, therefore we must first navigate to a specific country to see them. Let's quickly recap the two ways this can be done in JungleDragon.
See above. You can use the country selection screen to pick any specific country, using various sortings. Alternatively, if any photo has a country linked to it (which is true for about 75% of all photos currently), you can directly navigate to said country from the geo panel on the photo page:
See above, note the "Explore" link that takes you directly to the country, Madagascar in this case.
Arriving at the country level, we see these options:
The new option is named "parks". I have separated the normal map (which shows geotagged photos) from the parks map so that they do not interfere with each other. Another reason is that parks can take a while to load, depending on the country. The parks link for a country will only show when that country actually has this data. This is true for most countries, 225 of them.
So here is the big reveal, JungleDragon Parks in action for Madagascar:
Highlighted in a clear green color, we see both areas (polygons) as well as points (markers) that are considered protected areas, according to ProtectedPlanet. Let's zoom in a little:
As we zoom in, we see that the area shapes are half transparent. Furthermore, clicking on the area reveals the park's name.
I wanna share a story on how useful this feature can be. My girlfriend and I are considering to revisit Madagascar this year. Looking at this map we discovered many areas and parks that we missed during our first trip. Parks that are not in the travel magazines. Parks that were very close to us when we first visited, yet we did not know existed, and thus we missed. We are now using this feature to plan our 2nd trip.
Let's have a look at a few more examples.
Above are parks in India. As you can see, they supply mostly point data, not area data, but there are many of them.
Above is Tanzania, which seems quite well protected.
Above is a section of Australia, to demonstrate that protected areas can both be land or marine.
Above the highly complicated geography of Indonesia, with its wide range of conservation areas.
The bad news
I am quite proud of the above result. It took some soul crushing effort to realize. Unfortunately though, it is not perfect, there's a serious limitation to discuss.
The visualization of these parks is implemented as a KML layer on top of Google Maps. You may know KML layers from Google Earth, where they originated. In order to load such KML on Google Maps, one is restricted to certain constraints: a maximum of 1,000 shapes, a maximum file size of 3MB/10MB (compressed/uncompressed).
In the data set provided by ProtectedPlanet, there are 45 countries that do not fit within those constraints. I cannot lift those constraints, therefore my only remaining option was the delete selections of data for those countries specifically. I used this process to "smartly" delete parks:
- Delete by IUCN category. Category I and II are more important than III, IV and V.
- Delete by designation. For example keeping national parks, yet deleting "protected habitat"
- Delete by size of the area.
This was a truly painful exercise as I had to willingly delete useful data just to fit inside the constraints. The result is that some countries took a heavy hit. And as a result of that, for those countries the parks feature has become less useful, or less complete. A handful has even become close to useless.
As said, we're talking about 45 countries here, all other countries (180) have their full data. Out of those 45 countries, about half had acceptable deletions of parks, where the most important parks remain, and thus the feature remains usable. 20-25 countries though, had deletions that truly damage the usefulness.
And some of those countries are very important, making it extra painful. A prime example is the USA. This country had so much data that I had to delete 74 out of 75 entries. The end result is horrible. And it can get worse. For countries New Zealand and Finland, I had to delete all parks. Not a single area would fit in the constraints due to their polygon data being simply too large, even for just a single park.
So that is it, 20-25 countries are partly or barely usable, whilst ~200 countries are fine. It's a matter of seeing the glass half empty or half full I suppose.
Having worked with this data now for a few weeks, and due to the above limitations, there's another consequence. For now, I will not pursuit integrating photos with parks. Parks will remain on their own. The reason for that is that if I do some simple number crunching, even if I manage to auto detect a park from a photo's GPS coordinates flawlessly, it would only work for a too small percentage of photos, due to the nature of the data and the popularity of specific countries in JungleDragon. I may come back to this decision one day when we have more content, but for now it is too much effort for too little value. I have other things to do.
That was it, the good and bad about JungleDragon's all-new parks feature. I think it is a very cool extra. Not only can we now explore a countries' photos and species, we can also explore its natural areas, and plan to visit them. Remember that only a few weeks ago, the concept of a country did not even exist in JungleDragon, yet now we can do all these things.
As said, how well it works depends on the country. It works perfectly for 180 countries, good enough for 20 countries, and in limited/poor ways for 20-25 countries. This is the maximum I could get out of it, so I hope you appreciate it.
FERDY CHRISTANT - MAR 27, 2015 (12:34:10 AM)
Back in January, I wrote the Big Idea post, in which I set out the vision and focus for JungleDragon V4. Summarizing that post, the big idea was to dramatically improve the ability to explore wildlife geographically, by country. To realize that vision, I needed to implement the concept of a "country" in rich ways.
I'm very happy that in the past 4 weeks alone, much of that vision was already realized. We can now easily link photos to countries, and as a result, countries become rich things. Where before they were map pins only, they now have a photos section, and a species section. This in itself delivers on the core promise of JungleDragon V4 already.
However, there was more in that Big Idea post: I also hinted at the ambition of introducing the "Park" concept. Countries have national parks, and it would be great to have a representation of that in JungleDragon, one level below the country level.
I'm working on that concept right now, and it's still not entirely certain whether I will succeed. In this post I am going to share you everything you want to know as well as everything you don't want to know about my current approach, its many struggles and its promises. I'm not sure if it will be good read, but I really feel that I should write about it, to share both frustrations and some slight successes.
The idea of "Parks" in JungleDragon is not new, I already started work on it during JungleDragon V3. My approach was to pay somebody to collect data about all national parks and zoos in the world, along with some metadata. The idea would then be to integrate them in JungleDragon, but only in limited ways:
- To display a list of parks per country, purely for information purposes
- To display them as pins on a map (yet not as actual areas on a map)
As for linking them to photos, automatic detection of parks from a photo's geo coordinates would definitely not be possible, since we would have only point data and not area data for parks. That would mean that users have to manually associate photos to parks, if they can be bothered to do so.
This approach did not work out, due to the freelancer not delivering and missing deadlines for a period of about a year. In hindsight, not a bad thing, since the new approach has more potential, yet way more challenges.
The new approach
As I was discussing the former approach at the JungleDragon forums, one member (@Lilygirl) pointed out that IUCN could perhaps have a ready-to-use data source for national parks. After a bit more research into that, I came across this:
I don't know why I did not hear about this before, but it's quite a spectacular find. That site shows a map with all protected areas of the world, including shape data that actually draws the area on the map. The specific visualization on that site is powered by a database called WDPA: the World Database of Protected Areas.
Such a promising data source had me itching to take a deep dive into it, to see how and if it can be used for our purpose at JungleDragon. As such, this new data source became the starting point of my new approach.
About the WDPA
Let us first get out of the way a few basics about this data source...
First, the type of data. The database stores "protected areas", which differs from my first approach of national parks and zoos. Zoos are out, it's that simple. They are not considered protected areas, although some may appear in the set if they have a hybrid status.
I can live with that. What about national parks vs protected areas though, are they the same thing? No. Protected areas have a far wider scope than just national parks. "Protected areas" can mean many things in this database, it also includes many areas that one does not consider a national park. A protected area usually has a relation with nature conservation, but that relation can be strong or weak. For example, a protected area can be a floral protection zone, a game zone (which implies managed hunting), and many other things. There is no rock hard definition, although IUCN attempts to classify these area in various ways, with mixed success.
So we're looking at way more than just national parks. How complete is this data? The answer here varies wildly country by country. To give you an example, the USA has a whopping 75,000 entries in the database. And some countries have less than 10. You can't explain such differences by the size of the country alone, clearly there are enormous differences in how well individual countries are described. This is somewhat to be expected, as this data is delivered by said countries, there is no single central authority.
So it looks like some countries are"overcomplete" for our purpose, whilst other are "undercomplete" possibly. So that's data quantity, what about data quality? Again the answer here is mixed. In some countries, a lot of metadata is lacking. You can also see enormous differences in the accuracy of the area shapes. Some countries map them down to the pixel, whilst other draw large rough edged borders around areas.
Another thing I spotted is that some areas are not named as you would expect them to be named. Looking up some famous areas I came across names I did not expect, and most people would certainly not recognize.
So there's problems with this data. The quantity, quality and usefulness varies greatly between countries. Despite these problems, I don't think we're going to find a data source that is better than this. I certainly would not know of any. I also will absolutely not manage or maintain protected areas inside JungleDragon itself. In other words, the data is the data. We're making use of it, we're not maintaining it. With that principle in mind, I started working with the data...
Before we get to the nitty gritty, I'd like to list the functional goals I have for this data,what it can possibly realize inside JungleDragon if all goes well:
1. Listing protected areas per country
This feature would simply list all protected areas (their names and other metadata) as text entries per country. It would be for informational purposes only, to learn about the names of protected areas in any specific country. This feature, as basic as it is, is achievable.
2. Drawing protected areas on a country map
This variation would actually draw the protected areas on the country map, which is a lot more useful. Not only will you learn the names of the areas, you can actually see where they are and what their borders are. This feature too is achievable based on my current progress, yet with a major limitation, that I'll discuss in a minute.
3. Linking protected areas to photos
If and when we have protected areas inside JungleDragon, they would become a lot more valuable if they can be linked to photos. This would make it possible to see photos and species per protected area. Without that link, protected areas would just stand on their own.
You can imagine a photographer picking a park from some kind of list, filtered by country, to associate the photo with the specific park. This approach has so many problems that I will probably not build it. The amount of entries is too large in many countries, their names are not always as you expect, and photographers already need to set the species and country on their photos, which is challenging enough to enforce. This process would be to laboursome and too error-prone.
Which leaves one other option only: automatic park detection. For geotagged photos, the system would automatically calculate inside which protected area shape it would fall, and then automatically settle on a specific protected area. It would be quite an awesome feature.
Whether this can be technically done, is still to be investigated. I have great concerns about the feasibility, performance issues and accuracy. Furthermore, shapes do overlap which can cause additional trouble. Even if I succeed, there is the additional issue that it will only work on geotagged photos, which is about 50% of photos currently.
This goal #3 is the most uncertain goal at this point. The jury is still out on this one. Worst case, only #1 and #2 will be delivered.
Working with the data
One step at a time, I haven't arrived at goal 3 yet as I'm still struggling with #1 and #2. Let's see where we're at and what I did...
First, getting the data. Luckily, protectedplanet allows you to download their data (under conditions) in a number of formats. The one I needed was the KML format, which is an XML dialect to represent geographic shapes. This format is also used by Google Earth as well as Google Maps.
Protectedplanet offers a download of all protectedareas, which I did. It has about 75,000 entries. In addition to that, you can also search for specific countries and download only the KML for that country only. I did that as well for a few countries and something did not add up at this point. The total number of protected areas after downloading a few countries was already larger than the sum total in their single "all" file.
I'm not sure what the reason for that is, but it's highly confusing. And painful, as now I had to take the ISO countries list in hand and manually download the KML for each country, which brings the total number to about 225,000 protected areas. This results in 230 files, as some countries have no protected areas at all.
After the download exercise I started investigating the structure of the data. I had to do this on "small" countries as the larger files will crash any text editor, some are over 2GB in size.
Once I had a good idea about the structure of the data, I started writing a parser that could pull the KML into a relational database. This will be needed later on to integrate it into JungleDragon, plus it is useful to be able to query the data. Writing the parser was painful and I had to start over 3 times due to the enormous size of some files which would crash my script.
I eventually managed though, and had all entries of all countries sitting inside a database. At this point it was easier for me to investigate the actual data in large quantities. Unfortunately, I discovered new data issues as a result. First, WDPA has a data specification, which describes the purpose of each field. The problem is that the enforcement of the definition is very weak.
As an example, they have a field "wdpaid" which supposedly is a unique number given out to protected areas. Except it isn't unique, as there are several areas sharing the same "unique" ID. Likewise, they have a field "designation" which indicates the type of the protected area. An essential field for our purpose, but it's not standardized. Anybody can type in anything in that field, and they do. And many do it in their local language, so forget using this field as any kind of filter. And to make matters worse, many potentially useful fields are often set to "not reported".
And there's more bad news. It gets worse before it gets better. I talked about goal #2, which is to draw the protected areas on a map. The opening screenshot shows that concept for the example country Sri Lanka, where you can see the blue shapes drawn on the map.
That concept is implemented as a KML layer in Google Maps. The problem is that KML has constraints on it. In order to draw such a layer on Google Maps, the KML itself has these constraints:
- The KML file has a maximum file size of 3MB compressed, and 10MB uncompressed
- The KML layer has a maximum of 1,000 shapes in it
Go beyond those constraints, and nothing will get drawn at all. How do they do it at protectedplanet then? They don't use Google Maps, they use OpenMaps combined with a tile server. That's not an option I can consider, not only would it not be compatible with existing JungleDragon functionality, it also would be a major piece of infrastructure and investment to add.
So I have to stick with the constraints. In a way I don't have a problem with the constraints. I intend to show these areas per country, so not on the global map. Making it have less than a 1,000 entries per country seems reasonable given our original goal. Furthermore, it would not be sustainable at all to let users wait for a 100MB (even up to 2GB) of data just to draw this layer of shapes. It would crash browsers even.
The real problem is deciding which protected areas to delete. There are 41 countries exceeding these tresholds and I have to bring all 41 down to less than 1,000 entries per country. In some countries, that means getting rid of half the protected areas, in a country like the USA it means getting rid of 74 out of every 75 protected areas.
This is the step I'm currently in, and its troublesome. I can't go in and manually decide which areas to keep, there simply are too many of them. So I have to throw out protected areas smartly, for each of these 41 countries.
One major way in which I'm trying to do this is to filter them based on metadata. For example, I can use their IUCN category and only keep category I and II (most important), but that still leaves too much data in most cases. I actually have to decide on this smartly, one country at a time. The problem occurs when after "smartly" filtering, it still leaves more than a 1,000 entries. It's an impossible job to decide which one to keep and which one to delete. This is my major challenge currently, and I'm only still at the start of it.
This is a hard point to cross, that will take me a while. The good news though would be that once I overcome this part, goal #1 and #2 are guaranteed.
We've arrived at the state of progress that I'm currently in. As you can see, it is a road full of challenges. My take on the goals discussed earlier:
- Goal #1: listing parks: Yes, can and will be done.
- Goal #2: drawing parks: Yes, can be done but with major data selection challenges
- Goal #3: linking photos to parks: remains to be seen, to be further investigated
Despite all the setbacks, I believe I will deliver #1 and #2, with #3 being a stretch goal that may or may not be delivered. Although parks potentially is a useful and cool feature to add, I do see it as an "extra". A powerful and cool extra, but JungleDragon can do without it. With that I mean that the combination of species and countries already offers an endless and rich way of mapping and exploring wildlife.
Whatever happens, you can be sure that I will give it my best shot. I've had several nights where I regret even starting this, but now I am in too deep to just stop. I hope I did not bore you too much with this post :)
FERDY CHRISTANT - MAR 16, 2015 (09:43:47 PM)
Two updates in a single day, I hope you can keep up with the breakneck speed at which JungleDragon V4 is being developed. This update brings to life an important vision for JungleDragon V4, and JungleDragon overall.
Update 15, two updates ago, finally introduced species at the country level. It came along with a new concept, the country introduction (first photo of a species within a country). Whilst country species is a major breakthrough for JungleDragon, this specific update makes it infinitely more useful.
This update is the icing on the cake, as you can now filter country species by popular wildlife categories:
See above. At the country level, below the species link there is now an additional navigation bar. You start navigating all species of the country, yet you can filter using one of 8 categories.
Compared to just showing a large list of species for the country, this is a lot more powerful and intuitive. I'm going to show you an example of each category just ti highlight the importance of this. Each example is clickable, and will take you the real thing.
As the above closeup introduced, here's amphibians in India:
The incredible insect life of Sri Lanka:
The wide array of exotic birds in Australia:
The breath-taking marine life of Belize:
The mysterious world of fungi in the USA:
Tanzania's world-famous mammal life:
Bulgaria's extraordinary wild plant life:
And South Africa's intruiging reptile community:
As said, I'm showing so many examples to demonstrate how natural it is to look at wildlife this way, by country and then by species category. I hope you share my excitement regarding this update.