administration mode
Pssst...Ferdy is the creator of JungleDragon, an awesome wildlife community. Visit JungleDragon

 

JungleDragon 4 - Update 3 »

FERDY CHRISTANT - JAN 9, 2015 (01:52:58 PM)

Hereby the 3rd update since starting JungleDragon V4 development. This update dramatically improves personal tags.

What are personal tags?

Personal tags work like this: imagine a tag named "Tanzania", which is in use by the community. It has a thousand photos in it. Clicking that tag will show all photos within the tag, of all users that have used that tag.

If you have used this tag, say you have contributed 200 photos to it, there's a way to show only those 200 photos. These are your "Tanzania" photos.  To do so, one can visit the "tags" link in any given user profile:

The overview shows all tags that the user has used. After clicking a tag, only photos by that user within the tag are shown. Thus, we have the benefit of having shared tags, whilst also the ability to have a personal perspective on tags.

What was improved?

Before, the overview of tags on a user profile were sorted by the tag's size (number of photos in it). The problem with this is that it makes it vey hard to find any specific tag. As an example, a heavy user like me has used thousands of tags, finding one specific tag of which I know the name is difficult. 

This is now solved by adding a new sort option. By default, personal tags are still sorted by size, but there's now also an "alphabetical" sort option, as the screen above shows. Another improvement is made when clicking such a personal tag:

We're now looking at the personal tag "Tanzania". Before, this one was sorted "newest first". This is still true, however, an extra option "oldest first" is added:

See above. We're looking at the same personal tag, but this time sorted "oldest first", thus in the order they were posted. Note that I am using the post date for this. It would have been better to use the photo's capture date, but unfortunately this depends on EXIF data which may or may not be available. 

A third improvement is that I added a "play" button on the sort bar (to the right). This way, you can play your personal tag like a slideshow, and it will take into account the sorting you selected:

See above the new slideshow, with a proper title that indicates the tag name and the user's name. It has the full functionality of a JungleDragon slideshow, including the ability to add "autoplay" after the URL. 

Why is this useful?

This series of improvements is aimed at the particular scenario of being able to share a subset of your photos in the correct order. You are now able to easily find that subset, and to sort it correctly, after which you can share it. For example, given proper tagging I can easily share somebody all my photos within a country, a park, a photography style (for example macro), and present them in chronological order. 

Karma upgrade

I've doubled the image karma reward for promoted photos. Thus, a promotion will now have more impact on the photo's ranking in various aspects of JungleDragon, for example the best species photo.

Tech notes

A few technical remarks that come along with this update:

  • I've switched MySQL connectivity from MySQL to MySQLi. It seems to hold up so far.
  • I've carried out a SQL injection vulnerability test, JungleDragon did not break in any case.

JungleDragon 4 - Update 2 »

FERDY CHRISTANT - DEC 29, 2014 (08:14:19 PM)

Here's the 2nd JungleDragon 4 update, as I planned to start development in 2015, this one is a bonus as well. This update makes a single improvement, yet an important one...

Species identification

One of the most important user actions in JungleDragon is to identify the species shown on the photo, as this builds a rich connection between photos and species info, furthermore it makes possible species browsing. Given that you know a species, it is as simple as entering the species name and hitting [Enter] or clicking "Search":

Next, you'll see the message in grey above along with the spinning wheel, telling the user that JungleDragon is processing the action. The problem is, the action takes quite long. Before today's update, I timed it at 16 seconds for identifying a "Sri Lankan Elephant". 

I haven't had any complaints regarding this long wait time. I think I understand why. First, the spinning wheel is a good indication that the system is busy. Research shows that users are particularly annoyed in waiting when they don't know what the system is doing, or when there is no feedback that their request is received in order. In this case, the feedback is clear: "we're working on it". Second, the user may imagine that this "species searching" action is hugely complex, which justifies the wait time to them.

Whatever the reason, the wait time started to annoy me, and particularly so when you identify a series of photos. So I set out to improve it.

The result

I can be quite brief about the result: species identification is now near-instant, meaning 1 or 2 seconds. As this depends on the species you identify (in which part of the species tree it occurs), I used the exact same species "Sri Lankan Elephant". It took about 1 second, which obviously is a vast improvement over 16 seconds. Very roughly speaking, I think species identification just became 10 times faster!

Root cause

Only read this part if you're interested in the internals of JungleDragon. In case you're curious about why it was so slow and how it is solved, here goes...

What happens if you attempt to identify a species in JungleDragon is that indeed it will search for a matching species record, first in JungleDragon itself, and if that fails, at Wikipedia. That search is not what is taking all this time though.

JungleDragon species are placed in a tree, which allows for drill-down navigation in the tree of life. As you identify a species on an image, you are making a change in that tree, which requires parts of it to be recalculated. For example, the "Sri Lankan Elephant" example is placed in the tree as follow:

Animalia##Chordata##Mammalia##Proboscidea##Elephantidae##Elephas

Those are the 6 taxonomy levels assiged to that species, and as such it is placed in the tree. Now, as were are making a change in the tree (adding one image to the species), we have to recalculate the following:

  • The number of images in each node
  • The number of species in each node (this may be a new species)
  • The best image (judged by karma points) in each node

This works in a bottom-up way: we first start recalculating the counters and "best image" for  "Elephas", and keep going up until we reach the top of the tree at "Animalia". Therefore, in total this concerns 6 x 3 = 18 operations to update the tree.

By far the slowest operation in this set is recalculating the "best image" for each node. The further we go up, the more images it concerns. Yet it is an operation that needs to be done. What I did to so dramatically improve performance is to stamp the node paths not just on the species records, but also on the actual image records. This way I can very easily calculate the best image for a node path, without a need for all kinds of table joins.

I hope that makes sense. If it doesn't, no worries, just enjoy the major speed improvement :)

JungleDragon 4 - Update 1 »

FERDY CHRISTANT - DEC 24, 2014 (10:40:24 AM)

As announced in JungleDragon's last strategic update, Q4 of this year has seen no JungleDragon development due to me being occupied with travel, a training for work, and a few other tasks I had been delaying for too long. Plus, I needed some time to recharge, it's been a very busy year.

A little earlier than planned, I am hereby kicking of JungleDragon V4 development. This first update is a small one, and focuses on addressing a few issues collected during the last months:

  •  When identifying species on a photo, users have the option to indicate that species identification is not applicable, for example when it concerns a landscape photo. Sometimes this option is chosen in error, after which only administrators can re-enable species identification on the photo. As of now, moderators also have the option to re-enable species identification.
  • Another moderator improvement is that they are now allowed to geotag photos of others, in case the photo owner has not done so. If a photo is geotagged, however, moderators cannot edit it.
  • The Pioneer medal was sometimes incorrectly calculated for users, this should now be fixed.
  • The "more" button on the search results screen had a rendering issue, this is fixed now.

And the last improvement concerns inertia scrolling, which requires some explanation. This is mostly an iOS thing, although other platforms somewhat try to mimic the feature. Normally, when you open a web page on an iOS device (in Safari), you'll get "normal" scrolling behavior. This means that when you swipe down the page, the scrolling stops as soon as your finger leaves the screen. Inertia scrolling is different, the page will continue to scroll and then gradually slown down and utlimately stop. iOS users know this behavior from many native apps, but it's not enabled by default in Safari. 

Anyways, JungleDragon has had inertia scrolling enabled for quite a while, but the whole thing stopped working in iOS8. Only when you lose it, you will notice what a huge difference it makes in user experience. I've now fixed the issue so that it works again.

What will V4 be about?

As times goes on, my ideas about V4 are getting stronger, and more focused. I have quite a long todo list. 75% of that list can be seen as a continuation of V3, meaning it generally improves the site but makes no truly drastic changes. It is the 25% that I am most excited about though. They are new, important features that I believe in particular the core members will enjoy a lot. More on those ideas soon...

For now, happy holidays to all. Thank you for being interested in JungleDragon.

So long, s3maphor3.org! »

FERDY CHRISTANT - DEC 18, 2014 (03:51:20 PM)

In the period 2004-2008, before dedicating myself so much to JungleDragon, I actively developed a few smaller open source projects which I shared on a custom site called s3maphor3.org. Yes, I know it's a weird name. Since then, further development of these projects have been halted. Both the site and its projects have slided into irrelevance so I decided to retire it.

Not all is lost though. If you for some obscure reason still have an interest in these projects, I have moved them over to my Github account:

They are basically in an archived state, meaning on very limited to no support from me. Each project has a detailed manual though, so you should be able to help yourself. A special note on TagDragon: this used to require a mandatory donation, but is now entirely free.

Goodbye Diablo, welcome Lucifer »

FERDY CHRISTANT - DEC 16, 2014 (07:40:17 PM)

7 years ago, when we moved to our current house, I put a lot of effort into setting up a family server. It concerned a powerful Dell PowerEdge 1900 with Debian as the OS. I named it "Diablo". It has been a very useful server, both for its file sharing and for its many other services, primarily in LAMP development.

When I cleaned up and improved our home network two weeks ago, I also installed a new 4TB NAS. Hence, file sharing wasn't needed anymore on the home server. Given the age of the server, I figured it was finally time to retire it. 

However, it has a lot of services on it that I depend upon, mostly for JungleDragon development. This includes Apache, MySQL, PHP, SCP, SSH and Subversion. My idea was to create a new VM that has the same services and just run it from hy home desktop. The only thing I'd lose was the always-on aspect, but for development that isn't a big deal. 

Last weekend I finished the process and I could finally turn the server off for good. It may get a 2nd life as I will be donating it to a person that refurbishes hardware and makes it usable to poor families and small businesses.

Anyway, as of now I will be using the VM, which I named "Lucifer". And I'm loving it. Removing the old server saves space, energy, money and noise. I also love the flexibility in taking snapshots and easily backing up an entire machine. Any effort I've put into creating the new VM, will be put to good use, as it can be reused over and over again. 

Below I'll share some notes on how I set up the VM and which choices I made. It will not be a full tutorial, just some high level notes.

VM software

As virtualization software I used VMWare Workstation (which is paid). It can be done with the free Player edition but I like the added features, such as the snapshot manager.

OS install

I've chosen Debian 7 (stable) as the OS, mostly because I am used to this system and it simply is rock solid. I simply downloaded the DVD ISO and pointed VMWare workstation to that file in order to start the installaion. During installation, I made pretty standard choices. The only explicit choice I made was to include SSH, SCP and Apache.

Allocation

I allocated 4GB of RAM (out of 16GB on my desktop) to the new VM and 100GB of disk space. As for network mode, I started out with NAT, but came back to this later on. More on that below.

VMWare tools

Once Debian 7 powered up for the first time, it is time to install VMWare tools, these give additional control and ease of use regarding your VM. In a Linux system like this, it is a matter of untarring a file and running it.

SSH/WinSCP

From my host OS (Windows 8.1) I tested SSH using Putty and SCP using WinSCP, both worked out of the box. Sweet.

File share

In order for me to do development on this VM, I prefer my Windows-based IDE, so I need a file share to the VM. For this I installed and configured Samba. I pretty much followed this tutorial, after which I could succesfully map a drive in Windows.

PHP

Apache was installed based on my choice during the OS setup, but PHP was not. Luckily it is installed with a single command:

apt-get install php5 php-pear php5-mysql

I really love Debian package management. It has only gotten better. 

MySQL

Installing MySQL is equally easy, yet it requires a few more additional steps. After installation I binded the MySQL to the public IP of the VM, in order to make it accessible outside the VM. Another step required to enable remote access is provide the correct grant rules in MySQL's internal users table. 

With remote access working, I used Navicat to transfer databases from the old server to the new server. This is a great feature that saves a lot of import and export jobs (that may go wrong due to the wrong order of statements). 

PHPMyAdmin

To manage the databases, I'll use Navicat. But for completeness sake, I installed PHPMyAdmin as well. This too is a single command.

Apache

As said, Apache was already installed in the initial installation, however I discovered an essential module missing: mod_rewrite. I often use this in .htaccess files. This is when I learned that Apache has a very friendly utility to install and enable mods: a2enmod. Just run this command and pick what you need. It saves a lot of manual work in config files.

Zend debugging

By now I had a basic LAMP setup working. But we're not there yet. One feature I absolutely cannot live without is remote debugging on PHP. This requires the installation of Zend Debugger. I struggled a bit with this, since you need to find the exact version required for your choice of architecture, OS and PHP version. As my PHP was newer than the target version, I had to try out a few versions. Once correctly installed on the PHP end, you still need to configure 2 files to get it to work, but I ultimately managed. 

SVN

Although on this server I am the only developer, I still absolutely need Subversion for source code management.  Out of the steps in this overview, it is the most challenging to install, but easy once you know how. Here's the high level steps:

  • Use apt-get to install subversion and libapache2-svn
  • Create a repository parent dir and a users dir
  • Create a HTTP user in said users dir
  • Add user to a group and give that group recursive access on the repository parent dir
  • Set up the apache configuration for subversion in dav_svn.conf
  • Create a repository

If all went well, you can navigate to your repos parent dir in a browser and see the repository. You can also try to check out a working copy using a client like TortoiseSVN.

WebSVN

To allow for powerful repository browsing in a browser, I installed WebSVN. This too can be a single command, after which you can customize it to your needs. 

JungleDragon

By this time I had all services of my old server moved to the new VM, or so I thought. A true test would be moving JungleDragon's development instance to the new VM. After doing that, several things were broken, turns out I forgot a few things:

  • As said, I needed to enable "mod_rewrite" on Apache
  • I had to install curl, which was missing
  • I had to install Imagick, the image management library that JungleDragon uses for image resizing
  • I had to change some php.ini settings, in particular the maximum post and upload size
  • oAuth, which is used to sign in to JungleDragon using Google, Facebook or Twitter did not work (more below)

That said, email worked out of the box and copying crons from the old server also posed no new issues. The only issue that was remaining, was the oAuth issue.

oAuth

As said, JungleDragon uses oAuth to authenticate users. The problem with oAuth in this context is that it requires a publicly accessible redirect URL. That means that my VM should a) have a fixed IP b) have that address accessible outside my home network (or better said, within my network and then port-forwarded via my router).

Later on, I discovered that oAuth is not the only reason to give the VM a fixed IP, as a reboot in the guest OS may lead to a new IP each time, handed out by VMWare's virtual DHCP manager.

Hence I went on a journey to have NAT whilst having a fixed IP, with the additional requirement of it not conflicting with my router's subnet. I failed miserably. I've read every article, played with hidden config files, used VMWare's obscure virtual network manager, all to no avail. Every combination satisfied 2 requirements, but not all 3. 

I started reading on "Bridged mode", which makes a VM really part of the home network, instead of just virtually. Again I did not succeed and upon further investigation it turns out that the VM bridged network services were not installed on my network adapters, which is a typical Windows 8.1 problem. Deep in the night and in desperation, I made one more attempt after reading a post of somebody claiming a VMWare reinstall fixed his problem. And guess what, after reinstalling and putting the VM in bridged mode, all just worked. The IP is on the network, it is fixed, and it doesn't conflict with my router. 

README.txt

Some closing words on the VM: I document every single step that I make in great detail. The effort I put in now will make this a usable VM for years, and there will be a point where you will forget how it all works. In addition, I made it a habit to create a VM snapshot before any potentially destructive move on the guest OS, so I can always go back.

Anyway, that's it. I'm happy with the new VM and equally happy to retire the old server. And as a truly final remark: I consider it amazing how powerful open source has become. This entire stack is free of charge. We should be grateful for that.

Update: one more thing I just did is to try to automatically sync the projects folder of the VM (about the only thing dynamic, containing source code) to my new NAS. I'm using Synology Cloud Station for it. I'm still evaluating how it works, I'm particularly interested in what happens when it cannot sync due to the VM being offline.

Continue reading...