FERDY CHRISTANT - MAY 30, 2015 (09:42:35 PM)
Hereby I am announcing a new side project, code-named "Calumma", after the chameleon genus. It is a vaguely defined project of an uncertain length and outcome, yet I still want to share its existence with you.
Project Calumma combines two goals that I am constantly after:
- To improve myself as a web developer
- To improve JungleDragon
Let's go into those goals a bit deeper...
Web development, with an emphasis on front-end engineering, has been one of my passions for a long time. But it's more than that, it's also my bread. I earn a living with it. As the world of front-end engineering is facing one technological revolution after the other, I must keep up.
The way I keep up, other than working on challenging projects at work, is to devote spare time on experimenting with new technology. Therefore, in this new side project, I will freely experiment with such technologies. I will specifically focus on development workflows, frameworks and new design techniques.
This part of the project is all about learning and personal development in my field. However, I intend to do it in a way that will also benefit JungleDragon, as well as anybody reading this blog with a technical interest.
This is the part where I am trying to kill two birds with one stone. From JungleDragon V2 until V4 right now, the platform has seen an avalanche of new features. Although there is always more to wish for, I believe we have arrived at a state where we have the major parts in place.
I am happy with the result. I think it is a great site. It works quite well, is fast, has a good usability (despite a few small flaws), and looks good. I am proud of it.
Yet pride should never get in the way of further improvement. The idea I have for project Calumma is to take the learnings from experimenting with new ways (phase 1), to see if I can raise the bar for JungleDragon from good to world-class.
There are definitely things in JungleDragon that I would have coded differently, as well as designed differently in hindsight. With project Calumma, I will freely experiment with technology and design to see what is possible.
In order to not be constrained by current choices, all of this will be developed completely separate from JungleDragon V4. JungleDragon V4 for now is going nowhere. I will obviously keep running it, managing the community, running contests and fixing bugs. Meanwhile, however, I am researching the next-gen JungleDragon in project Calumma.
The reason why I combine these two goals is because it maximizes both the efficient use of my time, as well as the output of that time, which will benefit both me and JungleDragon. Two birds, one stone.
What this will actually mean for JungleDragon is uncertain. It may mean nothing. It may mean small tweaks to current JungleDragon, it may mean JungleDragon V5. All of this depends on phase 1. The project is in a too early phase to tell.The plan
Although some vagueness still surrounds the project, I do have a plan in place. It may be subject to change, but here's the outline of some of the steps I have in mind:
Recently I have been experimenting with optimizing the development workflow, using tools like gulp, bower, and more. I will further invest time in learning these tools, as it will benefit both my personal development as well as my productivity. Better tools means better output.
In project Calumma, I am piecing many best practices together to arrive at a robust and powerful front-end framework. This consists of many components, for example for layout, responsive design, typography, feature detection, the list goes on and on.
This is a crucial phase of the project, as it will the base to build anything on. I will tune this framework to the point where it is very robust and flexible.
Why? If I have an ultra powerful front-end framework, I can build almost anything. The time from idea to execution becomes shorter, and its implementation more reliable and consistent. This phase of the project will take a lot of time, and lies at the core of both my learning goal and my JungleDragon goal.
To test my front-end framework, I will have a series of test pages, which showcase and test specific aspects of the framework. For example, there will be a page to test the grid, the typography, etc. In addition, there will be a styleguide showing all common styles and UI components in the framework.
After all of the previous steps, I'm going to put both my learnings and new framework to the real test. In this step I will design some crucial JungleDragon pages from scratch, such as the homepage, species page, country page, etc. This will not be on the live site, they will be static pages, using sample content.
As said, the goal is to go from good to world-class, so this one challenges both my skills and my framework. If I find the framework to be lacking, I will go back to the previous step. If I find my skills to be lacking, I will focus on learning.
That, in a nutshell, is the goal of Calumma: to challenge myself as well as JungleDragon's current design.
In this last phase of the project, I'm going to decide what to do with the results of the previous steps, and draw some conclusions. They are yet unknown. It could mean all of it was simply a good learning exercise. Or, it could mean the newly designed pages are so awesome that JungleDragon V4's front-end will be redesigned. Or anything in between those conclusions. I will obviously involve the community in getting feedback.
Work on Calumma has already started, and I am progressing quite well. I am working on multiple steps in parallel. I have a basic development workflow going, I have a base version of my framework (which still needs lots of work). I also have some test pages in place, and am currently working on the styleguide. The coming weeks I expect to be absorbed by these 3 steps. They require a hefty investment in time.
As for further timing, I don't have any hard targets, other than as fast as possible within my time constraints. This is a research project with an uncertain outcome and timing. It needs time for exploration, creativity, learning and quality. It very much is a long term engagement.
I'm sorry if the above sounds vague or too technical. This is only the first announcement, things will become more clear as I progress in this ambitious new side project. I am quite excited about it :)
FERDY CHRISTANT - MAY 15, 2015 (10:40:51 AM)
JungleDragon update 23 of V4 is now live. This update brings a few small improvements at species records. We'll be using the "Red fox" species page to show the improvements:
The improvements are in the right column. First, the distribution map:
This image is automatically parsed from the corresponding Wikipedia page for the species. Not all species records have them. Usually, such an image is aided by a legend text below it. Occassionally JungleDragon has trouble in accurately parsing that legend text from the Wikipedia page, as it can come in many formats. Therefore, if there's any issue or weird characters in the legend text, it can now directly be edited by moderators on the species edit form:
A small but welcome improvement. Before, I had to directly hack this in the database, without any user interface. Up next is the taxonomy block;
The change here is that the values are now links, where before they were just text. In a way, they are the same links as shown in the breadcrumb. Here's what happens when clicking "Canidae":
All canids directly in the species browser. Nice.
The next block is the "Photographed in" block:
This lists the countries in which this particular species was photographed. Before, when clicking such a country, it would take you to the species tab of that country, showing all species of the country. A nice starting point, but maybe not what you expect. This is now changed, when clicking the country in this list, it will show you all photos of that species in that country. Here's all photos of the Red Fox taken in the Netherlands:
Finally, on the moderator panel, both total lists of species are now divided across pages. Instead of loading thousands, it now loads 240 per page.
That's it, hope you like these improvements.
FERDY CHRISTANT - MAY 9, 2015 (01:38:49 PM)
JungleDragon update 22 is now live. This is a small maintenance update, with the following improvements:
- Before, there was a maximum of 3 nested comment replies, this has now been raised to 10. In addition, when you get to level 5 in the nesting or higher, it becomes more complicated to see to which comment somebody replied, this is solved by showing thin dotted lines to connect a main comment to a reply comment.
- JungleDragon is now slightly faster due to a series of small performance improvements, mostly made possible by better compression and caching of some resources.
- I solved a bug in the process that automatically tweets videos to JungleDragon's Twitter account.
- Users who are inactive (not activated, suspended, banned) could still upload photos if they still had a login session. This is now fixed.
FERDY CHRISTANT - MAY 1, 2015 (11:05:03 PM)
Since a year or two, build tools in the world of front-end (web) development are all the rage. Follow a few blogs on web development and you will be overwhelmed with names like Grunt, Gulp, Bower, an endless stream of tools. These tools can come across as intimidating to newcomers, due to the command line and myriad of config files.
Yet they cannot be ignored, and I found a reason for myself to dive in. First, in my project JungleDragon, I had a strong need to automate my asset workflow (CSS, JS). Second, at work I am increasingly dealing with 3rd party web design agencies who work with these tools as a default, therefore I must master them as well.
Time to get my hands dirty.
Define your desired workflow
Before you dive into these tools heads-first, I recommend first thinking about which parts of your workflow you want to automate. Just list the steps. This will avoid you from drowning in possibilities. The point of using these tools is to solve real-world problems.
In my case, I wanted to automate the handling of my CSS and JS, as follow:
- Compile SASS into CSS, with Compass support (I author SCSS, not CSS)
- Minify the resulting CSS file
- Gzip the resulting file
- Give the file name a unique timestamp (to bust cache)
- Uglify JS (minification)
- Gzip the resulting file
- Give the file name a unique timestamp (to bust cache)
The Grunt way
With the above plan in mind, I started playing with Grunt. You can find the steps to get started on their website. After a brief setup, I was quick on my way to start implementing the above plan.
I was surprised and excited at how easy it is to learn this tool. After a general project setup, you just install the plugins you need and configure how to run them in a file called "gruntfile.js". Next, from the command line you run "grunt" in the directory and it executes your task.
After a bit of tinkering, however, I came across this article. It is a most excellent article that explains how Gulp is better than Grunt. Indeed, I did find a few issues with Grunt:
- It's not very fast (although usable)
- Larger gruntfiles can become quite messy and hard to read
- There is no direct way to use the output of one task as input for the next task, other than writing temp files in between both steps.
That article convinced me to throw my grunt setup to the side, and to try and replace it with Gulp.
The Gulp way
There is no real point in me repeating the setup steps in the article I mentioned, hence I am simply going to show you my entire gruntfile.js and explain it line by line:
Line 1-12. At the beginning of our gulp file, we declare and include the plugins we require for the actual tasks to perform. Each of these plugins should be installed prior to using them. Whenever you need a gulp plugin, just search online for "gulp [taskname]" and very likely you will find a packet to support that task.
As you can see, most require statements just name the plugin to load, however, some take additional options, such as the gulp-s3-upload plugin.
Line 14. Here we are declaring our first task, in this case called "clean". Within the tasks' curly brackets, we find the statement to run. In this case there is a single statement that clears out any JS and CSS files produced by prior runs.
This task is independent of anything else in the gulp file. To run this task, all you need to do is to open a command line at your project's root, and run the command "gulp clean".
Line 18. A 2nd task, called "styles". This one is infinitely more interesting than the clean task, as it contains many steps and shows the true power of gulp.
Line 19. The return statement acts as a filter to select the files to run all the next statements on. In this case, it runs on all my SCSS files.
Line 20. Our first meeting with "pipe", which is what gulp is all about. Pipe executes a task on the previous stream. In this case it runs the Compass task on all the SCSS files. This means it will compile SCSS into CSS with full Compass support. In the curly brackets you can see some options being passed in, such as the location of my SASS (input) and CSS (output), as well as a pointer to a special config file. The compass config file is no big deal, it merely sets the image path for any images referred to in the CSS itself.
I'm very happy with this task, since it completely replaces ScoutApp, which I have been using for SCSS compilation.
Line 21. At this second statement within the task you should now already see the true power of gulp, which is to chain tasks. With this single line of code, the output of the previous task (SCSS compilation) is used and then minified. This happens in memory.
Line 22. The minified CSS is gzip compressed, in a single statement.
Line 23. The rev task takes the file name from the previous task and adds a unique hash to it.
Line 24. It is time to actually write our CSS, which is now compiled, minified, compressed and versioned, from memory to file. gulp.dest does just that.
Line 25. This statement creates a version manifest. Don't worry about it for now, I'll explain that concept separately later on.
Line 26. We're using gulp.dest again, yet this time to write the manifest file, not the CSS file, since that was written to disk already.
Line 29-37. My workflow for JS is slightly different from the workflow for CSS. Therefore, I have a different task for it. I will not explain each step since it is largely similar to the CSS tasks. The only difference is that Compass is not used, and instead of minify, we use uglify.
Line 39-48. When I am doing development and am making changes to JS and CSS, the "styles" and "scripts" tasks explained above will do useful build automation tasks. At one point, however, I am done with my changes, and I need to deploy them to production. In production, JungleDragon uses Amazon S3 (in combination with Cloudfront) to host CSS and JS assets.
Before, I uploaded them manually to S3. Next, I had to go through each uploaded file and set the permissions and meta data header. This gulp task does all of that for me in one go, simply by typing "gulp s3_styles" from the command line. This saves me several minutes for each production deployment!
Line 50-59. The same S3 task, but this time for scripts. The only difference is the storage location in S3.
Line 61-63. A general s3 task. The purpose of this is so that I can simply run "gulp s3" from the command line in order to run both s3_styles and s3_scripts in one go. Note that these tasks run concurrently, not after each other, so be sure to not have tasks that can interfere with each other.
Line 65-67. The default tasks is a special kind of task. When you simply run "gulp" from the command line, without any specific task name, all statements in this task will be run. In this case, it will build my styles and scripts. Pay particular attention to the "clean" parameter. By passing it along like this, the styles and scripts tasks will only be ran when "clean" has finished. We don't want the clean task to run when the other tasks are producing new files in the same directory.
Line 69-78. Another incredible power of gulp. The watch task is a special task. You run "gulp watch" from the command line, next this task will keep polling for changes to files and run tasks on them in case something changed.
For example, if I make a change to SCSS in my IDE, "watch" will pick up on it and instantly build the file according to the logic explained earlier.
I hope you found the above code listing to be intuitive to understand, and am excited as I am about how powerful and simple this is. Anyway, I promised you to get back to version manifests, what they are and how they work.
Let us say that in production you have a CSS file called main.css. From within your pages, there are references to that file. You have set a far away cache expiration date on the file for reasons of performance.
Next, you make a change to main.css. Before deploying it, you need to rename it, otherwise main.css would still be retrieved from the user's cache. For the sake of argument, let's say we call it main_2.css and deploy it to production. Next, we need to update all references in our code base to main_2.css.
This is quite a pain when we're talking about dozens of assets and even more references to it. Let's see how this is solved in my workflow.
First, there is the input side. I author SCSS and JS in my IDE. These files have stable names, without versions in their file names. As I make a change to that SCSS file, gulp watch will build it. In the code listing above, you can see that I'm making use of "rev". This will rename the compiled file in the output dir (not the input dir).
As an example, JungleDragon's main CSS file "jd4.css" becomes "jd4-f4c97482.css" in the output dir. Great, it is uniquely named so when I deploy it, users will get fresh content. However, now I need to update every reference to "jd4.css" in my code base into "jd4-f4c97482.css".
Using version manifests, this is unneeded. A version manifest is simply a JSON file that maps an input file to an output file. For example, here is my automatically generated "styles.json" manifest:
The final step to make all this work is to transform your code base to not make direct references. Instead, you will create a helper function that calculates an asset path. In my case, it's a simple PHP function. It uses json_parse to build an object, and then will look up the requested resource from the manifest.
This means that all your references will ask for the stable name, i.e. loadStyle('jd4.css'). Your helper library will automatically return the real path of the versioned file. Once you have transformed your references in your code base like this, a one-time exercise, you will never ever need to change them anymore.
The above setup took only a few hours to implement, mostly because I was new to these tools. To summarize the accomplishment, my entire asset management problem is now solved by two commands:
gulp watch: To do all asset handling whilst I am developing.
gulp s3: To deploy the latest assets to production, not needing to update any references to it.
That's pretty powerful, don't you think?
Possible next steps
The above implementation focuses on my biggest pain points, but likely I will extend it over time to automate even more. Promising options to me would be SVN integration, concatenation of files and live reloading my browser. All in good time.
Having worked with gulp for a bit now, my general impression is that it is incredible. I consider it a tool that I will use in most, if not all projects from now on. Yet as nothing is perfect, I will also list some drawbacks:
- Plugins can have dependencies on other plugins or go into a state of being unmaintained. You can come across some articles where the installation instructions no longer work.
- Not every plugin support streams. That's an issue because the "pipe" mechanism relies on it, and it is considered gulp's most important benefit over grunt. Plugins without stream support need to write in-between files just like grunt.
- Another claimed benefit of gulp over grunt is the readability of the syntax. This is true for simple setups, but as your gulp file grows, the benefit largely diminishes.
- Gulp is claimed to be much faster than grunt due to using in-memory streams. Whether this is true depends on the nature of your tasks (and whether they really do support streams). In my workflow, there is no meaningful speed improvement at all, so your mileage may vary.
- I've found Grunt to have much better debug info than gulp in case something goes wrong. This is not in the system itself, rather in the plugins.
To end this article, my recommendation is simple. Create a local web project dir, and make your way through the article I linked to. In an hour or so, you will find yourself having some powerful working build tasks. It's far from intimidating, in fact so simple that I regret not investing this time earlier.
FERDY CHRISTANT - APR 20, 2015 (09:34:48 PM)
JungleDragon V4 development started less than 4 months ago, 117 days ago to be exact. In that period, a whopping 21 site updates were deployed, and many of them were big, not just small changes. Update 21 closes an important goal of JungleDragon V4, therefore it is time for a strategy update. In this post I will look back at the previous period, as well as forward as to what is to come.
JungleDragon 4 - core goals completed
Hereby an overview of what happened during the last 21 updates:
Update 1: Small moderator improvements and bug fixes, the kickoff of a new season of development.
Update 2: A core feature, species identification, became 10 times faster!
Update 3: Personal tags radically improved, allowing you to organize personal sets any way you see fit, with full slideshow support.
Update 4: Species introductions, identifications and species photographed are now linked to user profiles. At a single glance you can now see for any member their species "collection" and their overall contribution to JungleDragon.
Update 5: Another brand-new way to explore photos: by season, where the season is automatically detected from your photo's timestamp and location. Furthermore, every single list in JungleDragon now has a counter in the "More" button.
The big idea: An in-between post in which I envision what V4 will really be about: exploring wildlife by geography. Most of the updates that came after this are related to these ideas.
Server move: Unplanned, yet JungleDragon had to move to a brand new server and software stack. As a result, the site is faster and more scalable overall. Some specific actions (such as processing photos after an upload) are now dramatically faster, Im talking 10 times as fast. As a result, there's virtually no actions anymore in JungleDragon where you are really waiting for the system.
Update 6: A new slider carrousel on the homepage explains to new users the unique aspects of JungleDragon, which are not immediately clear otherwise.
Update 7: A better photo edit flow, and a brand new feature for handling photo deletions, which greatly streamlines the old process. Furthermore, several fixes.
Update 8: Species introductions, JungleDragon's most heroic action for any user to realize are now further celebrated by means of a new indicator on the photo page as well as any thumb of a photo.
Update 9: Moderators who spent a lot time educating other users in how to properly share photos now have a new guidance tool which saves a lot of time explaining the same concept over and over again to new users. The guidance features automates standard messages and is further supported by help content. In addition, this update introduces feature control, which allows administrators to selectively enable/disable site features without bringing the whole site down.
Update 10: This update marks the true start of the "country idea". In this update, I prepared the back-end and data sources, enabling the country features in the updates that followed.
Update 11: An essential update which introduces exploring navigating to countries via a country selection screen, guided by a cool widget. For the first time, a "country" is a very visible thing in JungleDragon.
Update 12: Finally, photo overviews (not just map pins) are added at the country level. As of now, anyone can explore photos by country, which is crucial. And of course, there is full slideshow support.
Update 13: In order for the new country features to work well, photos need to be mapped to countries. This works automatically for geotagged photos, yet this update introduces simple yet powerful tools to set the country for non-geotagged photos. Besides species identification, this brings a new discipline to photographers and moderators.
Update 14: This update further supports the phenomenal progress the community made in mapping photos to countries. Using personal "unmapped" overviews and country counters, it is a task now easier to track and manage.
Update 15: A historic JungleDragon update which introduces species at the country level. This feature is a breakthrough of which the impact will only be felt longer term. In addition, this update also introduces an entirely new concept: country intros. Another way to fuel to obsession of the community in collection species.
Update 16: Many country updates packed in a single update: country intros at the user profile, a brand new link from a species record back to countries, new sortings for selecting a country, better country borders on a map, and moderation help updated.
Update 17: A crucial follow-up to the country species feature: you can now filter them based on popular species categories. This makes possible things like "birds in Costa Rica", "Mammals in the USA", etc. A very natural and intuitive way to go about exploring wildlife. A vision coming to life.
Update 18: A spectacular update that introduces parks and protected areas for countries. On a country map, that countries' protected areas an parks are drawn as polygons and points. A great guide in planning wildlife trips in your home and travel countries!
Update 19: A simple update that adds member overviews at the country level, allowing you to find peers in your country.
Update 20: This update introduces direct messaging, allowing members to contact each other privately, in a secure and controlled way. Combined with update 19 this makes connecting with people in your country possible, for example to plan a photo trip together.
Update 21: This updates introduces videos at the country level, similar to how one adds videos from Youtube and Vimeo for species.
Phew, those are a lot of updates, with a lot of progress. Mountains were moved in a very short timeframe, and we did it together. The community has been excellent in appreciating the updates, using them, and reporting bugs. In just a few months time, JungleDragon has radically changed in many ways.
The most important takeaway for me is that the vision to explore wildlife by geography in rich ways is now a reality. The ideas outlined in The Big Idea are now here. They are being used and will only grow in value over the years to come.
That's right, I'm calling the "country" feature complete as of update 21. If you read the Big Idea post, you may note that I also mentioned country books and travel tabs, but those I consider of minor importance, something for the future. In addition, the parks feature has been delivered, but not as deep as initially imagined.
More importantly though, we now have country photos and species and tools to map photos to countries, which is the real core of the concept. It's done!
In a way, what I envisioned for V4 is now done. Yet I will not close V4 development, it will continue.
For the 2nd half of V4 development, meaning the months to come, I will focus on streamlining the features we have. JungleDragon has a lot of features currently. Instead of building even more, I am now going to focus on optimizing the features we already have. Delivering them better, faster, more beautiful, and better usable on mobile.
Instead of some updates introducing big things, this phase will be about many updates improving small things. It will be a tweak phase. The devil is in the details, as they say. Ultimately I want every page and interaction within JungleDragon to make sense, be usable, clear, beautiful and fast.
There's also a personal note attached to it. I tend to do the heavy lifting in winter, when the days here are dark and cold. These last months have been a bit extreme in my devotion to V4 development. Don't get me wrong, I love doing it, but it was a bit of an unhealthy balance compared to the other responsibilities in my life.
Therefore, development will go back to a more normal pace. Regular updates, but not into the extreme. Catching some breath for both myself and the community. A more balanced approach. Yet don't forget: many small updates over time will still add up to big changes a few months from now.
Thank you as always for being part of this. It's a long road, but I seriously believe that we are now at a point where JungleDragon is unmistakingly valuable and useful. I hope you agree.