A Deep Dive into 20 Years of DG Tech

Mihai Postelnicu, based in Romania, has been with DG since 2006. In his time, he has worked on Aid Management Platform (AMP), EU-DEVFIN, IMS, OC Makueni, OC-Explorer, and DG-Toolkit. 

At our core, Development Gateway has always been a technology-driven organization. Since our early days, we have typically comprised 50% software professionals and 50% international development professionals, and referred to ourselves as working “at the nexus of two developments: international and software.” 20 years in technology is multiple lifetimes: we have seen many technologies, companies, and trends come and go – some leaving lasting impact, others unfulfilled promises. In this post, we’ll explore our history to reflect on how we have tried to constantly evolve and stay ahead of the curve, while not falling prey to chasing each new fad (hi there, blockchain), and maintaining a focus on building tools that our partners can use, expand, and sustain.

“The mandate of the Development Gateway is ambitious. The Internet offers the opportunity to reach diverse beneficiaries, and involve them in the development process in new ways.”
      World Bank Review – The Start-Up of Development Gateway

Since our early days, DG has been committed to using open-source tools and technologies. We believe this approach works well with our values, lowers the maintenance costs of our tools, and ensures community ownership. Besides being a user, DG is also a contributor to the open-source community, with the majority of our repositories being open source. You can find a full list of our open source projects on our github page. Larger open source projects include the Aid Management Platform (AMP), DG-Toolkit, and OC-Explorer – see more on each of these tools below.

Pre-History (2001-2002)

In the early years of DG, when we were first pursuing our mandate to “harness this new internet thing,” we used tools built upon TCL language and a fork of ArsDigita Community System (OpenACS). These were used to run a portal on the AOLServer, for the Gateways Communities project and other internal projects, like dgMarket.

During this time, DG used good old CVS for sharing code. Maintaining code branches was very expensive with this ancient versioning system, as it basically duplicates the entire code base. So branches were used very little and everybody was working on one single branch – not ideal for quality assurance and division of labor!

Java Times

In 2003, DG started one of its first open-source initiatives as part of our transition to Java: DiGi was used to administer users, permissions, and multisite requirements for portals. This was used to build the portal system and replaced the AOLServer/TCL portal for the Country Gateways (check out the still operational Burkina Faso Gateway). DiGi used Java and as web technologies it was built using Struts 1 and Tiles.

On top of this new platform, new projects like AiDA – a precursor to the partnership with the College of William & Mary and Brigham Young University’s PLAID initiative, which yielded AidData.org – were developed, dgMarket and dgCommunities were ported from the old TCL infrastructure.

This was also the period when DG started to use JIRA as a ticket tracking tool, which is probably the most enduring software used in DG – 17 years later.

Birth of the AMP

In the earliest days of AMP, Mihai would sometimes have to fix bugs during live demos. Thankfully, those days are behind us. Here he is during a successful 2008 AMP Pilot in DRC.

The very first version of AMP was also built on top of DiGi, using Struts, Tiles, and Hibernate. The first functional AMP version was deployed in 2006, and has been deployed in over 30 countries globally. Despite being almost 15 years old, AMP has been significantly re-engineered with a new architecture and continuous modular upgrades since 2014.

Subversion (SVN) Move

Starting from 2007, we moved our code repositories from CVS to Subversion (SVN). Because subversion makes branching cheap, this opened new possibilities in version control and software releases. Branching and tagging of versions was no longer an expensive operation that would limit its use – this advancement was particularly crucial for projects like AMP, which grew from an individual country to over a dozen in just a few short years – many with code versions deployed in to meet different country requirements.

Productivity Tools and Methodologies

In 2011, the same year the Agile Alliance created the Guide to Agile Practices, we implemented our first true Agile project. Over the past 9 years, DG has adapted the Scrum methodology to fit with our team culture and project needs – expanding and contracting the level of process depending on the size of the project and team.

As a global team, with technical experts in Argentina, Armenia, Kenya, Moldova, Romania, and the US, having strong productivity and communications tools has been a consistent requirement for our work. We have added new productivity tools into JIRA (sprint boards), we rely heavily on Slack (after a decade on IRC, Google Chat, and Skype), and use time tracking apps (ActiTime in 2008, JIRA integrations with T3 Systems, and ClickTime in more recent years). These tools have helped us to stay on the same page, compare actual time spent to estimations, and ensure progress toward project goals.

Git and Gitflow

The arrival of Git and Gitflow Workflow has proven to be another huge leap in team productivity and code quality control. Git is a distributed version control system that replaced Subversion in our infrastructure. While more complex than Subversion, it has been an excellent aid to developers working remotely. Having a copy of the repository on your local computer means faster commit times and easier-to-segment work for multiple commits based on the type of changes, without the need to wait for a repository server response in between commits. We have been using Git since 2013 and it has been transformational for our developers.

Gitflow Workflow is a Git branching model that improves code quality, productivity and stability of applications. It segments work into features, and applications into stable and development branches. This segmenting ensures concurrent work can be done on multiple versions of the same app, with the maintenance (bug fixes) and new features updated separately. DG has used gitflow on the majority of our products since 2013, and we like blending it with Github’s pull requests.

Some Current Go-To Tools

Apache Wicket

Apache Wicket is an open-source component-based web application framework, which graduated into an Apache top-level project in 2007. DG was an early adopter, by 2009 we were using it to build complex forms. It is still in use today in some projects where complex data entry and validation are a must.

Spring Framework

Spring Framework is an application framework for the Java Platform, it dominates the Java ecosystem with 60% using it for their main applications. DG started using Spring in 2006, and Spring Boot (a convention-over-configuration solution for creating Spring-based production-grade apps) since Spring Boot 1.0 (2014). It provides a robust application framework for developing the backend of our tools.

ReactJS

As was the case with Wicket and Spring, we were very early adopters of React.js – after a lengthy internal evaluation of React.js, Angular.js, and Backbone.js – with the first apps built on the library traced back to 2014. Because of its lightweight footprint, efficiency, and relatively mild learning curve, React.js has quickly become the building block of any frontend-centered application developed by DG.

Bringing it All Together: The DG-Toolkit

To bootstrap projects, our team has developed a boilerplate repository, analogous to a template in Microsoft Word. Based on Spring Boot, Apache Wicket, React, Hibernate and MongoDB, the open-source DG-Toolkit project sits at the base of many of our tools. It provides pre-built user and role management, test form components and integration with database layer, monitoring tools, and built-in caching. With the DG-Toolkit, our coders can quickly jump to developing project features and not waste time configuring and wiring together technologies. All told, DG-Toolkit saves us multiple weeks at the start of each project in which it is used.

Open Contracting Explorer

DG has actively contributed to the development of the Open Contracting Data Standard since 2015, as well as to the tools built around the standard. With our work in Vietnam and Ukraine, the Open Contracting Explorer (OC-Explorer) was born. Built on DG-Toolkit, this Open Contracting template project is the foundation of our Open Contracting work, and provides building blocks for publishing, validating, and in-depth analysis of open contracting data, including corruption flagging.

Where Do We Go Next?

Predicting the future of technology is always challenging, and never more-so than during 2020. While we expect much to change – new frameworks will emerge and dominate, new buzzwords will emerge and fizzle – we have seen over 20 years that a thoughtful, open source-driven, agile approach to technology implementation has created efficient and usable systems, and has served us and our partners well. As we continue to build out new tools to track data on critical issues like tobacco control; agricultural input supply, price, policy, and use; government procurement; aid management; and extractive resources; our priority will remain building simple, stable, sustainable tools that directly meet the needs of our partners and clients.